The Root Of All Evil

For a million years you are being trained to reason about the physical world as perceived by your sensors. You evolved to search for patterns and assume animal agency by default (simply because the cost of the mistake is lower with this assumption). Then came computer programs… they are invisible for your sensors, they do not follow any patterns, they can make computers appear animate, they can disguise as a reasonable actor, or fool your senses otherwise. And on top of it all they do not obey the laws of physic, the laws that your brain perceive as unbreakable for any agent in the visible world. This is a disaster for your neolithic brain.

Bushes shaking --> tiger; tools missing --> thief; supplies eaten --> mice; topsoil disturbed --> boars… This is the world your brain evolved in. But these are only the most trivial examples of pattern recognition, your brain is much more into this game: looking for patterns in animal tracks, weather consequences, food effects, and above all, the ultimate skill that made us all survive and now backfires: LEARNING ABOUT ANIMAL BEHAVIOUR.

Picture that: you are transported in a time machine 40000 years ago, you stay surrounded by beautiful green calm wild forest, and hear a sudden noise. Which assumption is more likely to be true?: this must be an animal; this must be a tractor. It is obvious that a brain that goes for the latter assumption will not survive in this forest long enough to replicate self. I assume here that you can comprehend the evolutionary logic without digging into details, so many times explained by the authors much more eloquent than myself. So, don't think too much of yourself, no amount of college education would rid you of the assumption that saved lives of your every ancestor. Things that act are by-default assumed to be animate.

When you present your brain with a computer that obviously REACTS to various stimuli the first automatic assumption your brain makes is it must be a strange animal — no matter what you think afterwards, the assumption is made, neurons are fired. Then your brain moves on to looking for patterns, and walks right into the trap: there are no patterns in a computer behaviour — the computer behaviour is (1) completely dictated by a program (2) this program can produce any behaviour (and by any I mean ANY, look up the definition turing machine). From the behavioural perspective computer is not possible to be understood (to the extent of being non-entity, if you wish) — your brain deals with a program while thinking to be dealing with a computer, and the program remains invisible, thus being completely non-reflected in your brain, the actual actor «program» does not present in your default model that the brain is building (why? because your brain has no sensors to detect a computer program!).

Furthermore, all those patterns, that your brain is so highly trained to recognize and comprehend, have been encountered in the physical reality and consequently obey all the laws of physics and bear plentiful apparent limitations that are assumed by your brain to be totally necessary for all occurred cases. A gazelle can not run faster than sound. You live with this experience at heart. It is so astonishingly obvious for you that you are now likely angry at me mentioning the fact about gazelles.

Let's make a thought experiment: Imagine a clay pigeon shooting simulator. You are «shooting» a fast moving «clay pigeon» drawn on a computer screen by using a toy shotgun. (It is likely you have performed this experiment in real life many times, both digital and tangible variants.) Are you going to aim with deflection? Certainly you are. But why? Because you know. You know what? You know physics! Physics, Carl!!!

You blindly project all your hard learned physics knowledge to the computer. Well, before anything else you have projected the knowledge to the clay pigeon itself, assuming that it will maintain its speed for the duration of the shot, and with a probability close to 1 beyond statistical error your guess is correct. But the computer emulated «pigeon» does not have to behave like its real prototype, in one case it is programmed to behave well, in other cases it could jump across the screen irregularly. There is no limit to that. I can hear you screaming: How so?! It should emulate clay pigeons accurately! Why should it? Because I know the authors! Because it is called «simulator»! Because it is a trustworthy vendor! Not this one, please! Because I paid money for this program! Because we played this game 1000 times having the same outcome. (This last one is very interesting, we will return to it later.) You may create thousands of «reasons» for a program to behave the way you expect it to behave, despite some of them are probably true, the statement holds: there is no physical law that affects a program. And you know, what? Only the physical laws are unbreakable. A program may be written to satisfy your needs, but it may as well be written to pursue any other goal. The issue here is that you do not keep this simple fact in mind when you apply your reality-based reasoning to a program. Once again, I do not challenge your knowledge at all, I just emphasize the fact that a computer program has nothing to do with your reality, no matter how well you have learnt your physics lesson. You may now curse me and remain ignorant. It's fine, as long as you are playing this «clay pigeons» with your friends; as soon as you dare play the very same «so well known» game against a casino, your ignorance will cost you real money.

Well, you may create a procedure to ensure a desired output of a program (in the end of the day you can write this program yourself), but this behaviour is equally unnatural for a program as all the rest alternative variants. ANY PROGRAM IS ARBITRARY. I don't know how many different ways I have to phrase this idea… it seems no amount of reasoning can thwart people from bringing their «natural» expectations to a casino, ...but shouldn't it have been different this time?

Here is another way to express the futility of all your spoken and unspoken presuppositions and rebukes. Corollary to the absence of physical laws: NO AMOUNT OF OBSERVATION CAN LEAD TO A VALID DEDUCTION IN REGARD TO A COMPUTER PROGRAM. We have run the program 1000000000000000 times, and the output is constantly X. We expect it to be X from now on. — this expectation is wrong, the program could be designed to give you exceptional output exactly on the 1000000000000001st run.

I understand the temptation to make this deduction, in fact the entire present article is trying to explain the nature of this temptation. So, please, understand your temptation and reject it. Exactly this pattern searching temptation is responsible for a significant share of the gambling and the Infosec markets. Again, I hear you arguing that the program from my example is less likely than a program that gives «consistent» output. NO, IT IS FUCKING NOT. I would not question a similar probability claim about a ping-pong ball, but you have absolutely no ground for making this claim regarding a computer program. Try to understand that. You simply project your physical experience.

Every time your brain is dealing with a black box (should it be a wild animal or a computer program) a very well defined set of natural laws is always always always assumed, while none of them is applicable to a computer program — this lays the foundation for all the misconceptions in Computer Science, and consequently in Infosec.