Game developer: Beware algorithms running your life

22 August 2011 by Alison George
Magazine issue 2826. Subscribe and save
For similar stories, visit the Interviews Topic Guide
Our decisions, our culture, even our physical landscapes are being shaped by computer algorithms, says Kevin Slavin. He tells Alison George why we should be worried

You claim that our lives are ruled by algorithms. In what way?
Put simply, an algorithm is a set of instructions that a computer uses to make a decision about something. They are like an invisible architecture that underpins almost everything that's happening. The way Wal-Mart prices its goods, the movies you rent on Netflix, the contours of the car you drive - they can all be traced back to an algorithm. Seventy per cent of trading in the US stock market is "algotrading" - executed autonomously by computer algorithms.

Why should we be worried about this?
The pernicious thing about algorithms is that they have the mathematical quality of truth - you have the sense that they are neutral - and yet, of course, they have authorship. For example, Google's search engine is composed entirely of fancy mathematics, but its algorithms, like everybody's, are all based on an ideology - in this case that a page is more valuable if other pages think it's valuable. Each algorithm has a point of view, and yet we have no sense of what algorithms are, or even that they exist.

You believe that algorithms are starting to shape our culture. How so?
Take Netflix, which is used by 20 million people to rent and watch movies. Of the movies rented in the US, 60 per cent are chosen because Netflix recommended them. It does this using an algorithm called Pragmatic Chaos, which takes into account other movies you like and how many movies you rated before you rated your most recent one. The algorithm is taking ideas about human behaviour and coding them and reinforcing them.

The danger is that such an algorithm can create a monoculture. But this is not the way culture works - it is actually much spikier, much less predictable. The movie Napoleon Dynamite, for example, always breaks the Netflix algorithm: people who really should love this movie hate it, and people who should hate it, love it.

Why is it so important that we be aware of algorithms' impacts on culture?
If you know that machine control is part of the picture, you might behave differently. Once you are aware that most of what you are renting from Netflix is based on a very specific model of the human brain that might not correspond to reality, maybe you would start asking your friends what they recommend - which is what we used to do.

It is also important to understand how algorithms shape what you learn and know. There's a quiet war in the US between Google and a company called Demand Media, which generates content that is optimised for Google searches. When Google changes its algorithm, Demand Media's output becomes worthless until it can figure out what Google has done and rewritten its content to match.

It used to be that you wrote news for how people read - now it's written for how machines read. Imagine if we all had to change our handwriting to a certain style so that computers could recognise it. That is effectively what is happening, but inside our heads. It is shaping our expression and behaviour.

How else are algorithms changing our world?
They are changing the infrastructure and the terrain. Take New York as an example. Wall Street became a market centre because this is where the ships and goods came in. Later, the Western Union building became the communications hub in part because that was where the telecommunications infrastructure was.

Today, however, the network hub for Wall Street is in this little town called Mahwah, New Jersey, because this was the safest place to put the critical infrastructure - within 10 miles of Wall Street but as far as possible from nuclear power plants, geological fault lines and flight paths. All the buildings going up in the area house and refrigerate the servers that run the algorithms - there are basically no people in these buildings. They are constructed entirely for network topology.

So algorithms are influencing the planning of towns and cities?
Yes, because the speed at which they operate is core to their efficacy, and speed is determined by proximity to network hubs. If you can make trades before somebody else, you have a tremendous advantage. Fibre-optic cables used to be laid alongside railroad lines, but that is changing because railroads have circuitous routes through cities and that's too slow for algorithms. It is better to lay them in a direct straight line. One company, called Spread Networks, has done just that, building a 1300-kilometre straight-line network connection from New York to Chicago, just to shave milliseconds off trades between these two markets.

Tell me more about these financial algorithms. Presumably they are top secret and very valuable...
Yes, but occasionally they leak. One of Goldman Sachs's algorithms leaked and was in the wild. They had to do a whole investigation to figure out how that happened.

Can algorithms get out of control?
Here's an example. A postdoc wanted to buy a copy of a developmental biology textbook called The Making of a Fly on Amazon. There were 17 copies for sale, starting at $40, but two copies were priced at $1.7 million. When he checked later, the price was $27 million. He tried to work out what was going on. Basically, two pricing algorithms had got caught in a loop, multiplying the existing price by 1.3 and offering it again. Because algorithms have the logic to raise the price but not the common sense to recognise the value, they just kept escalating.

I guess it wouldn't be so funny if financial algorithms behaved in the same way.
Right. This was just a pricing algorithm, but the ones used in Wall Street are not only determining the price, they are also executing against it. The Amazon one is harmless because there is still a human on the buying end saying: "That's crazy, I'm not going to spend $27 million on a book!" But if an algorithm were buying the book, which is the case on Wall Street, the price would keep going up until it reached the limit set by the system.

Has this ever happened?
There was this thing called the Flash Crash on 6 May 2010, where suddenly 9 per cent of the US stock market disappeared in a couple of minutes. One theory is that when one algotrader executed an unusually large trade, all the high-frequency algorithms tore it apart and kept selling and re-selling to each other, throwing the market into chaos.

Some people might think you are being alarmist about the risks of algorithms.
I am not saying that algorithms don't have immense value. But I think it's important to understand that something invisible is happening all around us. I don't think it's the end of the world, but I think it needs to be stated in extreme terms to make it visible.

How extreme could it get?
In the science fiction version of this, my friend Russell Davies proposed that it's 1000 years from now, there are no humans left and no companies, but computer controlled algorithms are still trading on a stock market that ran out of stock long ago.