This is a comprehensive summary of “Human Compatible” by Stuart Russell. The book explores artificial intelligence and the problem of control.

what’s in it for me? rethink your fundamental assumptions about ai.#

Introduction

stuart russell, human compatible, artificial intelligence and the problem of control.

narrated by arian stanley and morag sims.

artificial intelligence will be the defining technology of the future.

already ai is rapidly pervading all levels of society.

councils willfully bring ai into their homes to help them organise their daily lives.

city councils and corporations employ ai to help optimise their services.

and states take advantage of ai to undertake large-scale surveillance and social engineering campaigns.

but as ai becomes more intelligent, and our social systems become more and more to depend on it, the threat presented by out-of-control ai becomes more dire.

the risks and downsides of new technologies are far too often left unexplored, as scientists and engineers fixate on their feverish quest to realise the utopias of the future.

in fact, many ai experts and corporate high-ups even downplay the risks of ai out of fear of being more strictly regulated.

these chapters attempt to remedy this imbalance.

the question of how to control ai and mitigate its more disastrous consequences is the big question facing humanity today.

and it's precisely this question we'll explore.

we need several breakthroughs in software before ai surpasses human intelligence.#

chapter 1 of 6 today's computers can process information at astounding speeds.

but even as early as the 1950s, computers were being touted as super-fast brains that are faster than einstein.

of course, computers back then had nothing on the human brain.

but we still compared the two.

in fact, from the very beginning of computer science, we've tended to measure computational intelligence and progress against human intelligence.

so what about today's computers?

some of them surely can give us a run for our money.

the key message here is, we need several breakthroughs in software before ai surpasses human intelligence.

the fastest computer in the world today is the summit machine, housed at the oak ridge national laboratory in the us.

compared to the world's first commercial computer, the ferranti mark 1, the summit machine is 1,000 trillion times faster and has 250 trillion times more memory.

that's a lot of zeros.

in terms of raw computing power, the summit machine actually slightly exceeds the human brain, although it requires a warehouse full of hardware and a million times more energy.

still, it's impressive.

but can we say that today's supercomputers, the summit machine included, are as powerful as the human brain?

the answer is decidedly no.

sure, these computers have impressive hardware, which allows their algorithms to operate faster and process more information, but there's far more to intelligence than just processing speed.

the real problem in designing intelligence is in the software.

as of now, we still need several major conceptual breakthroughs in ai software before we witness anything resembling human-level artificial intelligence.

the most important breakthrough we need is in the comprehension of language.

most of today's so-called intelligent speech recognition ai are based on canned responses and have trouble interpreting nuances in meaning.

that's why you get stories of smartphone personal assistants responding to the request of, call me an ambulance, with, ok, from now on i'll call you an, a-n-n, ambulance.

genuinely intelligent ai will need to interpret meaning based not just on the words said, but on their context and tone as well.

we can never really say when conceptual breakthroughs will take place, but one thing's for sure, we shouldn't underestimate human ingenuity.

consider the following example.

in 1933, the distinguished nuclear physicist ernest rutherford announced at a formal address that harnessing nuclear energy was impossible.

the very next day, the hungarian physicist leo szilárd outlined the neutron-induced nuclear chain reaction, essentially solving the problem.

we don't yet know whether super-intelligence, intelligence beyond human abilities, will emerge soon, later, or not at all, but it's still prudent to take precautions, just as it was when designing nuclear technology.

we’ve been operating under a misguided conception of intelligence.#

chapter 2 of 6 if we don't treat ai with caution, we may end up like the gorilla.

just consider that, thanks to human-caused habitat loss, every gorilla species today is critically endangered.

sure, in recent decades, conservation efforts have successfully pulled some species back from the brink of extinction.

but whether gorilla numbers dwindle or thrive, their fate depends largely on the whims of humans.

the concern is that in a world controlled by super-intelligent ai, we'd be in much the same position as the gorillas.

can humans maintain supremacy and autonomy in a world where they rate second place to more intelligent beings?

thankfully, there's one important difference between us and the gorillas.

we're the ones designing this new intelligence.

it's paramount that we take great caution in how we design intelligent ai if we're to ensure they remain under our control.

but we have a crucial problem.

the key message here is, we've been operating under a misguided conception of intelligence.

in the current paradigm of ai design, an ai's intelligence is measured simply by how well it can achieve a pre-given objective.

the big flaw in this approach is that it's extremely difficult to specify objectives that will make an ai behave the way we want it to.

pretty much any objective we come up with is liable to produce unpredictable and potentially very harmful behaviour.

this problem is known as the king midas problem, named after the fabled king who wished that everything he touched would turn to gold.

what he didn't realise was that this included the food he ate and even his own family members.

this ancient tale is a perfect example of how a poorly specified objective can end up causing more strife than good.

the danger from unpredictable behaviour increases as ai becomes more intelligent and wields greater power.

the consequences could even present an existential threat to humanity.

for example, we might ask a super-intelligent ai to find a cure for cancer, only for it to start giving people cancer in order to do experiments on them.

you might be wondering, if we're not happy with what an ai is doing, why don't we just turn the blasted thing off?

unfortunately, for the vast majority of objectives, an ai would have an incentive not to allow itself to be turned off.

that's because being turned off would threaten its objective.

even apparently straightforward objectives like make a coffee would lead an ai to prevent itself from being turned off.

after all, you can't make coffee if you're dead.

instead of just intelligent machines, we should be designing beneficial machines.#

chapter 3 of 6 until now, the mantra among ai researchers has been, the more intelligent, the better.

but is this really what they should be chanting?

as we've just seen, an ai given a carelessly stated objective can end up engaging in very harmful behaviour.

and it's not much consolation if the ai engages in harmful behaviour intelligently.

if anything, it makes it worse.

what we need is a different mantra, one that aims to build ai that will stay tuned to human objectives, no matter what.

the new mantra should be, the more beneficial, the better.

the key message here is, instead of just intelligent machines, we should be designing beneficial machines.

there are three principles that designers should follow if they're to make beneficial ai.

the first principle is that ai should only have one objective, which is the maximal fulfilment of human preferences.

the author calls this the altruism principle.

it ensures that an ai will always place human preferences above its own.

the second principle is that the ai should initially be uncertain about what those preferences are.

this is the humbleness principle.

the idea here is that an uncertain ai will never fixate on a single objective, but will change its focus as new information comes in.

uncertain ai systems would be more cautious and more likely to defer to humans.

being uncertain, an ai would continuously search for clarifying information.

this means they'll often ask for permission, solicit feedback, and might even do trial runs to test human reactions.

and crucially, uncertain ai will allow themselves to be turned off.

this is because it would interpret a human trying to turn it off as having the preference that it be turned off.

the third and final principle for making beneficial ai is that their ultimate source of information about human preferences should be human behaviour.

this is called the learning principle.

it ensures that an ai will always remain in a direct and sustained relationship of learning with humans.

it means an ai will become more useful to a person over time as it gets to know her better.

these three principles represent an alternative understanding of what genuine intelligence involves.

this is the ability to scrutinise and redefine one's own objectives in the light of new information.

ai with this kind of intelligence would be much closer to human intelligence, since we're also capable of examining and altering the goals that we strive towards.

and if ai could change its objectives in light of human preferences, we would have the basis for a radical new relationship between humans and machines.

one in which machine and human objectives are essentially the same.

we can expect ai to benefit us in many ways.#

chapter 4 of 6 it's been reported that some virtual home assistants, unable to differentiate who's talking to them, have been obeying commands to buy products that they've overheard on the television.

clearly virtual assistants aren't quite super intelligent yet.

but this is going to change.

virtual assistant technology is improving in leaps and bounds thanks in part to massive investment from the private sector.

the reason for such great interest in this technology is that the range of tasks virtual assistants could perform is seemingly unlimited.

we're not just talking about making shopping lists and turning the stereo on.

we're talking about performing the work of highly skilled specialists.

the key message here is, we can expect ai to benefit us in many ways.

virtual lawyers, for example, are already vastly outperforming real lawyers at sourcing legal information quickly.

similarly, virtual doctors are outperforming human doctors at providing correct diagnoses for illnesses.

eventually, there may be no need for specialists at all.

instead, we'll all have our own personal, all-in-one doctor, lawyer, teacher, financial advisor and secretary in our pockets, on call, 24 hours a day.

thanks to virtual assistants, these vital services will be democratised, no longer accessible only to the rich, thereby raising the standard of living for everybody.

the benefit of ai to scientific research will also be colossal.

an ai with even basic reading comprehension skills would be able to read everything the human race has ever written between breakfast and lunchtime.

by comparison, it would take 200,000 humans reading full-time just to keep up with the world's current level of publication.

with the help of super-intelligent ai, scientists will no longer have to sort through immense amounts of published research, as ai will be able to extract and analyse the relevant data for them.

super-intelligent ai will also have global applications.

by collecting information from surveillance cameras and satellites, we should expect ai to be used to create a searchable database of the entire world in real time.

from this data, we could produce models of global systems, such as economic activity and environmental changes.

these models would make it feasible to design effective interventions into these systems, helping us to, for example, mitigate the effects of climate change.

however, the potential for privacy violation implied by a system of global monitoring is obvious.

this leads us to our next chapter, which tackles the darker side of ai that we will all need to brace for.

ai is going to make life less secure for everyone.#

chapter 5 of 6 the stasi of former east germany were one of the most effective and repressive intelligence agencies ever to have existed.

they kept files on the majority of east german households, listening to their phone calls, reading their letters, and even placing hidden cameras within their homes.

this was all done by humans and written on paper, requiring a vast bureaucracy and massive storage units containing literally billions of physical paper records.

just imagine what the stasi could have done with ai.

with super-intelligent ai, it would be possible to monitor everyone's phone calls and messages automatically.

people's daily movements could also be tracked, using surveillance cameras and satellite data.

it would be as though every person had their own operative watching over them 24 hours a day.

the key message here is, ai is going to make life less secure for everyone.

ai could lead to yet other dystopias.

this includes the infopocalypse, the catastrophic failure of the marketplace of ideas to produce the truth.

super-intelligent ai will be capable of manufacturing and distributing false information without any human input.

they'll also be able to target specific individuals, altering their information diet strategically to manipulate their behaviour with surgical accuracy.

to a large extent, this is already happening.

content selection algorithms used by social media platforms, ostensibly designed to predict people's preferences, end up changing those preferences by providing them only a narrow selection of content.

in practice, this means users are pushed to become more and more extreme in their political views.

arguably, even these rudimentary forms of artificial intelligence have already caused great harm, entrenching social division and proliferating hate.

while the infopocalypse is still at its infant stage, the next dystopia is well in the making.

this is the state of constant fear caused by autonomous weapons technology.

autonomous weapons, machines that seek out and neutralise targets by themselves, have already been developed.

such weapons identify targets based on information like skin colour, uniform or even exact face prints.

miniature drones called slaughterbots are already being primed to search for, locate and neutralise specific individuals.

in 2016, the us air force demonstrated the deployment of 103 slaughterbot drones.

they described the drones as a single organism sharing one distributed brain, like a swarm in nature.

the us is only one of many nations currently building, or already using, automated weapon technology.

as autonomous weapons come to displace conventional human warfare, all of our lives will become less secure since anyone will be targetable, no matter where they are in the world.

mass automation will either liberate humanity’s potential or debilitate it.#

chapter 6 of 6 we've just looked at three terrifying scenarios that could be caused by ai.

but we haven't yet considered what is perhaps the most worrying and socially destructive threat from ai.

automation.

automation may well grant more people access to important services like healthcare and legal advice.

but it could also cause widespread unemployment.

exactly how much of a threat this is, is open to debate.

optimists point out that in every previous industrial revolution, automation produced at least as many jobs as it eradicated.

if you look at the research, however, it shows that over the past 40 years, jobs have fallen significantly in every industry that has implemented automation technology.

so should we be worried?

the key message here is, mass automation will either liberate humanity's potential or debilitate it.

the truth is, in the long run, ai is likely to automate away almost all existing human labour.

this will not only affect low-skilled work like truck driving.

as we saw earlier, even highly skilled professionals like doctors, lawyers and accountants are at risk.

so when a machine replaces your labour, what will you have left to sell?

well, not much.

but what if you didn't need to sell anything?

what if we could let machines do all the work and still make sure we all had enough to live by?

one way to do this might be to institute a universal basic income, or ubi.

a ubi would provide every adult person a reasonable monthly income, regardless of circumstance.

those who want to earn more would be free to work, if there's any available.

but for everyone else, liberated from the need to earn a living, they would be free to pursue whatever they want.

this is a rosy picture, isn't it?

but would this scenario really be a utopia?

if all labour learning and striving towards skill acquisition were taken over by machines, wouldn't we become diminished beings?

this is a genuine concern.

until now, the only way to sustain our civilization has been to pass knowledge from one human to another and over successive generations.

as technology develops, we increasingly hand that knowledge and expertise over to a machine that can do the task for us.

once we lose the practical incentive to pass knowledge onto other humans, that knowledge and expertise will wither.

if we're not careful, we could become an enfeebled species, utterly dependent on the machines that ostensibly serve us.

you've just listened to our chapters to human compatible by stuart russell.

final summary#

Conclusion

the key message in these chapters is that the way we currently design ai is fundamentally flawed.

we're designing ai to be intelligent, but not necessarily to have humanity's best interests at heart.

we therefore need to make the fulfilment of human goals ai's only objective.

if we can successfully control superintelligent ai, we'd be able to harness its immense power to advance our civilization and liberate humanity from servitude.

but if we fail, we're in danger of losing our autonomy as we become increasingly subject to the whims of a superior intelligence.

Discover more book summaries in the same category or by the same author.

The Odyssey

Society & Culture22 min read