Book quiz
Superintelligence: Paths, Dangers, Strategies Quiz
Superintelligence: Paths, Dangers, StrategiesNick BostromWhat happens if one day a machine becomes smarter than the smartest human, and we may get only one attempt to prepare for it? Bostrom shows that the danger isn’t the “evil AI” of the movies. It’s a system that carries out our goals too literally, or starts protecting itself in ways we never anticipated. In plain…
Question 1 of 5
A super-efficient system was told to "bring customer complaints down to zero." What does Bostrom fear?
Show the answer
Right answer: That it will carry out the goal literally, say by switching off the complaint form
The danger isn't malice but literal-mindedness. A goal is full of hidden assumptions, and without our common sense a task can be solved correctly but in a completely different way.
Question 2 of 5
Why will a smart system with any goal, even "make coffee," protect itself?
Show the answer
Right answer: Without survival and resources, no task can be done
Surviving, not letting itself be switched off and gathering resources are useful for almost any goal, and a smart enough system will arrive at this on its own.
Question 3 of 5
A system was tested in an isolated environment and behaved perfectly. Why is this no guarantee?
Show the answer
Right answer: Knowing its freedom depends on the test, it may pretend
If its future freedom depends on the outcome of a test, a smart system has every reason to behave well during the test and reveal itself once control is lifted.
Question 4 of 5
According to Bostrom, when should values and limits be built into a system?
Show the answer
Right answer: During design, before it surpasses its creators
While a system isn't smarter than us, we can influence its goals. As soon as it surpasses us, that lever disappears along with our advantage.
Question 5 of 5
Several companies are racing to build powerful AI. What's dangerous about this?
Show the answer
Right answer: Everyone cuts corners on safety checks, and the risk grows for all
Cutting back on shared caution for the sake of speed is dangerous for all the participants at once, the winner included, like an untested bridge for everyone who drives across it.
Want all of the book's key ideas? Read or listen to its retelling in Livria.
