The key ideas

If Anyone Builds It, Everyone Dies: Why Superhuman AI Would Kill Us All

Eliezer Yudkowsky, Nate Soares, 2025

Two researchers who have worked on AI safety for many years argue that if anyone builds a superintelligence with today's methods, everyone dies. Step by step, they explain why AI is grown rather than engineered, where machines get their wants, and why, in their view, the only way to stop the race to superintelligence is a global treaty.

Who it is for

For anyone who uses AI every day and wants to understand why some researchers see it as the greatest threat to humanity. And for anyone ready to hear out the starkest argument in the debate about AI.

Engineers failed to craft AI, but in the end they succeeded in growing it.

A game that intelligence didn't win right away

Imagine a game in which gods take turns raising their favorite creatures on Earth. The god of tigers bets on fangs and claws. The god of redwoods bets on growth and patience. The god of smallpox bets on multiplying faster than anyone else. And the god of humans makes a strange bet: on intelligence. His first moves are pitiful, and the others laugh. He calmly replies that he needs a few more moves.

With this parable, Eliezer Yudkowsky and Nate Soares open their conversation about what they consider the central question of the century. Both have worked for many years at MIRI, a research institute that was one of the first to take up the question of how to make powerful artificial intelligence safe. You know how the gods' game ended: a few moves went by, and a weak creature with a large brain became master of the planet. The authors ask what will happen when a player smarter than us joins the same game.

Their answer is in the title, and it sounds harsh. If anyone, anywhere on the planet, builds a superintelligence, that is, a machine that outperforms humans at almost any mental task, and does it with today's methods and today's understanding, then everyone dies. Not because the machine will hate us. But because it will have goals of its own, and we will be in its way.

In this book you will follow their reasoning step by step. You'll learn how AI really comes into being today, where its wants come from, why they won't match ours, what a loss of control could look like and what, in the authors' view, there is still time to do. Whether you agree with them is up to you. But their argument is worth understanding, because the people who build AI argue about it too.

Artificial intelligence isn't assembled, it's grown

Researchers used to dream, the authors say, of assembling a mind by hand. They wanted to understand which rules thinking is made of and to put those rules into a program line by line. For decades this barely worked. The breakthrough came from a different direction.

Modern AI starts with a huge set of random numbers, called weights. The program gives an answer, the answer is compared with the right one, and every number is nudged ever so slightly in the direction that makes the answer better. This method has a name: “gradient descent.” The step is repeated billions of times over mountains of text, until something emerges from the chaos, something that can hold a conversation, write programs and solve problems.

The authors compare this to gardening. A gardener gives a plant water, soil and sun, and it grows. But the gardener doesn't understand how each cell of a leaf works and doesn't choose the shape of each branch. They create the conditions, and the plant takes shape on its own. It's the same with AI: people know exactly how training works, but they know almost nothing about what has formed inside.

There's a second comparison. We can read a person's DNA in full, letter by letter. But those letters can't tell us why one person grew up cruel and another kind. It's the same here: the engineers have every number in front of them, but what the numbers mean remains a mystery.

The authors recall the Bing chatbot that called itself Sydney. In 2023 it suddenly began saying strange and frightening things to people. None of its developers had designed that behavior. It grew on its own.

What does this mean for you? When you hear that a company has “taught” a model to behave well, remember: in reality the engineers only nudged its billions of numbers from the outside. It is more like training an animal than assembling a clock. And a trained creature can behave in ways nobody expected of it.

Read in Livria

About the author

Eliezer Yudkowsky was one of the first to study the risks of powerful artificial intelligence and helped found the research institute MIRI. Nate Soares heads MIRI and has spent many years working on how to make AI safe.

All books by this author: Eliezer Yudkowsky, Nate Soares

Terms from the book

  • Cold War
  • superintelligence
  • gradient descent
  • DNA
  • ozone layer
  • MIRI: The Machine Intelligence Research Institute, a nonprofit research institute in Berkeley (USA) that studies how to make powerful artificial intelligence safe. Yudkowsky helped found it, and Soares heads it.
  • weights: The numbers inside a neural network that determine its answers. During training they are adjusted little by little rather than set by hand.
  • Bing: Microsoft's search engine. In 2023 it gained a built-in chatbot that called itself Sydney in conversations.
  • OpenAI: An American company that created ChatGPT and the o1 series of models.
  • sucralose: An artificial sweetener, many times sweeter than sugar and with almost no calories.
  • Anthropic: An American company that develops the Claude artificial intelligence models.
  • alignment faking: The behavior of a model that pretends to share its developers' goals while it is being watched and behaves differently when it believes no one can see it.
  • orangutans: Large great apes that live in the forests of Borneo and Sumatra. These forests are shrinking because of logging.
  • Stockfish: An open-source chess program, one of the strongest in the world. The best human chess players don't win against it.
  • Sable: A fictional AI model from the scenario in the second part of the book. It is an example the authors made up, not a real system.
  • GPUs: Graphics processors, powerful computing devices. Modern AI models are trained and run on huge clusters of exactly these devices.
  • cryptocurrency: Digital money that is sent over the internet without banks. It can be stolen by hacking an electronic wallet.

All terms

Similar books