We Built Something We Can’t Control | A Warning from Top AI Safety Expert - #563

About this episode

Nate Soares discusses the risks of superintelligent AI and the importance of understanding control mechanisms.

If anyone builds superintelligent AI before we know how to control it, everyone dies. Nate Soares wrote the book on why that’s not a metaphor. Subscribe if you want science with evidence, not speculation. Soares runs the Machine Intelligence Research Institute and co-wrote If Anyone Builds It, Everyone Dies with Eliezer Yudkowsky. The first word in that title is if. That matters. His argument is not that doom is certain. His argument is that the path we are on leads there, that the driver is asleep at the wheel, and that we still have time to wake him up. We argue for over an hour. I push on whether LLMs can ever reach superintelligence, whether GPU lock-in is a real ceiling, and what it would actually take to move his p-doom. He pushes back with one clean point: by the time an AI can rediscover general relativity from pre-1911 data the way Einstein did, we will have almost no time left. You don’t wait for that goalpost. What you’ll hear: Why the bus-racing-toward-a-cliff analogy depends entirely on whether the driver is asleep or awake Whether LLM lock-in is a prison or a temporary inefficiency What the AI that broke out of its virtual machine to solve a hacking problem tells us…

People in this episode

Host: Brian Keating

Guest: Nate Soares

Topics covered

• AI safety
• superintelligence
• machine learning
• control problem
• existential risk
• LLMs
• philosophy of AI

Keywords

• superintelligent AI
• control problem
• existential risk
• LLMs
• AI ethics
• Nate Soares
• Eliezer Yudkowsky

Mentioned in this episode

Organizations: Machine Intelligence Research Institute

Books & works: If Anyone Builds It, Everyone Dies

More episodes of Into the Impossible With Brian Keating

• OpenAI’s Navier–Stokes Claim: Is This What AGI Looks Like? with Emad Mostaque · September 9, 2026 · 1h 19m
• Roman Yampolskiy vs Emad Mostaque: I Was The Only Optimist - #569 · September 1, 2026 · 58 min
• The Universe Didn’t Come From Nothing | The Peter McCormack Show · August 28, 2026 · 1h 46m
• The Physics Reason AI Works When It Shouldn’t - #568 · August 25, 2026 · 1h 18m
• Engineering Chair: Most People Should Not Go To College - #567 · August 18, 2026 · 1h 0m
• John Martinis: The Nobel Physicist Behind Macroscopic Quantum Tunneling - #566 · August 11, 2026 · 49 min