How to Avoid Two AI Catastrophes: Domination and Chaos (with Nora Ammann)
About this episode
Nora Ammann discusses strategies to navigate the risks of AI, focusing on slow takeoff and cooperative futures.
Nora Ammann is a technical specialist at the Advanced Research and Invention Agency in the UK. She joins the podcast to discuss how to steer a slow AI takeoff toward resilient and cooperative futures. We examine risks of rogue AI and runaway competition, and how scalable oversight, formal guarantees and secure code could support AI-enabled R&D and critical infrastructure. Nora also explains AI-supported bargaining and public goods for stability. LINKS: Nora Ammann site ARIA safeguarded AI program page AI Resilience official site Gradual Disempowerment website CHAPTERS: (00:00) Episode Preview (01:00) Slow takeoff expectations (08:13) Domination versus chaos (17:18) Human-AI coalitions vision (28:14) Scaling oversight and agents (38:45) Formal specs and guarantees (51:10) Resilience in AI era (01:02:21) Defense-favored cyber systems (01:10:37) AI-enabled bargaining and trade PRODUCED BY: https://aipodcast.ing SOCIAL LINKS: Website: https://podcast.futureoflife.org Twitter (FLI): https://x.com/FLI_org Twitter (Gus): https://x.com/gusdocker LinkedIn: https://www.linkedin.com/company/future-of-life-institute/ YouTube: https://www.youtube.com/channel/UC-rCCy3FQ-GItDimSR9lhzw/…
People in this episode
Host: Future of Life Institute
Guest: Nora Ammann
Topics covered
- AI risks
- slow AI takeoff
- scalable oversight
- formal guarantees
- AI-enabled R&D
- human-AI coalitions
- public goods
Keywords
- AI
- domination
- chaos
- oversight
- resilience
- bargaining
- public goods
Mentioned in this episode
Organizations: Advanced Research and Invention Agency, AI Resilience, Gradual Disempowerment
More episodes of Future of Life Institute Podcast
- Why AI Evaluation Science Can't Keep Up (with Carina Prunkl)
- Defense in Depth: Layered Strategies Against AI Risk (with Li-Lian Ang)
- What AI Companies Get Wrong About Curing Cancer (with Emilia Javorsky)
- AI vs Cancer - How AI Can, and Can't, Cure Cancer (by Emilia Javorsky)
- How AI Hacks Your Brain's Attachment System (with Zak Stein)
- The Case for a Global Ban on Superintelligence (with Andrea Miotti)
Explore listener stats, chart rankings, contacts and more on the Future of Life Institute Podcast podcast page.