About this episode

The episode reviews objections to the AI Futures Project's AI 2040 normative plan regarding powerful AIs.

The "AI Futures Project" has released their AI 2040: Plan A scenario. While their previous scenario AI 2027 was a forecast of what they thought a future with many powerful AIs would look like, AI 2040 is intended to be normative -- it's a description of what one ought to do, granted the likelihood of a future with many powerful AIs. I'm going to review some objections I have to their proposal as a normative plan. Some are within-frame objections -- reasons that I expect trying for the AI 2040 plan that would fail to accomplish the goals of the authors. Others are my own objections -- reasons that I expect trying for the AI 2040 plan would destroy things that I, personally, care about. Before I start: two elements of the plan that I like. First, in their "incremental AI policy wishlist" -- the ideal policy that one should execute soon -- AI 2040 recommends trying to limit the gap between the intelligence of internal and external model deployments, i.e., the gap between the "intelligence" accessible to Anthropic / OpenAI employees and to everyone else. I'm a fan of efforts in this direction; equality [...] --- Outline: (02:07) 1. The Deal Has No Actual Decision Procedures (09:50)…

People in this episode

Host: 1a3orn

Topics covered

• AI policy
• normative scenarios
• powerful AIs
• objections to proposals
• surveillance
• China's perspective

Keywords

• AI 2040
• AI Futures Project
• normative plan
• objections
• surveillance
• China
• policy wishlist

Mentioned in this episode

Organizations: AI Futures Project, Anthropic, OpenAI

More episodes of LessWrong (30+ Karma)

• “What do I mean by “Artificial General Intelligence”?” by Steven Byrnes · July 21, 2026 · 7 min
• “Epistemics and Coordination: It’s complicated!” by Raymond Douglas · July 21, 2026 · 14 min
• “Differential acceleration of alignment-relevant capabilities is a bad bet” by Zephaniah Roe · July 21, 2026 · 16 min
• “Measuring Reward-Seeking via Contrastive Belief Updates” by Jérémy Scheurer, Axel Højmark, jenny, Felix Hofstätter, Theodore Ehrenborg, Bronson Schoen, Alex Meinke · July 21, 2026 · 29 min
• “11 Open Empirical Problems in Reward-Seeking” by Alex Meinke, Jérémy Scheurer, Axel Højmark, Theodore Ehrenborg · July 21, 2026 · 17 min
• “Demis Hassabis on the New Coming Age” by Zvi · July 21, 2026 · 21 min

Explore listener stats, chart rankings, contacts and more on the LessWrong (30+ Karma) podcast page.