“OpenAI’s red line for AI self-improvement is fundamentally flawed” by Charbel-Raphaël

About this episode

Charbel-Raphaël critiques OpenAI's thresholds for AI self-improvement, arguing they are too permissive and flawed.

Epistemic status: could have been a short form. Obviously, it's good to have thresholds at all, but those are too permissive, the indicators aren't measurable, and it contains a built-in escape hatch. 1. Too permissive The Preparedness Framework v2 defines the Critical threshold for AI Self-improvement as: “either: (leading indicator) a superhuman research-scientist agent OR (lagging indicator) causing a generational model improvement (e.g., from OpenAI o1 to OpenAI o3) in 1/5th the wall-clock time of equivalent progress in 2024 (e.g., sped up to just 4 weeks) sustainably for several months. [...] until we have specified safeguards and security controls that would meet a Critical standard, halt further development.(By default, I would expect not to stop at 5x and to go quickly at 10x, 20x, … if we reach this point.)” Both halves fire too late. The leading indicator only triggers once a model can already do AI research above the best humans. That's not early enough to act on, and we can basically ignore it. The real meat is in the lagging indicator, which requires 5x generational acceleration sustained for several months. If we are charitable, by interpreting several as 6 months…

People in this episode

Guest: Charbel-Raphaël

Topics covered

• AI self-improvement
• safeguards
• indicators
• Preparedness Framework
• critical thresholds

Keywords

• AI
• self-improvement
• OpenAI
• Preparedness Framework
• indicators
• safeguards
• critical threshold

Mentioned in this episode

Organizations: OpenAI

More episodes of LessWrong (30+ Karma)

• “The Uncertainty That Matters Isn’t Fundamental” by jimmy · June 13, 2026 · 22 min
• [Linkpost] “US government directive to suspend access to Fable 5 and Mythos 5” by Capybasilisk · June 13, 2026 · 0 min
• “Claude Fable 5 and Mythos 5: The System Card” by Zvi · June 12, 2026 · 60 min
• “Citations Needed: Magic Encyclopedias to Save the World” by Oliver Sourbut · June 12, 2026 · 10 min
• “Simulating Simulators” by kromem · June 12, 2026 · 27 min
• “Implications of Continual Learning for LLM Agents: Introduction” by RohanS, Rauno Arike, Owen Terry, Achu Menon, Zhijing Jin, Francis Rhys Ward, Seth Herd · June 12, 2026 · 14 min