The Pre-Training Wall and the Treadmill After It
From CoRecursive: Coding Stories by Adam Gordon Bell - Software Developer
May 9, 2026 · 56 min · Episode 121
About this episode
The episode discusses the concept of the pre-training wall in AI and its implications for future developments in machine learning.
I've been confusing Don with frontier-lab links late at night for a bit. Ilya Sutskever told a NeurIPS audience that pre-training as we know it would unquestionably end. There's only one internet, and the data isn't growing. The frontier labs call this the pre-training wall. A leaked Google memo from 2023 argued they had no moat. R1 is on GitHub. Llama is on Hugging Face. OpenAI's secondary-market valuation has climbed past $850 billion. Don was confused. So he came over and we made an episode about it. Episode Page Support The Show Subscribe To The Podcast Join The Newsletter
People in this episode
Host: Adam Gordon Bell
Guest: Don
Topics covered
- AI development
- pre-training
- machine learning
- data limitations
- technology news
- Google memo
- OpenAI valuation
Keywords
- pre-training wall
- Ilya Sutskever
- Google memo
- OpenAI valuation
- machine learning
- data growth
- AI technology
Mentioned in this episode
Organizations: Google, OpenAI, GitHub, Hugging Face
More episodes of CoRecursive: Coding Stories
- Story: The Aging Programmer · April 2, 2026 · 42 min
- From Hacker News to TikTok - How Algorithms Learned to Hook Us · March 2, 2026 · 42 min
- Notes: The Universal Paperclip Clicker · February 4, 2026 · 11 min
- Story: Inside Early Google - Race Conditions, Java Pain, and the Birth of AdWords · January 2, 2026 · 38 min
- Story: The Bug He Couldn't Name - A 15-Year Fight Inside One Developer's Mind · December 2, 2025 · 44 min
- Story: Godbolt's Rule - When Abstractions Fail · November 4, 2025 · 44 min
Explore listener stats, chart rankings, contacts and more on the CoRecursive: Coding Stories podcast page.