About this episode
This episode explores how to benchmark AI models and the complexities involved in comparing their performance.
How do you know if a new AI model is actually better than the last one? It turns out answering that question is a lot messier than it sounds. This week we dig into the world of LLM benchmarks — the standardized tests used to compare models — exploring two canonical examples: MMLU, a 14,000-question multiple choice gauntlet spanning medicine, law, and philosophy, and SWE-bench, which throws real GitHub bugs at models to see if they can fix them. Along the way: Goodhart's Law, data contamination, canary strings, and why acing a test isn't always the same as being smart.
People in this episode
Host: Katie Malone
Topics covered
- AI models
- benchmarking
- LLM benchmarks
- Goodhart's Law
- data contamination
- canary strings
Keywords
- AI models
- benchmarking
- MMLU
- SWE-bench
- Goodhart's Law
- data contamination
- canary strings
Mentioned in this episode
Organizations: MMLU, SWE-bench, GitHub
More episodes of Linear Digressions
- Many Agents, Many Problems (The Agents Season, Episode 8) · June 8, 2026 · 28 min
- How Do You Evaluate An AI Agent? (The Agents Season, Episode 7) · June 1, 2026 · 32 min
- AI Agent Failure Modes (The Agents Season, Episode 6) · May 25, 2026 · 33 min
- Agentic Planning (The Agents Season, Episode 5) · May 18, 2026 · 24 min
- Memory Management for AI Agents (The Agents Season, Episode 4) · May 10, 2026 · 25 min
- Lost in the Middle (The Agents Season, Episode 3) · May 4, 2026 · 20 min
Explore listener stats, chart rankings, contacts and more on the Linear Digressions podcast page.