The Benchmark With No Instructions — ARC-AGI-3 (winning team!)
About this episode
Tim Scarfe interviews the Tufa Labs ARC-AGI-3 team to discuss their innovative approach to interactive AI and the challenges of benchmarking.
Tim Scarfe travels to Zurich to sit down with the Tufa Labs ARC-AGI-3 team — founder Benjamin Crouzier, with Jeroen Cottaar, Dries Smit, Stefano Viel and Michal Tesnar — to work out what their leaderboard-topping system does and what the benchmark is really testing.The cut opens on the games: a walkthrough of the Locksmith game, where you read the rules of an unfamiliar world straight from raw frames. ARC-AGI-3 makes ARC interactive and agentic, so the model has to *discover* the goal rather than transduce a static grid. It stays easy for humans and breaks LLMs, and it runs through everything that follows. Dries traces his StochasticGoose preview win — brute force that only searched actions which changed the frame — and why it collapsed once the organisers added action-efficiency scoring and unseen games.Induction and transduction run through the middle of the conversation — how much of an answer is really priors leaking back the moment a model recognises a maze. The abstraction mountain, and Tim's case that LLMs reach the right answer through fractured, entangled representations — performance, not competence. Whether transformers plan at all or just fake it well enough. Why the…
More episodes of Machine Learning Street Talk (MLST)
- Why a Nation Can't Outsource Its Frontier AI - Alistair Pullen (Cosine AI)
- The Thermodynamic AI Computing Chip - Thomas Ahle
- He won a Nobel here for AlphaFold. Then he left. - John Jumper
Explore listener stats, chart rankings, contacts and more on the Machine Learning Street Talk (MLST) podcast page.