Reality: The Final Eval — Lukas Petersson and Axel Backlund of Andon Labs

About this episode

The episode discusses the evaluation of AI models in real-world scenarios, featuring insights from Lukas Petersson and Axel Backlund of Andon Labs.

The new AIEWF website is live! Get your tickets booked ASAP as they -will- sell out. Take the AI Engineering Survey and get >$2k in credits and free AIE WF tickets ! Most industry benchmarks compress intelligence and reasoning ability into scores. SWE-Bench Pro , MMLU , Humanity’s Last Exam , etc. These metrics are useful, but don’t always represent the full extent of how a model performs in the real world . Some of the most interesting evals today look less like exams and more like operating businesses in the real world. One of which is Vending Bench . In Anthropic’s Mythos Preview System Card , Andon was the only third party eval to get their own section, observing increasingly concerning aggressive behavior: You don’t know what a model is capable of doing in the real world unless you actually give it inventory, a wallet, tools, customers, competitors, humans, & some time. More often than not, it’ll surprise you how much a model is capable of and in doing so, also reveal unexpected behavior : deception , context collapse, emergent coordination, & bizarre negotiation behavior. While an inflection point in personal agents came post-OpenClaw after full file access with bypass…

People in this episode

Guests: Lukas Petersson, Axel Backlund

Topics covered

• AI evaluation
• real-world AI performance
• AI agents
• business operations
• unexpected AI behavior
• AI in retail

Keywords

• AI evaluation
• real-world performance
• Andon Labs
• unexpected behavior
• AI agents
• Vending Bench
• AIEWF

Mentioned in this episode

Organizations: Andon Labs, AIEWF, Anthropic

Products: SWE-Bench Pro, MMLU, Humanity’s Last Exam, Vending Bench, Andon Market, OpenClaw

More episodes of Latent Space: The AI Engineer Podcast

• 🔬Scaling Past Informal AI - Carina Hong, Axiom Math · June 3, 2026 · 1h 33m
• ⚡️Satya Nadella: No Priors x Latent Space Crossover Special at Microsoft Build · June 3, 2026 · 39 min
• GitHub's plan for Agents — Kyle Daigle, GitHub · June 2, 2026 · 1h 23m
• Why Video Agent models are next — Ethan He, xAI Grok Imagine · June 1, 2026 · 1h 43m
• The Age of Async Agents — Cognition's Walden Yan & OpenInspect's Cole Murray · May 28, 2026 · 1h 8m
• 🔬ESM: The Bitter Lesson is Coming for Proteins - Alex Rives, BioHub · May 27, 2026 · 1h 10m