It's 2026, and We're Still Talking Evals

About this episode

Maggie Konstanty discusses the challenges and realities of evaluating AI agents in production environments.

Maggie Konstanty is an AI Product Manager at Prosus , one of the world's largest consumer internet companies, where she builds and evaluates AI agents for food ordering and ecommerce at scale. She's been inside the messy reality of LLM evaluation longer than most — and her take is unfiltered. It's 2026, and We're Still Talking Evals // MLOps Podcast #372 with Maggie Konstanty, AI Product Manager at Prosus 🧪 Why accuracy metrics lie — Maggie breaks down why "95% accurate" tells you almost nothing about whether your agent is actually working in the real world, and what to measure instead. 🏗️ Pre-ship vs. production evals — Your eval suite before launch will not survive first contact with real users. Maggie explains the structural disconnect and how to close the gap. 👻 The silent failure: user drop-off — Users who are unhappy don't complain — they just leave. Discover why drop-off analytics are one of the most underutilized eval signals in production. 🎯 Instruction to fail: the 20-evaluator trap — Setting up 20 types of evaluators not connected to your product goal is a fast path to wasted time. How to design evals that are tied to real outcomes. 🍽️ The "surprise…

People in this episode

Guest: Maggie Konstanty

Topics covered

• AI evaluation
• LLM evaluation
• user experience
• product management

Keywords

• accuracy metrics
• user drop-off
• evaluation design
• food ordering
• ecommerce

Mentioned in this episode

Products: AI agents, food ordering systems, ecommerce platforms

Books & works: We're Still Talking Evals

More episodes of Agentic Conversations (formally mlops.community)

• The Winchester Mystery House Problem in AI Development · August 24, 2026 · 60 min
• How Predictive Analytics Stops Budget Overruns Before They Happen? · August 20, 2026 · 29 min
• How To Delegate To An Agent Like You Would An Employee? · August 17, 2026 · 55 min
• Why Your AI Bill Will Double Before It Gets Better · August 3, 2026 · 32 min
• MCP Goes Stateless · July 27, 2026 · 52 min
• AI Hype vs. Real Value · July 24, 2026 · 43 min