About this episode
Maggie Konstanty discusses the challenges and realities of evaluating AI agents in production environments.
Maggie Konstanty is an AI Product Manager at Prosus , one of the world's largest consumer internet companies, where she builds and evaluates AI agents for food ordering and ecommerce at scale. She's been inside the messy reality of LLM evaluation longer than most — and her take is unfiltered. It's 2026, and We're Still Talking Evals // MLOps Podcast #372 with Maggie Konstanty, AI Product Manager at Prosus 🧪 Why accuracy metrics lie — Maggie breaks down why "95% accurate" tells you almost nothing about whether your agent is actually working in the real world, and what to measure instead. 🏗️ Pre-ship vs. production evals — Your eval suite before launch will not survive first contact with real users. Maggie explains the structural disconnect and how to close the gap. 👻 The silent failure: user drop-off — Users who are unhappy don't complain — they just leave. Discover why drop-off analytics are one of the most underutilized eval signals in production. 🎯 Instruction to fail: the 20-evaluator trap — Setting up 20 types of evaluators not connected to your product goal is a fast path to wasted time. How to design evals that are tied to real outcomes. 🍽️ The "surprise…
People in this episode
Guest: Maggie Konstanty
Topics covered
- AI evaluation
- LLM evaluation
- user experience
- product management
Keywords
- accuracy metrics
- user drop-off
- evaluation design
- food ordering
- ecommerce
Mentioned in this episode
Products: AI agents, food ordering systems, ecommerce platforms
Books & works: We're Still Talking Evals
More episodes of MLOps.community
- MCP, Agents & the $40M Bet on Multiplayer AI · June 12, 2026 · 1h 21m
- From Single-Player to Multi-Player: Operating AI Agents at Scale · June 9, 2026 · 56 min
- The Control-vs-Magic Spectrum Building Agents · June 5, 2026 · 43 min
- Logs Are All You Need: Rethinking Observability with AI Agents · June 2, 2026 · 47 min
- AI Is Fast. AI Projects Are Slow. Let's Fix That. · May 29, 2026 · 57 min
- Architecting Modern AI Systems: Platforms, Agents, and Integration · May 28, 2026 · 57 min
Explore listener stats, chart rankings, contacts and more on the MLOps.community podcast page.