Episode 70: 1,400 Production AI Deployments
From Vanishing Gradients by Hugo Bowne-Anderson
February 12, 2026 · 1h 10m
About this episode
The episode discusses the challenges and risks associated with AI deployments in production, including infinite loops and silent failures.
There’s a company who spent almost $50,000 because an agent went into an infinite loop and they forgot about it for a month. It had no failures and I guess no one was monitoring these costs. It’s nice that people do write about that in the database as well. After it happened, they said: watch out for infinite loops. Watch out for cascading tool failures. Watch out for silent failures where the agent reports it has succeeded when it didn’t! We Discuss: * Why the most successful teams are ripping out and rebuilding their agent systems every few weeks as models improve, and why over-engineering now creates technical debt you can’t afford later; * The $50,000 infinite loop disaster and why “silent failures” are the biggest risk in production: agents confidently report success while spiraling into expensive mistakes; * How ELIOS built emergency voice agents with sub-400ms response times by aggressively throwing away context every few seconds, and why these extreme patterns are becoming standard practice; * Why DoorDash uses a three-tier agent architecture (manager, progress tracker, and specialists) with a persistent workspace that lets agents collaborate across hours or days; * Why…
People in this episode
Host: Hugo Bowne-Anderson
Topics covered
- AI deployments
- infinite loops
- technical debt
- agent systems
- production risks
- emergency voice agents
- continual learning
Keywords
- AI
- infinite loop
- technical debt
- agent systems
- production
- emergency voice agents
- continual learning
- DoorDash
- ELIOS
Mentioned in this episode
Organizations: DoorDash, ELIOS
More episodes of Vanishing Gradients
- The Future of Agentic Data Science · May 25, 2026 · 1h 5m
- Agent-Harness.ipynb* · May 20, 2026 · 1h 20m
- Agentic Engineering and the Lost Art of Verification · May 12, 2026 · 1h 32m
- Next Level AI Evals for 2026 · April 23, 2026 · 54 min
- Privacy Theater Is Not Privacy Engineering: What It Actually Takes to Ship Safe AI · April 15, 2026 · 1h 7m
- LLM Architecture in 2026: What You Need to Know with Sebastian Raschka · April 13, 2026 · 1h 18m
Explore listener stats, chart rankings, contacts and more on the Vanishing Gradients podcast page.