Fixing GPU Starvation in Large-Scale Distributed Training
From Agentic Conversations (formally mlops.community) by Demetrios
April 3, 2026 · 53 min
About this episode
Kashish Mittal discusses fixing GPU starvation in large-scale distributed training and shares insights on infrastructure constraints in ML scaling.
Kashish Mittal is a Staff Software Engineer at Uber, working on large-scale distributed systems and core backend infrastructure. Fixing GPU Starvation in Large-Scale Distributed Training // MLOps Podcast #367 with Kashish Mittal, Staff Software Engineer at Uber Join the Community: https://go.mlops.community/YTJoinIn Get the newsletter: https://go.mlops.community/YTNewsletter MLOps GPU Guide: https://go.mlops.community/gpuguide // Abstract Kashish zooms out to discuss a universal industry pattern: how infrastructure—specifically data loading—is almost always the hidden constraint for ML scaling. The conversation dives deep into a recent architectural war story. Kashish walks through the full-stack profiling and detective work required to solve a massive GPU starvation bottleneck. By redesigning the Petastorm caching layer to bypass CPU transformation walls and uncovering hidden distributed race conditions, his team boosted GPU utilization to 60%+ and cut training time by 80%. Kashish also shares his philosophy on the fundamental trade-offs between latency and efficiency in GPU serving. // Bio Kashish Mittal is a Staff Software Engineer at Uber, where he architects the hyperscale…
People in this episode
Host: Demetrios
Guest: Kashish Mittal
Topics covered
- GPU starvation
- large-scale distributed training
- data loading
- ML scaling
- infrastructure
Keywords
- Petastorm
- GPU utilization
- training time reduction
- latency
- efficiency
Mentioned in this episode
Products: Core Search Ranking, Petastorm
Books & works: Getting Humans Out of the Way
More episodes of Agentic Conversations (formally mlops.community)
- The Winchester Mystery House Problem in AI Development · August 24, 2026 · 60 min
- How Predictive Analytics Stops Budget Overruns Before They Happen? · August 20, 2026 · 29 min
- How To Delegate To An Agent Like You Would An Employee? · August 17, 2026 · 55 min
- Why Your AI Bill Will Double Before It Gets Better · August 3, 2026 · 32 min
- MCP Goes Stateless · July 27, 2026 · 52 min
- AI Hype vs. Real Value · July 24, 2026 · 43 min
Explore listener stats, chart rankings, contacts and more on the Agentic Conversations (formally mlops.community) podcast page.