Fixing GPU Starvation in Large-Scale Distributed Training
From MLOps.community by Demetrios
April 3, 2026 · 53 min
About this episode
Kashish Mittal discusses fixing GPU starvation in large-scale distributed training and shares insights on infrastructure constraints in ML scaling.
Kashish Mittal is a Staff Software Engineer at Uber, working on large-scale distributed systems and core backend infrastructure. Fixing GPU Starvation in Large-Scale Distributed Training // MLOps Podcast #367 with Kashish Mittal, Staff Software Engineer at Uber Join the Community: https://go.mlops.community/YTJoinIn Get the newsletter: https://go.mlops.community/YTNewsletter MLOps GPU Guide: https://go.mlops.community/gpuguide // Abstract Kashish zooms out to discuss a universal industry pattern: how infrastructure—specifically data loading—is almost always the hidden constraint for ML scaling. The conversation dives deep into a recent architectural war story. Kashish walks through the full-stack profiling and detective work required to solve a massive GPU starvation bottleneck. By redesigning the Petastorm caching layer to bypass CPU transformation walls and uncovering hidden distributed race conditions, his team boosted GPU utilization to 60%+ and cut training time by 80%. Kashish also shares his philosophy on the fundamental trade-offs between latency and efficiency in GPU serving. // Bio Kashish Mittal is a Staff Software Engineer at Uber, where he architects the hyperscale…
People in this episode
Host: Demetrios
Guest: Kashish Mittal
Topics covered
- GPU starvation
- large-scale distributed training
- data loading
- ML scaling
- infrastructure
Keywords
- Petastorm
- GPU utilization
- training time reduction
- latency
- efficiency
Mentioned in this episode
Products: Core Search Ranking, Petastorm
Books & works: Getting Humans Out of the Way
More episodes of MLOps.community
- MCP, Agents & the $40M Bet on Multiplayer AI · June 12, 2026 · 1h 21m
- From Single-Player to Multi-Player: Operating AI Agents at Scale · June 9, 2026 · 56 min
- The Control-vs-Magic Spectrum Building Agents · June 5, 2026 · 43 min
- Logs Are All You Need: Rethinking Observability with AI Agents · June 2, 2026 · 47 min
- AI Is Fast. AI Projects Are Slow. Let's Fix That. · May 29, 2026 · 57 min
- Architecting Modern AI Systems: Platforms, Agents, and Integration · May 28, 2026 · 57 min
Explore listener stats, chart rankings, contacts and more on the MLOps.community podcast page.