Maximizing GPU Utilization: Heterogeneous Pipelines with Ray and Kubernetes

About this episode

Robert Nishihara discusses maximizing hardware utilization for AI and data-intensive workloads using Ray and Kubernetes.

Summary In this episode Robert Nishihara, co-founder of Anyscale and co-creator of Ray, talks about maximizing hardware utilization for AI and data-intensive workloads. He explores Ray’s evolution alongside Kubernetes and PyTorch, and why consolidation at these layers has enabled a new generation of complex, heterogeneous workloads. Robert explains how data preparation has shifted to GPU- and inference-heavy, multimodal pipelines; where Ray fits compared to Spark and workflow orchestrators; and why Ray excels at composing heterogeneous pools of compute, handling failures, and scaling complex systems like multi-node LLM inference and reinforcement learning. He digs into practical strategies for boosting GPU utilization across training and inference, elasticity and prioritization of workloads, topology-aware scheduling, and the importance of fast failure recovery as hardware scales from nodes to racks. If you’re wrestling with expensive GPUs, multimodal data curation, or cross-node LLM inference, this conversation offers concrete mental models and architectural guidance. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Your host…

People in this episode

Host: Tobias Macey

Guest: Robert Nishihara

Topics covered

• GPU utilization
• heterogeneous pipelines
• AI workloads
• data preparation
• Kubernetes
• Ray
• workflow orchestration

Keywords

• GPU
• AI
• data-intensive workloads
• Ray
• Kubernetes
• multimodal pipelines
• workflow orchestration
• failure recovery
• elasticity
• scaling

Mentioned in this episode

Organizations: Anyscale, Ray, Kubernetes, PyTorch, Spark, LLM

More episodes of Data Engineering Podcast

• Text to Data Products: Kaarvi’s End-to-End AI for Ingestion, Quality, and Dashboards · June 8, 2026 · 53 min
• Scaling Graph Analytics Without ETL: Inside PuppyGraph’s Architecture · June 1, 2026 · 54 min
• The AI-First Data Engineer: 10–50x Productivity and What Changes Next · April 7, 2026 · 59 min
• Treat Metering Like Finance: Building Data Platforms for Consumption Economics · March 29, 2026 · 50 min
• Beyond the PDF: Rowan Cockett on Reproducible, Composable Science · March 22, 2026 · 43 min
• Beyond Prompts: Practical Paths to Self‑Improving AI · March 16, 2026 · 1h 2m