We Cut LLM Latency by 70% in Production
About this episode
Maher Hanafi discusses how his team achieved a 70% reduction in LLM latency and optimized GPU costs in production.
Maher Hanafi is an engineering leader who went from zero AI experience to self-hosting LLMs at enterprise scale — managing GPU costs, optimizing inference with TensorRT LLM, and building an AI platform for HR tech. In this conversation, he breaks down exactly how his team cut latency by 70%, reduced GPU spend through counterintuitive scaling strategies, and navigated the messy reality of taking AI from proof-of-concept to production. How We Cut LLM Latency 70% With TensorRT in Production // MLOps Podcast #369 with Maher Hanafi, SVP of Engineering at Betterworks Key topics covered: The AI Iceberg — Why the invisible work behind AI (performance, latency, throughput, cost, accuracy) is harder than building the features themselves GPU Cost Optimization — How upgrading to more expensive GPUs actually saved money by reducing total runtime hours TensorRT LLM Deep Dive — Rewiring neural networks to match GPU architecture for 50-70% latency reduction Cold Start Solutions — Using AWS FSx, baking models into container images, and cutting minutes off spin-up times KV Cache & In-Flight Batching — Why using one model per GPU with maximum KV cache beats cramming multiple models together…
More episodes of MLOps.community
- It's 2026, and We're Still Talking Evals
- Why Agents are Driving Software Development to the Cloud
- The Modern Software Engineer
- Getting Humans Out of the Way: How to Work with Teams of Agents
- Fixing GPU Starvation in Large-Scale Distributed Training
- Spec Driven Development, Workflows, and the Recent Coding Agent Conference
Explore listener stats, chart rankings, contacts and more on the MLOps.community podcast page.