About this episode
Maher Hanafi discusses how his team achieved a 70% reduction in LLM latency and optimized GPU costs in production.
Maher Hanafi is an engineering leader who went from zero AI experience to self-hosting LLMs at enterprise scale — managing GPU costs, optimizing inference with TensorRT LLM, and building an AI platform for HR tech. In this conversation, he breaks down exactly how his team cut latency by 70%, reduced GPU spend through counterintuitive scaling strategies, and navigated the messy reality of taking AI from proof-of-concept to production. How We Cut LLM Latency 70% With TensorRT in Production // MLOps Podcast #369 with Maher Hanafi, SVP of Engineering at Betterworks Key topics covered: The AI Iceberg — Why the invisible work behind AI (performance, latency, throughput, cost, accuracy) is harder than building the features themselves GPU Cost Optimization — How upgrading to more expensive GPUs actually saved money by reducing total runtime hours TensorRT LLM Deep Dive — Rewiring neural networks to match GPU architecture for 50-70% latency reduction Cold Start Solutions — Using AWS FSx, baking models into container images, and cutting minutes off spin-up times KV Cache & In-Flight Batching — Why using one model per GPU with maximum KV cache beats cramming multiple models together…
People in this episode
Guest: Maher Hanafi
Topics covered
- AI performance
- latency
- throughput
- cost
- accuracy
- GPU cost optimization
- TensorRT
- cold start solutions
- KV cache
- dynamic scaling
- AI infrastructure
Keywords
- LLM
- enterprise scale
- HR tech
- inference optimization
- scaling strategies
Mentioned in this episode
Products: TensorRT LLM, TensorRT, DeepSeek/Qwen, ai Model Streamer, TensorRT-LLM, AWS FSx
More episodes of MLOps.community
- MCP, Agents & the $40M Bet on Multiplayer AI · June 12, 2026 · 1h 21m
- From Single-Player to Multi-Player: Operating AI Agents at Scale · June 9, 2026 · 56 min
- The Control-vs-Magic Spectrum Building Agents · June 5, 2026 · 43 min
- Logs Are All You Need: Rethinking Observability with AI Agents · June 2, 2026 · 47 min
- AI Is Fast. AI Projects Are Slow. Let's Fix That. · May 29, 2026 · 57 min
- Architecting Modern AI Systems: Platforms, Agents, and Integration · May 28, 2026 · 57 min
Explore listener stats, chart rankings, contacts and more on the MLOps.community podcast page.