We Cut LLM Latency by 70% in Production

About this episode

Maher Hanafi discusses how his team achieved a 70% reduction in LLM latency and optimized GPU costs in production.

Maher Hanafi is an engineering leader who went from zero AI experience to self-hosting LLMs at enterprise scale — managing GPU costs, optimizing inference with TensorRT LLM, and building an AI platform for HR tech. In this conversation, he breaks down exactly how his team cut latency by 70%, reduced GPU spend through counterintuitive scaling strategies, and navigated the messy reality of taking AI from proof-of-concept to production. How We Cut LLM Latency 70% With TensorRT in Production // MLOps Podcast #369 with Maher Hanafi, SVP of Engineering at Betterworks Key topics covered: The AI Iceberg — Why the invisible work behind AI (performance, latency, throughput, cost, accuracy) is harder than building the features themselves GPU Cost Optimization — How upgrading to more expensive GPUs actually saved money by reducing total runtime hours TensorRT LLM Deep Dive — Rewiring neural networks to match GPU architecture for 50-70% latency reduction Cold Start Solutions — Using AWS FSx, baking models into container images, and cutting minutes off spin-up times KV Cache & In-Flight Batching — Why using one model per GPU with maximum KV cache beats cramming multiple models together…

People in this episode

Guest: Maher Hanafi

Topics covered

• AI performance
• latency
• throughput
• cost
• accuracy
• GPU cost optimization
• TensorRT
• cold start solutions
• KV cache
• dynamic scaling
• AI infrastructure

Keywords

• LLM
• enterprise scale
• HR tech
• inference optimization
• scaling strategies

Mentioned in this episode

Products: TensorRT LLM, TensorRT, DeepSeek/Qwen, ai Model Streamer, TensorRT-LLM, AWS FSx

More episodes of Agentic Conversations (formally mlops.community)

• The Winchester Mystery House Problem in AI Development · August 24, 2026 · 60 min
• How Predictive Analytics Stops Budget Overruns Before They Happen? · August 20, 2026 · 29 min
• How To Delegate To An Agent Like You Would An Employee? · August 17, 2026 · 55 min
• Why Your AI Bill Will Double Before It Gets Better · August 3, 2026 · 32 min
• MCP Goes Stateless · July 27, 2026 · 52 min
• AI Hype vs. Real Value · July 24, 2026 · 43 min