About this episode
This episode discusses vLLM, a high-throughput serving system for large language models that improves memory management and increases throughput.
vLLM is a high-throughput serving system for large language models. It addresses inefficient KV cache memory management in existing systems caused by fragmentation and lack of sharing, which limits batch size. vLLM uses PagedAttention, inspired by OS paging, to manage KV cache in non-contiguous blocks. This minimizes memory waste and enables flexible sharing, allowing vLLM to batch significantly more requests. As a result, vLLM achieves 2-4x higher throughput compared to state-of-the-art systems like FasterTransformer and Orca.
Topics covered
- large language models
- memory management
- high-throughput systems
- batch processing
- KV cache
Keywords
- vLLM
- high-throughput
- KV cache
- PagedAttention
- memory waste
- batch size
- FasterTransformer
- Orca
Mentioned in this episode
Organizations: vLLM, FasterTransformer, Orca
More episodes of Large Language Model (LLM) Talk
- Context Engineering · January 21, 2026 · 14 min
- Manus AI · January 19, 2026 · 17 min
- Kimi K2 · July 22, 2025 · 16 min
- Mixture-of-Recursions (MoR) · July 18, 2025 · 17 min
- MeanFlow · July 10, 2025 · 7 min
- Mamba · July 10, 2025 · 8 min
Explore listener stats, chart rankings, contacts and more on the Large Language Model (LLM) Talk podcast page.