Compute scarcity is an engineering problem
About this episode
Angelos Perivolaropoulos discusses how to address GPU scarcity through innovative engineering solutions.
Angelos Perivolaropoulos, a research engineer at ElevenLabs, on turning GPU scarcity into an inference-engineering problem: how to serve far more users on the same hardware, from batching to frontier architecture changes. Recorded at RAAIS 2026. 00:00 Introduction: ElevenLabs and the GPU squeeze 00:38 The question: how to scale when you can't add capacity 01:11 About Angelos: Scribe, speech-to-text and text-to-speech 01:56 GPU scarcity meets exponential demand 02:44 What a token actually costs: compute vs memory bandwidth 03:38 Prefill, decode and the KV cache 05:53 Batching and continuous batching (1 → 15 users/GPU) 08:37 FP8 quantization and quantize-aware training (→ 20) 11:29 Speculative decoding and multi-token prediction (→ 28) 15:13 Compressing the KV cache: TurboQuant and distillation (→ 70) 17:27 Frontier architectures: MLA, linear attention, state-space (→ 140) 20:39 Trade-offs: nothing is free 22:03 Q&A: papers vs production, token subsidies, TTS evals
People in this episode
Host: Nathan Benaich
Guest: Angelos Perivolaropoulos
Topics covered
- GPU scarcity
- inference engineering
- scaling technology
- batching
- quantization
- frontier architectures
Keywords
- GPU scarcity
- inference engineering
- batching
- quantization
- frontier architectures
- compute cost
- speech-to-text
Mentioned in this episode
Organizations: ElevenLabs
More episodes of Air Street Press
- The case for intelligence beyond language
- Europe’s air defense gap
- From driving the world to dreaming it
- Turning compute into intelligence
- Accelerating science and medicine with collaborative agents
- Beyond hill climbing: the path to superhuman scientific discovery
Explore listener stats, chart rankings, contacts and more on the Air Street Press podcast page.