About this episode
The episode discusses the evolution and adoption of hybrid architectures in machine learning models, focusing on recent developments and historical context.
So-called hybrid architectures are far from new in open-weight models these days. We now have the recent Qwen 3.5 (previewed by Qwen3-Next ), Kimi Linear last fall (a smaller release than their flagship Kimi K2 models ), Nvidia’s Nemotron 3 Nano (with the bigger models expecting to drop soon), IBM Granite 4 , and other less notable models. This is one of those times when a research trend looks like it’s getting adopted everywhere at once (maybe the Muon optimizer too, soon?). To tell this story, we need to go back a few years to December 2023, when Mamba and Striped Hyena were taking the world by storm — asking the question: Do we need full attention in our models? These early models fizzled out, partially for the same reasons they’re hard today — tricky implementations, open-source tool problems, more headaches in training — but also because the models fell over a bit when scaled up. The hybrid models of the day weren’t quite good enough yet. These models are called hybrid because they mix these new recurrent neural network (RNN) modules with the traditional attention that made the transformer famous. They all work best with this mix of modules. The RNN layers keep part of the…
People in this episode
Host: Nathan Lambert
Topics covered
- hybrid architectures
- open-weight models
- recurrent neural networks
- transformer models
- deep learning
- machine learning trends
Keywords
- hybrid models
- RNN
- attention mechanism
- deep learning
- machine learning
- Qwen 3.5
- Nvidia
- IBM Granite 4
Mentioned in this episode
Organizations: Nvidia, IBM
Products: Qwen 3.5, Kimi Linear, Kimi K2, Nemotron 3 Nano, Granite 4, Muon optimizer, Mamba, Striped Hyena, LSTM
More episodes of Interconnects
- Claude Fable 5 and new AI safety fables · June 9, 2026 · 12 min
- Farewell Ai2 · June 2, 2026 · 16 min
- Open and closed models are on different exponentials · June 1, 2026 · 7 min
- Some ideas for what comes next, May 2026 · May 26, 2026 · 10 min
- Notes from inside China's AI labs · May 7, 2026 · 17 min
- The distillation panic · May 4, 2026 · 9 min
Explore listener stats, chart rankings, contacts and more on the Interconnects podcast page.