About this episode

The episode discusses the Loopie series of Mixture-of-Experts models and their performance in comparison to traditional Transformers.

🤗 Upvotes: 56 | cs.CL, cs.AI Authors: Zitian Gao, Yilong Chen, Yihao Xiao, Xinyu Yang, Ran Tao, Joey Zhou, Bryan Dai Title: Loop the Loopies! Arxiv: http://arxiv.org/abs/2607.16051v2 Abstract: We present the Loopie series, consisting of two Mixture-of-Experts (MoE) models: a 20B-parameter model with 2B active parameters and a 6B-parameter model with 0.6B active parameters. Looped Transformers have long faced a challenge: given an N times increase in pre-training compute, increasing the parameter count by a factor of N usually outperforms looping a model N times. Loopie addresses this challenge. Extensive ablation studies, including comparisons with a vanilla 30B-A3B model, show that Loopie substantially outperforms vanilla Transformer baselines trained with the same compute budget. With a novel post-training method, Loopie develops strong reasoning abilities and achieves frontier-level reasoning performance.

More episodes of Daily Paper Cast

• RESOURCE2SKILL: Distilling Executable Agent Skills from Human-Created Multimodal Resources · July 21, 2026 · 19 min
• RAGU: A Multi-Step GraphRAG Engine with a Compact Domain-Adapted LLM · July 21, 2026 · 20 min
• Xiaomi-Robotics-1: Scaling Vision-Language-Action Models with over 100K Hours of Real-World Trajectories · July 21, 2026 · 21 min
• xHC: Expanded Hyper-Connections · July 20, 2026 · 21 min
• Cura 1T: Specialized Model for Agentic Healthcare · July 20, 2026 · 21 min
• On-Policy Delta Distillation · July 20, 2026 · 20 min

Explore listener stats, chart rankings, contacts and more on the Daily Paper Cast podcast page.