MiniMax Sparse Attention

About this episode

This episode discusses the MiniMax Sparse Attention method for improving the efficiency of attention mechanisms in large language models.

🤗 Upvotes: 83 | cs.AI Authors: Xunhao Lai, Weiqi Xu, Yufeng Yang, Qiaorui Chen, Yang Xu, Lunbin Zeng, Xiaolong Li, Haohai Sun, Haichao Zhu, Vito Zhang, Pengyu Zhao Title: MiniMax Sparse Attention Arxiv: http://arxiv.org/abs/2606.13392v1 Abstract: Ultra-long-context capability is becoming indispensable for frontier LLMs: agentic workflows, repository-scale code reasoning, and persistent memory all require the model to jointly attend over hundreds of thousands to millions of tokens, yet the quadratic cost of softmax attention makes this untenable at deployment scale. We introduce MiniMax Sparse Attention (MSA), a blockwise sparse attention built upon Grouped Query Attention (GQA). A lightweight Index Branch scores key-value blocks and independently selects a Top-k subset for each GQA group, enabling group-specific sparse retrieval while maintaining efficient block-level execution; the Main Branch then performs exact block-sparse attention over only the selected blocks. Designed around a principle of simplicity and scalability, MSA is deliberately streamlined, making it straightforward to deploy efficiently across a broad range of GPUs. To translate sparsity into practical…

People in this episode

Hosts: Jingwen Liang, Gengyu Wang

Topics covered

• sparse attention
• machine learning
• long-context models
• GPU execution
• blockwise attention

Keywords

• MiniMax Sparse Attention
• Grouped Query Attention
• long-context capability
• GPU utilization
• block-sparse attention

Mentioned in this episode

Organizations: Arxiv

More episodes of Daily Paper Cast

• EvoArena: Tracking Memory Evolution for Robust LLM Agents in Dynamic Environments · June 13, 2026 · 24 min
• SpatialClaw: Rethinking Action Interface for Agentic Spatial Reasoning · June 13, 2026 · 23 min
• InterleaveThinker: Reinforcing Agentic Interleaved Generation · June 13, 2026 · 21 min
• FORT-Searcher: Synthesizing Shortcut-Resistant Search Tasks for Training Deep Search Agents · June 13, 2026 · 23 min
• Robust-U1: Can MLLMs Self-Recover Corrupted Visual Content for Robust Understanding? · June 13, 2026 · 20 min
• MaxProof: Scaling Mathematical Proof with Generative-Verifier RL and Population-Level Test-Time Scaling · June 13, 2026 · 24 min