About this episode

This episode discusses the concept of Expanded Hyper-Connections in Transformers and its implications for memory scaling and model performance.

🤗 Upvotes: 47 | cs.LG, cs.CL Authors: Xiangdong Zhang, Xiaohan Qin, Sunan Zou, Tuo Dai, Xiaoming Shi, Huaijin Wu, Yebin Yang, Zhuo Xia, Shaofeng Zhang, Lin Yao, Yuliang Liu, Yu Cheng, Junchi Yan Title: xHC: Expanded Hyper-Connections Arxiv: http://arxiv.org/abs/2607.14530v1 Abstract: Hyper-Connections (HC) expand the residual stream of Transformers into $N$ parallel streams, providing a form of memory scaling beyond model width and depth. Manifold-Constrained HC (mHC) stabilizes this formulation at scale. The large gains from $N{=}1$ to $N{=}4$ suggest residual-stream expansion as a promising scaling axis. However, existing HC-family methods typically stop at $N{=}4$. Our experiments reveal why: scaling mHC beyond this point yields diminishing performance gains and rapidly increasing training cost. We attribute this limitation to two bottlenecks: insufficient write-back information for an expanding number of streams and residual-mixing generation whose cost scales cubically with $N$. To address both bottlenecks, we propose xHC (Expanded Hyper-Connections), the first HC-family method to achieve meaningful expansion beyond $N{=}4$. xHC combines temporal feature augmentation for…

More episodes of Daily Paper Cast

• RESOURCE2SKILL: Distilling Executable Agent Skills from Human-Created Multimodal Resources · July 21, 2026 · 19 min
• RAGU: A Multi-Step GraphRAG Engine with a Compact Domain-Adapted LLM · July 21, 2026 · 20 min
• Xiaomi-Robotics-1: Scaling Vision-Language-Action Models with over 100K Hours of Real-World Trajectories · July 21, 2026 · 21 min
• Loop the Loopies! · July 20, 2026 · 20 min
• Cura 1T: Specialized Model for Agentic Healthcare · July 20, 2026 · 21 min
• On-Policy Delta Distillation · July 20, 2026 · 20 min

Explore listener stats, chart rankings, contacts and more on the Daily Paper Cast podcast page.