xHC: Expanded Hyper-Connections scale residual-stream parallelism past prior limits
Shanghai Jiao Tong University / Xiaohongshu / USTC / Peking University / CUHK
xHC extends Hyper-Connections beyond N=4 parallel residual streams by combining causal-convolution write-back augmentation with sparse residual updates (activating only 4 of 16 streams) while keeping dense read access, fixing the cubic-cost and information bottlenecks that previously blocked scaling. On 18B and 28B MoE models it improves downstream scores by 4.0 and 3.1 points over prior mHC.
Importance: 2/5
Incremental architecture-scaling research building on prior Hyper-Connections work, with measurable but modest downstream gains.
Sources
official
xHC on HF Daily Papers
official
xHC arXiv preprint