xHC: Expanded Hyper-Connections scale residual-stream parallelism past prior limits

Shanghai Jiao Tong University / Xiaohongshu / USTC / Peking University / CUHK

Research official 2 src. ~1 min

xHC extends Hyper-Connections beyond N=4 parallel residual streams by combining causal-convolution write-back augmentation with sparse residual updates (activating only 4 of 16 streams) while keeping dense read access, fixing the cubic-cost and information bottlenecks that previously blocked scaling. On 18B and 28B MoE models it improves downstream scores by 4.0 and 3.1 points over prior mHC.

Importance: 2/5

Incremental architecture-scaling research building on prior Hyper-Connections work, with measurable but modest downstream gains.

Sources