Qwen team details the Qwen3.8-Next architecture: hybrid attention, n-gram embeddings, Muon
Qwen (Alibaba)
The design paper behind Qwen3.8-Flash-Next (125B total, 6B active) publishes full ablations: Gated DeltaNet linear-attention layers with one full-attention layer per four, Qwen Sparse Attention swapped in at continued pre-training, a four-branch gated residual stream, and 51B of n-gram embeddings stored off-accelerator. It beats its 397B-A17B predecessor on 8 of 14 pre-training benchmarks using ~1/9 the training FLOPs, and shows loss can improve while downstream accuracy saturates.
Why it matters
A rare fully-ablated architecture disclosure from a frontier open-weights lab; it documents the design of the architecture previewed by Qwen3.8-Flash-Next and likely carried into Qwen4.
Importance: 3/5
Full architecture disclosure from a frontier open-weights lab, official sources