Tencent Hunyuan open-sources SAS sparse-attention routers trained end-to-end on Qwen3
Tencent Hunyuan
Tencent's Hunyuan team released Simple Attention Sparsification (SAS), a gated sparse-attention method that learns to rank and select KV blocks via continuous gates optimized end-to-end with the language-modeling loss, instead of distilling dense attention scores. Router-only checkpoints (33-42M gate parameters) for Qwen3-4B/8B/14B went up on Hugging Face on September 14, alongside an SGLang-compatible seer_attn backend and paper arXiv 2609.13141.
Why it matters
Hard Top-K selection being non-differentiable is the core problem of trainable sparse attention; SAS aligns context ranking directly with prediction impact under a fixed attention budget, and the frozen-backbone router design makes it a cheap post-training add-on for already-deployed Qwen3 models.
Importance: 3/5
Clean fix to the non-differentiability problem of trainable sparse attention, with open checkpoints