Hidden Decoding at Scale: A New Axis for LLM Capacity Without Backbone Growth

Research official 1 src. ~1 min

Hidden Decoding (arXiv 2607.08186) expands each token into multiple independent computation streams with separate embeddings, scaling model capacity without enlarging the Transformer backbone. Stream-Factorized Attention keeps costs tractable by scoping most attention within streams. The approach is validated at 617B parameters with MoE, the first fixed-backbone scaling path demonstrated at frontier scale.

Why it matters

Opens a new practical axis for scaling LLMs beyond simply adding more layers or parameters to the backbone.

Importance: 3/5

First fixed-backbone scaling path validated at 617B frontier scale; novel architecture contribution

scaling architecture language-models llm-systems training

Sources

official Hidden Decoding at Scale