SMELT: looped MoE transformers match baseline scaling at matched compute

Published September 1 (arXiv 2609.01343) by ByteDance Seed authors, the paper studies layer looping in sparse MoE transformers while strictly matching per-token FLOPs, non-embedding parameters and KV cache against an unlooped baseline. The resulting recipe, SMELT, loops the middle half of layers twice and was scaled across four sizes up to 54B non-embedding parameters with a Chinchilla-style scaling law fitted per architecture; it saves 6.8-18% of training FLOPs on the compute-optimal frontier, and mechanistic analysis shows the second loop pass reduces attention-sink mass and redirects it to content-relevant tokens.