Super Weights in LLMs and the Failure of Selective Training
Amazon Web Services
This paper challenges the hypothesis that 'super weights' — a small set of parameters whose removal catastrophically degrades model performance — can be selectively trained to update model behavior. Targeting even 100–8,192 such parameters in isolation causes accuracy to collapse to random-guessing on OLMo-1B and OLMo-7B, while training an equal number of random parameters in the same layers actually improves performance. Accepted at COLM 2026.
Why it matters
Structural importance does not equal trainability, which means model editing and targeted unlearning techniques that rely on super-weight identity may be fundamentally limited.
Importance: 2/5
Challenges core assumption in model editing/unlearning; COLM 2026 accepted paper
Sources
official
arXiv paper 2607.08733