The Mask Is Not the Model: Auditing Prefix Invariance in Attention, State-Space, and Hybrid Sequence Models
Formalizes prefix invariance and introduces a lightweight two-forward-pass audit with no training or gradients, producing a per-layer score that pinpoints where causality breaks. Across 192 injected-fault trials on eight checkpoints, mask inspection detected zero, while the proposed audit localized all 192 to the exact layer. Static and dynamic analysis of chunked-scan code in transformers surfaced the same defect in Zamba2 and Nemotron-H, traced to an inter-chunk axis error corrected via the reference implementation.
Why it matters
HF Daily 29 upvotes on Aug 25. The audit found real causality defects in shipped hybrid models that the field's standard check (attention mask inspection) missed. This is a practical mechanism-interp tool that turns a property-level invariant into a per-layer diagnostic.
Importance: 3/5
HF Daily 29 upvotes + bugs found in published models