Random Attention: rethinking KV cache eviction for efficient reasoning
Salesforce AI Research
A provocative negative result: sophisticated importance scoring for KV cache eviction is largely unnecessary. Keeping the prompt intact and evicting reasoning tokens uniformly at random within each attention head matches the strongest prior evictor across four models and six reasoning tasks, while serving 32-43% higher throughput in vLLM. The reasoning trace protects itself — the model restates needed information in text, and each head keeps its own copy.
Why it matters
158 upvotes on HF Daily Papers. Reframes a whole line of KV compression work: prior selectors' gains came mostly from accidentally preserving the prompt, not from clever scoring — and random eviction is free.
Importance: 4/5
Notable paper + 158 upvotes on HF Daily Papers