Does on-policy distillation really distill? Teacher-free OPSA beats it on AIME24
Purdue University
The paper shows that in on-policy distillation the teacher's token-level supervision is noisy (noise grows with teacher scale) and the student barely uses it: learning comes mainly from suppressing low-probability tokens, which needs no teacher. The authors propose OPSA, a supervision-free method with entropy-adaptive negative advantages, which on Qwen3-1.7B gains 35.41 Avg@32 points on AIME24 (+263% relative) and beats on-policy distillation by 16.77 points.
Why it matters
31 upvotes on HF Daily Papers Sep 1; if teacher signal is largely unnecessary, distillation pipelines for small reasoning models can drop the expensive teacher entirely.
Importance: 2/5
Highly upvoted HF Daily Paper (31 upvotes), official sources