Weak-to-Strong Generalization via Direct On-Policy Distillation

ByteDance / Tsinghua University

Research official 1 src. ~1 min

Direct-OPD transfers RL gains from smaller models to larger ones by treating the weak model's RL-induced log-ratio shift as a dense implicit reward signal for the student. Applied to Qwen3-1.7B, the method raises AIME 2024 accuracy from 48.3% to 62.4% in four hours of training without a separate reward model.

Why it matters

Offers a compute-efficient route to scale reasoning improvements from small models to large ones — relevant as RL-for-reasoning compute costs grow with model size.

Importance: 2/5

Efficient weak-to-strong reasoning transfer; 4-hour training yielding +14pp AIME accuracy improvement.

rl weak-to-strong reasoning distillation post-training

Sources

official arXiv:2607.05394 — Weak-to-Strong Generalization via Direct On-Policy Distillation