DRPO: Rethinking Divergence Regularization in LLM Reinforcement Learning
Tencent Hunyuan
DRPO (Divergence Regularized Policy Optimization, arXiv:2606.09821) replaces the hard gradient-masking used in PPO/DPPO with a smooth advantage-weighted quadratic regularizer. Instead of discarding updates when a token crosses trust-region boundaries, DRPO applies bounded, continuous gradient weights that both attenuate harmful divergences and supply corrective signals. Validated across multiple model scales, architectures, and precision settings, showing improved stability and efficiency over existing LLM RL training methods.
Why it matters
With 324 upvotes on HuggingFace Daily Papers — highest for June 10 — this paper directly addresses a fundamental instability in RLVR training pipelines powering reasoning models like DeepSeek-R1 and Qwen3. A smoother trust-region control mechanism could improve reliability of post-training runs industry-wide.
Importance: 4/5
Top HF Daily Papers June 10 (324 upvotes); addresses core RLVR instability; practical impact on post-training pipelines for reasoning models. +1 for upvotes ≥100.