Annotations as Rollouts: Efficient and Scalable Reinforcement Learning for Video MLLMs

Research official + media 3 src. ~1 min

OraRL introduces a decoupled advantage estimator for RL post-training of video MLLMs. Policy rollouts set an oracle-free baseline; the oracle-policy gap modulates a directional gain and a separate detached oracle advantage. Sign-balanced pruning keeps only the oracle and strongest rollouts per sign, so the method needs only 2.2x the step time of SFT versus 4.9x for GRPO+CoT. Scales from 0.8B to 9B and up to 100k prompts. Resulting Video-ORA-9B decodes in 130 ms without CoT (vs 4,780 ms), raises temporal mIoU from 62.5 to 66.0, tracking AO 73.0 to 78.2, segmentation 64.3 to 70.4, and scores 73.1 on VSI-Bench versus 55.0 for GPT-5 and 55.1 for Gemini-3-Pro.

Why it matters

HF Daily 61 upvotes on Aug 26. Reframes the role of human annotations as oracle rollouts in on-policy groups, sidestepping the 'advantage inversion' failure mode. Demonstrates a practical path to RL post-training for video MLLMs without chain-of-thought, with frontier-beating results on VSI-Bench.

Importance: 3/5

HF Daily 61 upvotes

video mllm reinforcement-learning rlhf rlvr annotation post-training

Sources

official arXiv abstract

official Project page

media HF Daily Papers page