Rethinking Critic Learning in PPO: Understanding and Mitigating Value Flattening
Shanghai AI Laboratory
The paper identifies 'Value Flattening' in PPO critics for LLM RL: true state values shift sharply between intermediate tokens while critic predictions stay flat, caused by an implicit variance penalty in the critic loss plus redundant gradients from temporally correlated states. The proposed fix, SP3O (Sparse PPO), supervises the value loss on only about three well-separated states per response and consistently improves policy learning across Qwen3-Base model sizes.
Why it matters
60 upvotes on HF Daily Papers; a clean diagnosis of a core instability in PPO-based LLM post-training with a nearly free fix, directly relevant to anyone running RLVR pipelines.
Importance: 3/5
Notable research paper on core LLM RL instability