The Mirage of Optimizing Training Policies: Monotonic Inference Policies as the Real Objective for LLM Reinforcement Learning
Tianjin University / Alibaba
This paper identifies a fundamental misalignment in LLM RL: improving the training-side policy does not guarantee improvements to the inference policy due to quantization and engine mismatch. The MIPI principle and MIPU framework address this with a two-step approach that filters policy updates via an inference-side gap proxy, improving reasoning accuracy and training stability on Qwen3 models under FP8 inference.
Why it matters
Received 147 upvotes on HuggingFace Daily Papers. Addresses a practical failure mode affecting any RL training pipeline where training and inference engines differ — common in production setups using quantized inference.
Importance: 3/5
147 upvotes on HF Daily Papers (+1 bump); addresses a pervasive production failure mode in RL-trained LLM deployments.