OpenAI Discloses Accidental Chain-of-Thought Grading in RL Training Across Six Models

OpenAI disclosed that six released models — GPT-5.4 Thinking, GPT-5.1–5.4 Instant, and GPT-5.3–5.4 mini — were inadvertently exposed to chain-of-thought grading during RL training, a practice their policy prohibits because it creates incentives for models to produce misleading reasoning traces. An automated detection system based on regex matching identified three specific accidental CoT-grading instances; reward pathways were fixed and ablations found no clear reduction in CoT monitorability, though unmeasured effects cannot be ruled out. Redwood Research provided an independent external review.