3 июня 2026

TrOPD (arXiv 2606.01249, submitted May 31, 2026) addresses instability in on-policy distillation when teacher and student distributions diverge substantially — a common failure mode when distilling strong reasoning models into smaller students. The method combines trust-region-bounded training restricted to regions of reliable teacher supervision, clipping and masking for outlier handling, and off-policy forward-KL guidance to encourage exploration toward trustworthy areas. It consistently outperforms OPD, EOPD, and REOPOLD baselines on mathematical reasoning, code generation, and general benchmarks.

This Google/CMU paper (arXiv 2605.26099) proposes a sleep-like memory consolidation mechanism for language models. Periodically, the model converts recent context into persistent fast weights in SSM blocks through N offline recurrent passes, then clears its KV cache. On synthetic tasks (cellular automata, multi-hop graph retrieval) and math reasoning benchmarks, increasing sleep duration N improves performance, with the largest gains on examples requiring deeper multi-step reasoning.

QUBRIC (arXiv 2606.03968) addresses a structural weakness in rubric-based RLVR: open-ended queries produce vague rubrics, but narrowing queries introduces fabricated references. The method jointly refines queries and rubrics — using teacher-derived key points to convert open-ended questions into scenario-specific ones, generating contrastive rubrics based on observed policy gaps, and filtering for informative training pairs. Results show a 5.5-point improvement on ArenaHard over SFT baselines, with 6.3-point average gains on legal, moral, and narrative reasoning.

This Yale NLP paper (arXiv 2606.03969) investigates whether large reasoning models faithfully express their actual uncertainty. The authors compare linguistic confidence signals against three internal uncertainty measures: token probabilities, hidden states, and response sampling consistency. Key findings: (1) reasoning capability does not automatically improve calibration; (2) standard prompting techniques do not transfer to reasoning models; (3) different internal uncertainty measures yield conflicting results, revealing fragility in existing evaluation methodologies.