evaluation — AI Digest

15 мая EVA-Bench: End-to-End Framework for Evaluating Voice Agents ServiceNow AI research
18 мая SOOHAK: Frontier LLMs Solve Hard Math But Fail to Recognize Unsolvable Problems research
19 июн OpenAI Publishes Deployment Simulation: Predicting Model Behavior Before Release OpenAI research
7 июл GigaWorld-1: A Roadmap to Build World Models for Robot Policy Evaluation GigaAI research
13 июл Long-Horizon-Terminal-Bench: Testing Agent Limits on Long-Horizon Terminal Tasks Tencent Hunyuan research
8 июл LLM-as-a-Verifier: verification as an independent scaling axis for LLMs Stanford University / UC Berkeley / NVIDIA research
18 мая Judge Circuits: Mechanistic Explanation of LLM-as-Judge Format Inconsistency research
14 июн EvoArena: LLM Agents Score Only 40% on Dynamic Evolving Environments MIT / NUS / Salesforce research
14 июн WeaveBench: Computer-Use Agents Fail at Hybrid GUI+CLI Tasks — 41% Pass Rate Microsoft Research research
3 июл PerceptionRubrics: Atomic Rubric Evaluation Reveals 8% Perception Gap Between Open and Closed Models research
8 июл AutomationBench-AA: 657-task independent benchmark for AI agent SaaS automation Artificial Analysis tools
1 мая AutoResearchBench — a benchmark for autonomous scientific literature search by AI agents BAAI research
12 июн EvoArena: LLM Agents Score Only 39.6% on Dynamic Evolving Environments Benchmark MIT research
21 июн FAPO: Fully Autonomous Prompt Optimization of Multi-Step LLM Pipelines Cisco Foundation AI research
6 июл AgenticSTS: Bounded-Memory Testbed for Long-Horizon LLM Agents Alaya Studio research
6 июл EvoPolicyGym: Evaluating Iterative RL Policy Self-Improvement by Coding Agents University of Macau / CUHK research
7 июл SWE-Together: Multi-Turn Benchmark for Coding Agent Evaluation research
10 июл Ideas Have Genomes: Frontier LLMs Score Only 27% on Scientific Lineage Reasoning Shanghai Jiao Tong University research
27 июл K12-KGraph: A Curriculum-Aligned Knowledge Graph for Benchmarking and Training Educational LLMs research
19 июн Multimodal Evaluator Preference Collapse: Cross-Modal Contagion in Self-Evolving Agent Loops research
30 июн Formalizing Latent Thoughts: Axiomatic Framework for Evaluating LLM Reasoning Representations University of British Columbia research
4 июл Will Scaling Improve Social Simulation with LLMs? A Study of 85 Models Stanford / Columbia / Tsinghua research
11 июл UniClawBench: Benchmark for Proactive AI Agents on Real-World Tasks University of Hong Kong research