Schrödinger's Code Repository: have LLMs learned SWE-bench or memorized it?
Shanghai Jiao Tong University
SJTU's SchrodingerRepo treats the SWE-bench test repository as an evaluation-time variable, applying four transformation levels (problem-statement reconstruction, namespace remapping, intra-file reordering, functionality-preserving rewriting) to SWE-bench Verified and SWE-QA. Stripping familiar naming/layout cues consistently drops Pass@1 by 6.0-14.4 points, with 81.6-83.6% of the extra agent effort going into re-exploring the repository, evidence that part of current SWE-bench performance reflects memorized surface cues rather than repo understanding.
Why it matters
A direct contamination-style audit of the most-cited coding-agent benchmark; the finding reshapes how coding-agent numbers should be read.
Importance: 3/5
High-impact contamination audit of the most-cited coding benchmark