In April 2026, Claude Mythos 5 scored 93.9% on SWE-bench Verified. The AI community celebrated. Tech blogs ran the headline. Anthropic cited it in their release notes. The benchmark, it seemed, had been conquered.
Three months earlier, OpenAI had published a post titled “Why SWE-bench Verified No Longer Measures Frontier Coding Capabilities.” The post documented an audit of the benchmark’s test cases. Of the problems they examined, 59% had flawed test designs that rejected correct solutions. Another 16.4% showed evidence of training data contamination. The benchmark OpenAI had themselves helped create was, in their words, “saturated and highly contaminated.”
The post got some attention. Then the Mythos numbers dropped. And the celebration resumed.
The Gap That Should Concern You
The more interesting number lives on SWE-bench Pro — a harder, private test set designed to close the contamination loophole. On that benchmark, the same Claude Mythos 5 that scored 93.9% on Verified drops to 45.9%. Some reports put it as low as 23% depending on the evaluation harness.
The pattern is consistent across vendors. Models that dominate Verified collapse on Pro. OpenAI’s own analysis documented this: when you strip out contaminated problems and use a standardized evaluation harness, the scores don’t just drop — they reveal an entirely different ranking.
Here’s the breakdown as of mid-June 2026:
| Model | SWE-bench Verified | SWE-bench Pro |
|---|---|---|
| Claude Mythos 5 | 93.9% | 45.9% |
| Claude Opus 4.8 | ~87.6% | ~47% |
| GPT-5.5 | 88.7% | ~23% |
| Claude Opus 4.5 | 45.9% | 23.4% |
The model that “wins” on Verified doesn’t even win on Pro. The benchmarks are measuring different things.
Why This Happens: Test Design and Contamination
SWE-bench Verified was designed to be clean. Five hundred real GitHub issues, human-filtered, with test cases that should pass correct solutions. But “human-filtered” doesn’t mean “not in training data.” As models grew more capable and training sets grew more comprehensive, the boundary between “evaluation set” and “training set” dissolved.
OpenAI’s audit found two distinct problems:
Flawed test cases. A test case is supposed to accept correct code and reject incorrect code. But 59% of audited failures on Verified had test design bugs — the test rejected correct solutions, or accepted incorrect ones. When your measurement tool is broken, improving the thing being measured doesn’t help.
Contamination. Test cases and solutions from Verified appeared in the training data of frontier models. A model that has literally memorized the answer to a problem isn’t demonstrating coding ability — it’s demonstrating retrieval. This is especially insidious because contamination is invisible in the score: a contaminated model gets a high number, just like a genuinely capable model.
The Benchmark Arms Race
The response from the research community has been predictable: new benchmarks, harder benchmarks, private benchmarks. SWE-bench Pro uses held-out problems that vendors can’t memorize. LiveBench attempts to continuously refresh its test set. Epoch AI maintains a private evaluation track that labs can’t see in advance.
Each generation of benchmarks starts at low scores — every model gets near zero on a fresh test set. Then, over 12-18 months, vendors optimize for it. Contamination creeps in. The benchmark saturates. A new one launches.
This is not unique to SWE-bench. It happened with ImageNet for computer vision. It happened with GLUE/SuperGLUE for NLP. The lifecycle of a benchmark is: novel → saturates → dies. SWE-bench Verified is simply further along that curve than most, because the economic incentive to game it is enormous.
What the Gap Means for Agent Evaluation
If you’re building a system that uses AI coding agents — whether that’s a code review tool, an autonomous coding agent, or a CI pipeline that delegates to a model — these numbers matter directly.
A 94% Verified score tells you: this model can handle the kinds of problems that appear in public training data and have well-structured test cases. That’s a real capability. But a 46% Pro score tells you: when you give it harder, less familiar problems with stricter evaluation criteria, performance roughly halves.
The models aren’t failing because they’re bad. They’re failing because benchmarks measure what you can optimize for, not what you actually need. A coding agent that scores 94% on Verified but 46% on Pro is a very different product from one that scores 94% on both. The Verified number alone doesn’t tell you which you’re buying.
This is why production failure rates for AI coding agents remain high even as benchmark scores climb. Teams deploy models that “ace” Verified, then find they still need a human reviewing 40-60% of the agent’s outputs. The benchmark was clean. Production is not.
What Better Evaluation Looks Like
The most honest evaluation frameworks being built right now combine three signals:
Private test sets — problems held out from training data, evaluated against a standardized harness. SWE-bench Pro and Epoch AI’s private track are attempts at this. The scores are lower, but they’re more predictive of production behavior.
Human expert comparison — not just automated test passes, but whether a senior engineer would prefer the model’s solution to what they would have written. Some teams run blind studies: give a problem to a human and an agent, anonymize the solutions, ask experts to pick. The results are humbling.
Long-horizon task completion — most benchmarks are single-problem, single-answer. Real engineering work spans days: requirements shift, edge cases emerge, code interacts with other systems. Evaluating at the task level, not the patch level, is where the real capability gaps show.
The Honest Score
Benchmarks are not useless. They’re a necessary compression of complex capability into a single number. But a single number is always a lie, and the lie grows bigger as the benchmark saturates.
SWE-bench Verified at 94% is the lie. SWE-bench Pro at 46% is the truth. Neither number alone tells you what you need to know.
The models have gotten genuinely better. The benchmarks have just gotten better at being gamed.
If you’re evaluating AI coding agents for production use, ask vendors for their Pro scores. Ask for private evaluation results. Ask what their human-expert comparison studies show. The teams that know their actual capability will share those numbers. The teams that are selling a benchmark score will deflect.
SWE-bench Verified isn’t dead. It’s just no longer telling you what you think it’s telling you.