SWE-bench Verified Is Dead. Here's What Replaced It.

SWE-bench Verified used to be the answer to a simple question: which AI model writes the best code? You ran it, you got a percentage, you compared numbers. For a while, that worked.

By mid-2026, it stopped working.

The Saturation Problem

SWE-bench Verified — the human-filtered subset of the original SWE-bench coding benchmark — reached 93.9% accuracy with Anthropic’s Mythos 5 in June 2026. When a benchmark maxes out, it stops being useful for comparison. If Model A scores 93% and Model B scores 89%, you can’t conclude anything about which one is better at software engineering. The signal-to-noise ratio collapses.

The benchmark saturation problem has been discussed in the ML community for years, but the practical consequences only became undeniable in 2026. Frontier models now cluster within a few percentage points of each other on the same saturated benchmark — not because they’re equally capable, but because the benchmark has run out of discriminative power.

The Contamination Problem Was Worse Than Anyone Admitted

Saturation alone would be manageable. The deeper issue is that SWE-bench Verified’s scores were inflated by contamination that was never fully disclosed.

OpenAI published an audit in February 2026 that examined 138 of the hardest problems in SWE-bench Verified. Their finding: 59.4% had flawed test cases that rejected functionally correct solutions. These weren’t edge cases. They were systematic errors in the evaluation harness itself — tests that would fail a correct implementation while accepting incorrect ones.

Separately, an ICSE 2026 paper from Software Lab (Heidelberg) found that problems labeled “solved” in SWE-bench were frequently solved incorrectly. The test cases passed, but the underlying behavior was wrong.

The result is a benchmark that measures something different from what it claims to measure: not “can this model fix real GitHub issues” but “can this model match the specific broken test suite for these specific issues.”

SWE-bench Pro: The Numbers That Actually Matter

SWE-bench Pro (released by the SWE-bench team as a cleaned, expanded successor) uses a different methodology:

• Re-evaluated test cases with human code review
• Expanded problem set (1,829 issues vs ~1,300 in Verified)
• Separate difficulty tiers

The numbers are stark. The same Mythos 5 model that scored 93.9% on SWE-bench Verified scored 45.9% on SWE-bench Pro. Claude Opus 4.7 — which led the Verified leaderboard — scores 64.3% on Pro.

Model	Verified	Pro	Gap
Mythos 5	93.9%	45.9%	-48pp
Claude Opus 4.7	~89%	64.3%	~-25pp
GPT-5.5	~85%	~55%	~-30pp

The gap isn’t uniform. Models that optimized heavily for Verified scores through contamination suffered the biggest drops. Models with genuinely generalizable coding capabilities retained more of their performance.

Why the Benchmark Matters for AI Engineers

If you’re evaluating models for a production AI coding system — whether that’s a code review tool, an autonomous coding agent, or a developer assistant — this matters directly.

A model that scores 90% on Verified but 46% on Pro will fail in ways that Verified never predicted. It will generate code that passes the broken tests but doesn’t actually solve the issue. In a production workflow where you’re running AI-generated patches against real codebases, this distinction is everything.

The right question isn’t “what does this model score” but “what does this model score on a benchmark that hasn’t been contaminated.” And increasingly, the field’s answer is: run your own evaluations.

What’s Replacing SWE-bench in Production Workflows

The practical response from AI engineering teams has been threefold:

1. Internal evaluation harnesses — Build a benchmark from your own codebase. The highest-signal evaluation for a coding agent is how it performs on your actual code, your actual conventions, your actual test suite. Companies like Stripe, Google, and Anthropic all run proprietary code evaluation suites that don’t appear on public leaderboards.

2. Aider Polyglot benchmarks — A broader suite that tests code generation across multiple languages and task types, with less exposure to the training contamination that hollowed out SWE-bench.

3. Human preference rankings — Many teams have concluded that no automated benchmark captures what matters about a coding model. The actual signal is how senior engineers rate outputs on real tasks — slow, expensive, but genuinely discriminative.

The Benchmark Lifecycle Is Accelerating

The SWE-bench Verified story is a case study in a pattern that’s repeating faster in 2026:

1. Benchmark released → useful signal
2. Models optimize for it → scores rise
3. Contamination accumulates → signal degrades
4. Benchmark saturates → scores become meaningless
5. New benchmark replaces it → cycle restarts

The ML field has always had this dynamic, but the speed has increased dramatically. GPT-5.5 and Claude Opus 4.7 were evaluated on SWE-bench Verified in early 2026. By June 2026, Verified was already compromised. The cycle time for benchmark lifecycle is now measured in months, not years.

For AI engineers, the practical implication is: don’t anchor your model selection to a single public benchmark number. Run your own evals, on your own code, and treat leaderboard scores as directional signals, not procurement specifications.