SWE-bench Verified Is Dead. Here's What Replaced It

Every benchmark has a lifecycle. It starts clean, measures something real, and eventually dies the same way: saturation, contamination, or both. The AI engineering community learned this lesson again in July 2026 when the most-cited coding benchmark in the industry reached the end of its useful life. SWE-bench Verified—the human-validated subset of SWE-bench that OpenAI itself funded in August 2024—is no longer a reliable measure of frontier coding capability.

How We Got Here: The Birth of a Standard

SWE-bench originated as a dataset of real GitHub issues from popular Python repositories. The idea was sound: test whether an AI model could actually resolve a software engineering problem by modifying real codebases. The full dataset was too noisy for clean evaluation, so in August 2024, OpenAI funded the creation of SWE-bench Verified—a manually curated subset of 500 samples with cleaner test harnesses and betterground truth solutions.

For about eighteen months, Verified was the benchmark. When a new model claimed frontier coding capability, the first question was always: “What’s your SWE-bench Verified score?” Claude Opus 4.7 hit 87.6%. GPT-5.5 reached similar heights. Models started clustering near 90%, and the implication was clear: the benchmark was nearly solved.

But near-solve rates are a warning sign, not a celebration.

The Contamination Problem: When Solving Becomes Memorizing

The issue is deceptively simple. SWE-bench Verified has only 500 tasks. That’s a small enough number that a frontier model trained on effectively all public GitHub data will have encountered many of those exact tasks during pre-training. The model isn’t solving novel problems—it’s retrieving solutions it already memorized.

This isn’t speculation. OpenAI published its findings in February 2026 in a post titled “Why We No Longer Evaluate on SWE-bench Verified.” Their analysis found two compounding problems:

Flawed test harnesses. OpenAI audited the 500-task dataset and found that 59.4% of tasks have tests that reject correct solutions. The test infrastructure itself is broken—solutions that would genuinely fix the reported bug fail because the assertion logic is wrong. A model that appears to score 87% on Verified might actually be failing cases where it wrote the correct fix, and the benchmark is simply wrong.

Training set contamination. Every frontier model was trained on data that includes GitHub, the source of SWE-bench tasks. At the scale of modern pre-training, the probability that a model has seen a given task during training is high—especially for high-profile repositories that appear in most training corpora. The benchmark measures memorization retrieval, not genuine problem-solving ability.

The benchmark became a hall of mirrors: models performing well because they’d seen the answers, not because they could do the work.

SWE-bench Pro: The Harder Standard

The community’s response to Verified’s saturation was to build something harder. SWE-bench Pro, released by Scale AI in early 2026, evaluates coding agents on enterprise-grade software problems that don’t appear in any training set. The design constraints are stricter: no contamination from pre-training data, test harnesses that actually validate correct behavior, and a difficulty distribution that doesn’t cluster around the easy end of the scale.

The results are stark. The same models that scored 87-93% on Verified score 45-80% on Pro. On the July 2026 BenchLM leaderboard, Claude Mythos 5 leads SWE-bench Pro at 80.3%. Claude Fable 5 sits at 80%. For context, the production gap is real: models that looked like they were near-human on Verified are showing significant headroom on the harder evaluation.

The spread on Pro is also wider, which means the benchmark actually differentiates. On Verified, the top 10 models were separated by a few percentage points—statistically indistinguishable given the sample size. On Pro, you see real gaps between model generations. This is what a useful benchmark looks like: it has headroom, it differentiates, and it doesn’t reward memorization.

Why the Verified Score Was Always Wrong

There’s a second-order effect worth understanding. The Verified contamination wasn’t just a measurement problem—it actively misdirected model development. Teams optimizing for Verified were optimizing for the wrong thing. A model that scores 90% on Verified but 50% on Pro has a fundamental generalization problem that the Verified score was hiding.

This is the benchmark lifecycle at work. A benchmark starts as a clean measurement of some capability. As models improve, two things happen: the benchmark saturates (models approach ceiling), and contamination increases (models memorize the specific tasks). Both effects make the benchmark less useful precisely when it becomes most cited.

The right response is to retire benchmarks when they saturate and move to harder evaluations. But there’s institutional inertia—benchmarks become product differentiators, marketing claims, and community currency. Retiring a benchmark feels like admitting the previous generation of models wasn’t as good as advertised. The Verified-to-Pro transition required OpenAI to publicly deprecate a benchmark they had funded. That’s not a comfortable announcement for anyone.

What This Means for Evaluating Coding Agents

If you’re building AI coding agents in 2026, here’s the practical takeaway.

Don’t trust Verified scores from any model, regardless of vendor. The contamination is baked in—any model trained on public code has likely seen those 500 tasks. When a vendor claims an 88% Verified score, ask what they’re not telling you about production performance.

SWE-bench Pro is the current best signal for coding agent capability, but it’s not perfect either. A Pro score in the 70-80% range indicates strong production-capable coding ability. Scores below 50% suggest meaningful gaps. And the benchmark is young enough that contamination isn’t yet severe—but it will be in 12-18 months unless the dataset is actively maintained.

Terminal-Bench 2.0 is another useful complement. It tests agents in realistic terminal environments with multi-step tasks, rather than single-issue pull requests. GPT-5.5 scores 82.7% on Terminal-Bench 2.0 versus Claude Opus 4.7’s 69.4%—a reversal of the Verified ranking that better reflects what practitioners actually report about agentic coding workflows.

The Benchmark Lifecycle Is Accelerating

What strikes me about this cycle is its speed. SWE-bench Verified launched in August 2024. By February 2026, OpenAI had deprecated it. That’s less than two years from creation to obsolescence—a stark contrast to traditional software engineering benchmarks that remain relevant for a decade.

The acceleration comes from scale. Training data now effectively includes all public code. Benchmark tasks are a tiny sample of that space. The moment a benchmark becomes notable, it becomes part of what models are trained on. The lifecycle is now bounded by how quickly the benchmark community can create new evaluations and how fast models can saturate them.

This has implications for anyone building agent evaluation infrastructure. Static benchmarks are a losing game at frontier capability levels. The future is dynamic evaluation: synthetic task generation, adversarially constructed test cases, and continuous probing of model capabilities rather than point-in-time scores. The Verified→Pro transition is the pattern you’ll see repeated across every capability benchmark as AI improves.

For now, SWE-bench Pro is the standard. Claude Mythos 5 leads at 80.3%, Fable 5 at 80%, and the field is still pushing upward. When Pro saturates—and it will—the community will need another successor. The benchmark lifecycle doesn’t end; it accelerates.

The practical upshot: treat benchmark scores as directional signals, not precise measurements. A model at 80% on Pro versus 50% is meaningfully better at coding tasks. A model at 85% versus 83% is within noise. The difference between 90% on Verified and 50% on Pro is the real story—and it’s the story that should have been told all along.