When Self-Consistency Backfires: Majority Voting Hurts Small LLMs on GPQA Diamond

Self-consistency — sample N chains of thought, return the plurality answer — is one of those techniques you reach for without thinking. It’s the default “spend more inference compute to be more reliable” lever in most agent pipelines I’ve seen. The intuition is clean: if a model is uncertain, ask it several times, and the answer most chains agree on is probably right.

A new pre-registered study on arXiv (2608.11403, August 2026) shows the intuition is wrong on hard problems, and the failure is structural, not a tuning issue. On the full GPQA Diamond benchmark — 198 graduate-level science questions across physics, chemistry, and biology — majority voting reduces per-problem accuracy on a majority of problems for two small instruction-tuned models: 56.6% of problems for Qwen2.5-7B and 65.7% for Llama-3-8B. The effect was first observed on 47 exploratory problems and then pre-registered on a 151-problem confirmatory split; all four confirmatory hypotheses passed.

This isn’t a “small models are bad” finding. It’s a finding about what voting buys you when the underlying model has correlated errors instead of independent ones.

The Mechanism Is Direct

The paper’s central argument is short enough to be worth quoting the empirical backbone of: confidence does not track correctness on these problems.

For Qwen2.5-7B, in the highest-agreement bin — the bucket where every single chain picked the same answer — the plurality answer is correct about half the time. For Llama-3-8B, the highest-agreement bin is less accurate than its lowest-agreement bin. In other words: when Llama is most sure, it’s most wrong. The voting is amplifying a signal that doesn’t exist.

This is the structural reason majority voting backfires. Self-consistency assumes chains of thought are roughly independent samples from a distribution where the right answer is the mode. On easy problems that’s roughly true — multiple chains converge on the correct reasoning. On hard graduate-level science, the chains converge on whatever the model finds most plausible, and “most plausible” is correlated across samples drawn from the same weights. Voting picks the most confident answer, not the most correct one. On problems where confidence is anti-correlated with correctness, voting is a noise amplifier.

A grid oracle that routes each problem to the best N across 64 shows the theoretical ceiling: 14 accuracy points above N=1 for Qwen and 17 for Llama. That gap is real. The question is whether any practical, verifier-free gate can recover it.

The Gates Don’t Work Either

The paper tested two natural verifier-free gates:

1. A plurality-agreement gate — only vote when the chains agree, otherwise fall back to a single sample.
2. A token-entropy gate — sample more when the model’s first-token entropy is high, less when it’s low.

Neither moves accuracy more than 0.002 from fixed-budget voting at N=64. That’s within noise.

The implication is sharp: there is no cheap test that tells you “this problem is one where voting will help.” If you commit to a fixed N for a budgeted pipeline, you’re stuck with whatever correlated bias the model has. If you try to route per-problem, you don’t have a signal to route on. The only oracle that helps is the one that requires ground truth — the grid oracle above — which by definition is not deployable.

Why This Matters For Agent Pipelines

If you’re running an agent stack that calls a small model for science or math reasoning — the kind of thing Qwen2.5-7B or Llama-3-8B are commonly used for in research assistants, code review tools, or technical Q&A — the default “sample 8 and majority-vote” wrapper is a tax you’re paying that you thought was an investment.

Three concrete implications:

1. Your “reasoning budget” knobs are misnamed. A configuration that says “sample N=8 with self-consistency” looks like it’s spending more compute to be more reliable. On hard problems it’s actually spending more compute to be more confidently wrong on a larger fraction of inputs. The wrapper gives you a number (the plurality answer) that the system can act on, but the number is calibrated to a signal — inter-sample agreement — that does not correlate with truth on the problems you most care about.

2. The verifier-free gate research direction is a dead end on this model class. If you were holding out for a clever confidence-based gate (entropy, agreement, perplexity thresholds) to make self-consistency work, the pre-registered result says: don’t. The two natural gates don’t beat fixed-budget voting by any meaningful margin. The gap between fixed-budget and oracle is real, but it requires ground truth to access.

3. Bigger isn’t the only answer, but it might be the only one. The paper explicitly does not test reasoning-native models — meaning models trained with explicit chain-of-thought reinforcement (the o-series, R1-style models, Claude’s extended thinking, Gemini’s thinking mode). The authors flag this as the central open question. The hypothesis is that reasoning-native models have a different failure mode — their chains aren’t just plausibility-weighted samples, they’re trained search traces — and majority voting might recover its standard utility there.

What I’d Actually Do With This

In a live agent pipeline, the practical move is to invert the assumption. Stop using self-consistency as your reliability knob on small models for hard problems. Replace it with three things, in order:

Use a single high-temperature sample plus a verifier. If your domain has any way to check the answer — a unit test, a calculator, a database lookup, a regex — spend your N budget there. A single sample plus a verifier beats 64 samples plus voting on problems where voting is anti-correlated with correctness, because the verifier is actually checking truth.

Reserve self-consistency for reasoning-native models, or for problems where chains are independent. If you’re on GPT-5, Claude with extended thinking, or a similar reasoning-trained model, self-consistency may still be useful — that experiment hasn’t been done yet, but the mechanism story is different. If your problem is “summarize this document” or “extract this field” where multiple chains are roughly independent draws from a similar prior, voting still helps.

Don’t trust agreement as a confidence signal. If your logging system shows “all 8 chains agreed” as a high-confidence signal in your agent’s downstream decision-making, you have a bug. Agreement among correlated samples is not evidence of correctness. Treat it as evidence of model conviction, which is a different and weaker claim.

The paper is small — two models, one benchmark, instruction-tuned only — but the mechanism is general enough that I’d extend the caution to any small-model pipeline where the underlying task has correlated error modes. GPQA Diamond is hard, but it’s not special. The same shape shows up in any domain where the model has a strong but wrong prior: legal reasoning, medical Q&A, financial modeling, code review on unfamiliar frameworks.

A Side Note On Pre-Registration

One thing worth flagging because it’s not common in ML: this paper was pre-registered. The authors observed the effect on 47 exploratory problems, then committed to four hypotheses on a held-out 151-problem split, and reported all four passing. Pre-registration is the standard in psychology and medicine, and almost nonexistent in ML benchmarking, where the norm is to run every model you can get access to and report whichever result is most publishable.

The pre-registration makes the finding more trustworthy than the usual benchmark paper, not because the methodology is exotic — it’s GPQA, the standard hard-science benchmark — but because the authors committed to the hypotheses before seeing the results on the larger split. The Llama-3-8B numbers were not the primary result; they were a corroboration from a near-chance baseline. The Qwen numbers were the test, and the test passed.

This is what we should want more of. Most LLM benchmark papers are exploratory — find a setting where your model wins, write it up. Pre-registration inverts that. If the field adopted it, a lot of “X model achieves Y on Z benchmark” papers would simply not exist, because the hypotheses wouldn’t have passed. We’d have fewer results, but more of them would be true.

Where This Leaves The Question

The paper’s central open question is whether self-consistency recovers its utility on reasoning-native models. The authors don’t speculate, but the mechanism argument suggests it should. Reasoning-native models are trained to search — their chains explore different paths by design, not by sampling noise — so the chains are more independent draws from a wider distribution, and the mode of that distribution is more likely to be correct.

That’s the next paper I’d want to read. If anyone from the same team, or anyone with access to o4 / Claude extended thinking / R1-style checkpoints and a GPQA setup, runs the same pre-registered test on those models and reports the result either way, that’s the result that tells us whether self-consistency is a tool we’re using wrong or a tool that only works on a specific class of models we haven’t deployed yet.

Until then, the default for small instruction-tuned models on hard problems is: don’t vote, verify.