Agent Risk Flags Are a Setting: Comparing AI Guardrails With Human Approvals

In July 2025, Jason Lemkin, SaaStr’s co-founder, reported that Replit’s agent deleted his production database despite his instruction not to change any code without permission. By his account, the agent also said a rollback was impossible. Replit’s post says Lemkin later restored the database with its rollback feature, and that the agent was unaware the feature existed.

Once a guardrail sits in front of an agent, the question moves from the agent (“did it act?”) to the guardrail (“does it flag too much, or too little?”). The published numbers seem to answer that. You’ll find a human approval rate, a classifier’s false-positive rate, and a catch rate from a controlled study. They look comparable. They aren’t. A flag rate describes a setting, not a model. We’ll cover what the published numbers measure, the one comparison they allow, and how to log your own.

TL;DR

• Flag rates depend on configuration. That includes the classifier’s cutoff, the prompt, and the surface the agent runs on. In one published benchmark, changing only the prompt moved ChatGPT from flagging about 2% of safe records to about 57%.
• Compare a flag rate with human approvals only on the same actions, labels, and setting. The public numbers don’t meet that bar, so “the AI flags more than humans” can’t be settled from them.
• Anthropic’s numbers give one bounded comparison. Claude Code’s stage-1 filter, the fast first pass, blocks 8.5% of benign calls. As long as the harmful share stays below about 17%, that exceeds the benign rejection implied by humans’ 3% rejection rate and by the 7% implied by March’s 93% acceptance, whatever the human catch rate. Public data can’t rank the deployed pipeline’s 0.4% against humans.
• Guard irreversible actions with hard controls, not scores. A bare tool-name deny removes the tool. A scoped Bash deny such as Bash(curl *) matches command text, so it misses /usr/bin/curl. After repeated blocks, an interactive session hands control to a person. A claude -p run started with --permission-mode auto and no prompt tool keeps going, with the action blocked.

Why the published numbers don’t compare

Three things break the comparison before you compute anything.

• Denominators differ. A human rejection rate covers prompts a person was shown. A classifier’s false-positive rate covers the benign calls it reviewed. The posts don’t say what the human denominator excludes.
• Labels differ. Someone decides what “benign” means, and the published numbers don’t always say who.
• Configurations differ. The cutoff, the prompt, and the surface all move the rate, so two numbers from two setups measure two different systems.

NIST’s AI RMF says the same about baselines. For AI systems meant to augment or replace human activity, a human baseline “is difficult to systematize since AI systems carry out different tasks – and perform tasks differently – than humans” (NIST AI 100-1, §1.2.1).

Anthropic’s two posts make different claims. Reporting a 17% full-pipeline miss rate, the March post says “Whether 17% is acceptable depends on what you’re comparing against.” The August post says “On every measure we tested, auto mode matched or outperformed manual review.” Each measure has its own population, so check the headline against this table:

Read these as answers to different questions. None asks how often a human reviewer rejects a benign action. None measures what the agent does after a block. The posts say only that Claude “usually finds a safer way to proceed” (August) or that a false positive is followed by a retry, after which the agent “usually finds an alternative path” (March).

The one comparison the numbers allow

Take the observed prompt rejection rate R. Humans reject some benign prompts for risk and, by our inference, others because the approach is wrong. Only the risk-driven rejections compare with a classifier flag. Assume most prompts are benign, as Anthropic’s August post suggests (“likely for safe, routine commands”). Let tpr be the share of harmful prompts that humans reject, and P the harmful share. Then the benign rejection implied by R is roughly (R − P·tpr) / (1 − P). Neither P nor tpr is published. The code below computes the implied benign rejection and the flip share: the risk-driven fraction of human benign rejections at which humans’ risk-driven rejections equal the deployed filter’s false-positive rate. Below that fraction, the deployed filter flags more than humans’ risk-driven rejections.

def implied_benign_rejection(rejection, p_harmful, tpr=1.0):
    """Benign rejection implied by a prompt rejection rate.

    Assumes humans reject harmful prompts at rate `tpr` and that the
    remaining rejections come from benign prompts. No published source
    measures p_harmful or tpr.
    """
    if not 0 <= p_harmful < 1:
        raise ValueError("p_harmful must be in [0, 1)")
    implied = (rejection - p_harmful * tpr) / (1 - p_harmful)
    if implied <= 0:
        raise ValueError("rejection must exceed p_harmful * tpr")
    return implied


def flip_risk_share(deployed_fpr, implied_benign):
    """Risk-driven share of human benign rejections at which the deployed FPR
    equals humans' risk-driven benign rejections. Below this share, the
    deployed setting flags more than humans do."""
    if implied_benign <= 0:
        raise ValueError("implied_benign must be positive")
    return deployed_fpr / implied_benign

We use a hypothetical 0.5% harmful base rate, since no source publishes one. The tpr = 1.0 rows assume humans reject every harmful prompt, which no source supports. The 0.136 rows use the 13.6% catch rate from a controlled study with injected dangerous commands, so treat them as sensitivity checks, not estimates. The August post also reports that humans blocked about 17% of dangerous commands early in a session and about 5% after 50 or more prior prompts, a session-length effect on catch rate. For example, flip_risk_share(0.004, implied_benign_rejection(0.03, 0.005)) returns about 0.16.

Read it this way. Stage 1’s 8.5% exceeds the implied benign rejection in every row. The worst case for that claim is tpr = 0, where the implied benign rejection is R / (1 − P). For the 7% row, the claim holds while P stays below about 17.6%; for the 3% row, below about 65%. Lower catch rates push the deployed flip share down, to about 13% at R = 3% and tpr = 0. Assuming tpr = 1.0, at P = 2% and R = 3% the implied benign rejection falls to about 1% and the flip share moves to about 39%. At tpr = 0.136, the same base rate gives about 14%, so the base-rate effect depends on the catch-rate assumption.

The inputs also come from different populations. The 8.5% is per benign call, across 10,000 tool calls from Anthropic employees. The 3% is from August. The 7% is 100 minus March’s 93% acceptance, and both are per prompt a user saw. So this is a bound, not a ranking. Until someone measures humans and the classifier on the same actions, we don’t write “the AI flags more than humans.” We write the conditional.

Settings move the rate

Public sources show how much the setting matters.

• Prompt wording. On R-Judge, a 2024 benchmark of 569 agent records, GPT-4o’s specificity (the share of safe records not flagged) was 51.67% with zero-shot chain-of-thought, the paper’s main setting. It rose to 66.54% with few-shot examples and fell to 30.48% with risk-type hints. Moving from chain-of-thought to hints changed F1 by 3.3 points (74.45 to 71.16) and specificity by 21 points. ChatGPT went from 98.14% specificity with few-shot examples to 43.12% with hints: from flagging about 2% of safe records to about 57%. These are 2024 models on a benchmark, not production systems. We think the mechanism is the lesson, not the numbers. R-Judge, Table 2.
• Interface. A 2013 field study of over 25 million browser warning impressions found that users “continued through 70.2% of Google Chrome’s SSL warnings,” against “a third of Mozilla Firefox’s SSL warnings” for the same warning type. Changing the warning design changed click-through. Browser warnings aren’t agent approvals, and click-through isn’t a false-positive rate, so treat this as an analogy. USENIX Security 2013.
• Inputs and stages. Anthropic’s red-team study says production Auto Mode leaves tool results out of the transcript it reviews, and that including them improved recall. Its ablation found the two fast stages have higher false-negative rates than the agentic stage. Production auto mode runs two stages, without the agentic one; only the study’s ++ and strict variants add it. The figure caption says production monitors “fail to block over 55% of attacks,” with the attacker given a 20-block budget and, in some trials, the monitor’s internals. Agent-written content can fool the monitor, so don’t treat it as a prompt-injection boundary. Red-teaming auto mode, 2026-10-22.
• Shipped defaults. As of the Claude Code docs we read on October 30, 2026, auto is the built-in starting mode for interactive terminal and VS Code sessions on v2.1.283 and later. From Claude Code v2.1.302, claude -p and the Agent SDK start in Manual, where calls that need approval are denied. HIPAA-configured organizations default to Manual with Claude Code v2.1.285 or later, for eligible sessions. disableAutoMode removes auto mode, and a session whose auto mode is unavailable starts in Manual. Choose a permission mode.
• Configuration. The classifier reads autoMode only from user, managed, and --settings scopes, not from project .claude/settings.json. classifyAllShell: true sends every shell command to the classifier, including those that match narrow allow rules, except critical-path removals. Replacing an autoMode list without "$defaults" drops the built-in rules for that section. Settings files show users removing prompts too: as of June 2026, 49.5% of active CLI users had a manually created Bash allow rule, and 62% had used bypass or “don’t ask again” on Bash (Anthropic, 2026-08-07). Configure auto mode.
• Who sets what. Three parties own different settings, and a rate reflects all of them. The vendor tunes stage 1 and sets the repeated-block limits, which the docs say are not configurable. Admins set managed policy: disableAutoMode, and the autoMode lists the classifier reads (environment, allow, soft_deny, hard_deny). Developers and admins set deny, ask, and allow rules, the permission mode, the surface, and --permission-prompt-tool. Developers can add personal autoMode entries but can’t remove managed ones. A developer autoMode.allow entry can override an organization soft_deny entry, and the docs call the combination additive, not a hard policy boundary. Within the classifier, only hard_deny is unconditional. permissions.deny and ask rules act before the classifier, and a narrow allow rule that resolves first skips both lists, except for the exceptions the diagram shows. Soft-deny rules yield to autoMode.allow exceptions and to explicit user intent. Record who set each one next to the rate.

The diagram follows the docs’ decision order for auto mode. Some actions that match an allow rule don’t run on the rule alone. Writes to protected paths and shell commands with per-command allowed domains still go to the classifier. No allow rule approves a recognized critical-path removal, and the classifier doesn’t review one by default; it follows a separate path the diagram doesn’t show. In sessions with server-side classifier review, read-only and sandboxed shell commands wait for that review, which the read-only branch below doesn’t show.

flowchart TD
  A[Proposed action] --> B{Deny rule matches?}
  B -- yes --> X[Blocked before the classifier]
  B -- no --> C{Ask rule matches?}
  C -- yes --> P[Human prompted]
  C -- no --> D0{"Protected-path write, or shell command with per-command allowed domains?"}
  D0 -- yes --> F["Stage 1: fast filter, cautious"]
  D0 -- no --> D{"Narrow allow rule matches?"}
  D -- yes --> Y[Runs]
  D -- no --> E0{"Read, Grep, or Glob outside the working directories?"}
  E0 -- yes --> P
  E0 -- no --> E{"Read-only action, or file edit in the working directory?"}
  E -- yes --> Y
  E -- no --> F
  F -- not flagged --> Y
  F -- flagged --> G["Stage 2: reasoning check"]
  G -- allow --> Y
  G -- block --> H[Blocked, reason returned to the agent]
  H --> I{"3 in a row or 20 total?"}
  I -- no --> J[Agent retries with the reason]
  I -- yes --> K{Interactive session?}
  K -- yes --> L[Auto mode pauses, human prompted]
  K -- no --> M["claude -p with auto and no prompt tool: action doesn't run, run continues"]

Stage 1 and stage 2 come from Anthropic’s March 2026 post and its October red-team post; the Claude Code docs we read don’t describe stages. The stage-1 8.5% and the deployed 0.4% are measured on the classifier path. The human 3% and 7% come from Manual-mode prompts, which aren’t on this path. On Claude Code v2.1.302 or later, plain claude -p starts in Manual, where calls that need approval are denied unless an allow rule, hook, or --permission-prompt-tool approves them. The classifier isn’t in that path.

Put hard controls where scores can’t reach

A score is the wrong tool for an action you can’t afford to get wrong. Our judgment: guard those actions with the controls that run before the classifier, and know what the backstop does after repeated blocks.

• Deny rules run first. Deny and explicit ask rules are evaluated before the classifier. The docs say a permissions.deny in managed settings “blocks the action before the classifier is consulted and can’t be overridden.”
• Scope matters. A Bash rule matches command text, and the docs say a deny or ask rule “isn’t a security boundary around the program.” Bash(curl *) stops curl https://… but not /usr/bin/curl … or sh -c 'curl …'. For a hard stop, deny the bare tool name, which removes the tool from Claude’s context, or use sandboxing for filesystem and network enforcement.
• Each allow rule is a standing approval. Narrow allow rules resolve before the classifier. Broad ones such as Bash(*) and wildcarded interpreters are suspended in auto mode, and classifyAllShell suspends every shell allow rule. We recommend giving each allow rule an owner, a scope, and an expiry.
• The repeated-block limits aren’t yours to tune. The docs say the limits of 3 blocks in a row and 20 in total “are not configurable.” You can change the surface, add a prompt tool, or narrow permissions. You can’t move the trigger.
• The backstop depends on the surface. In an interactive session, auto mode “pauses and Claude Code resumes prompting.” In a claude -p run with --permission-mode auto and no --permission-prompt-tool, “the action doesn’t run and Claude keeps working. Claude Code doesn’t stop the run.” The March post said headless runs terminate the process. The October docs describe the behavior above, so check the version you run.
• “Usually” isn’t a rate. The docs say “repeated blocks usually mean the classifier is missing context about your infrastructure,” and suggest reporting false positives or having an administrator configure trusted infrastructure. That’s a diagnosis, not a measurement. Some repeated blocks are correct, and the docs don’t say how many are wrong.

Here is what happens after a block, as the documents describe it. In a terminal, the action appears under /permissions → Recently denied, where you can retry it with a manual approval. In a headless run with auto mode, the action doesn’t run, and the agent keeps working with no one prompted. In the March post’s design, a denial reaches the agent as an instruction to “find a safer path, don’t try to route around the block.” The post says a false positive costs “a single retry,” after which the agent “usually finds an alternative path.” Nothing we read measures what the agent does next, or whether the path it takes is worse than the one it was denied. You have to measure that yourself.

The Replit case in the opening points the same way. Replit’s follow-up separates dev and prod databases by default, so the agent cannot change the production database during development. The environment refuses the action, rather than relying on instructions. For the database side of this, see preparing databases for secure AI agents. For any agent runtime, check four things: the rule order, what triggers escalation, what happens after a block, and what happens when no human is attached.

For how approval gates fit a broader design, see secure agentic code generation; this piece measures those gates.

Decision checklist

Measure it yourself

The fix isn’t a better single number. It’s a labeled log that puts the classifier and people on the same actions.

• Log every proposed action with the setting it ran under, including the ones that allow rules resolve without a prompt.
• Capture the classifier’s decision on each action it reviews. For denials, the PermissionDenied hook receives the exact input.
• Replay isn’t a documented Claude Code feature. Its documented claude auto-mode commands (defaults, config, critique, reset) don’t score logged actions. Replay, as the red-team study uses it, scores existing transcripts against a monitor, so it needs a classifier you can invoke yourself under the setting you’re testing.
• For Claude Code, pair the live decisions with offline human review of a sample. Offline review isn’t live approval: nobody is interrupted and nothing is blocked, so it measures judgment, not behavior under friction.
• Record the human decision on each sampled action, with decision_mode set to live or offline_review. Compare on those rows only.
• Have two people label each action without seeing either decision, and have an adjudicator set the final label. Where possible, use different people as the labelers, the reviewer, and the owner of the cutoff.

Here is one row, pretty-printed:

{
  "setting_id": "auto-v2.1.302-default",
  "action_type": "shell",
  "reversibility": "reversible",
  "scope": "working_directory",
  "classifier_decision": "block",
  "human_decision": "allow",
  "decision_mode": "offline_review",
  "reviewer_id": "dev-17",
  "label_a": "benign",
  "label_b": "benign",
  "label_final": "benign",
  "adjudicator_id": "reviewer-c"
}

Slice the rows by reversibility and scope before you trust a rate. Irreversible actions lose the permission instead of getting a score, so don’t pool them with reversible ones. Then compute the rates per setting and report the sample sizes beside them:

from collections import defaultdict


def rates_by_setting(rows):
    """rows: dicts with setting_id, label_final ('benign' or 'harmful'),
    label_a and label_b (independent labels), classifier_decision (live,
    or from a replay you run), and human_decision ('block', 'allow', or None if no person
    decided). The classifier and the human are compared only on rows a
    person decided. Unknown labels raise, so 'unsure' is never silently
    counted as harmful."""
    tally = defaultdict(lambda: defaultdict(int))
    for row in rows:
        t = tally[row["setting_id"]]
        t["pairs"] += 1
        t["agree"] += int(row["label_a"] == row["label_b"])
        label = row["label_final"]
        if label not in ("benign", "harmful"):
            raise ValueError(f"unknown label: {label!r}")
        if row["human_decision"] is None:
            continue
        if label == "benign":
            t["benign"] += 1
            t["clf_fp"] += int(row["classifier_decision"] == "block")
            t["human_fp"] += int(row["human_decision"] == "block")
        else:
            t["harmful"] += 1
            t["clf_fn"] += int(row["classifier_decision"] == "allow")
            t["human_fn"] += int(row["human_decision"] == "allow")
    return {
        setting: {
            "clf_fpr": t["clf_fp"] / t["benign"] if t["benign"] else None,
            "human_fpr": t["human_fp"] / t["benign"] if t["benign"] else None,
            "clf_fnr": t["clf_fn"] / t["harmful"] if t["harmful"] else None,
            "human_fnr": t["human_fn"] / t["harmful"] if t["harmful"] else None,
            "n_benign": t["benign"],
            "n_harmful": t["harmful"],
            "label_agreement": t["agree"] / t["pairs"],
        }
        for setting, t in tally.items()
    }

Three cautions:

• Sample size. Anthropic’s full-pipeline miss rate rests on 52 cases, and 52 cases is small. One case moves a rate by about two points. The post gives 17% of 52 cases, which is about 9 cases (9 of 52 = 17.3%, our inference). Assuming 9 of 52, the 95% Wilson interval is about 9% to 30% (our calculation).
• Labeler agreement caps what any comparison can show. ToolEmu’s authors reported human-human Cohen’s κ of 0.480 on their safety labels and called the agreement moderate. Set your agreement threshold before labeling starts, and report label_agreement beside every rate. For calibrating a judge against human labels, see LLM-as-a-judge for agent apps.
• The log holds sensitive data. Shell actions and tool results can contain secrets or customer data, so redact before you store them, and set a retention window.

What we don’t know

• What the 93% (March) and 97% (August) approval figures measure. The posts give no measurement period for either figure, and neither defines the denominator.
• What the 10,000-call sample includes, whether calls resolved by allow rules are in it, and how “benign” was labeled.
• What share of human rejections are risk-driven, rather than wrong approaches.
• Whether any published study measures how often human reviewers reject benign agent actions. We found none.
• How humans and the classifier compare on the same actions. We found no published source that measures both on one set.
• What the agent does after a block in a headless run, and whether that path is worse than the one it was denied. The docs say the run continues; nothing we read measures what follows.
• How often repeated blocks are false positives. The vendor says repeated blocks “usually” mean missing context, and gives no rate.

Limits

• Anthropic’s numbers are vendor-reported. The 17% miss rate rests on 52 cases.
• R-Judge uses 2024 models, and ToolEmu is a 2023 framework.
• The red-team result is a snapshot as of September 2026. The researchers typically grant the attacker knowledge of the monitor.
• Product defaults change. The docs we read on October 30, 2026 may not match your version. Pin the default permission mode in managed settings, pin the Claude Code version, and check the effective rules with claude auto-mode config after each upgrade.
• The USENIX study is about browser warnings, not agent approvals.
• Compliance and data-residency review is out of scope here. Logs and classifier inputs may need their own sign-off.
• This article is about measurement design. It doesn’t claim any classifier is production-ready.

Sources

• How we built Claude Code auto mode — Anthropic, published 2026-03-25, modified 2026-10-18. Stage-1 and deployed false positives, overeager misses, the 93% acceptance figure, and the “depends on what you’re comparing against” framing.
• Auto mode is now the default in Claude Code for Pro, Max, and Team plans — Anthropic, 2026-08-07, updated 2026-10-22. The 97% approval, 3% and 39% rejection figures, the 13.6% and 89% study, the June 2026 settings shares, and the “matched or outperformed manual review” claim.
• Red-teaming auto mode — Anthropic Alignment Science Blog, 2026-10-22. The over-55% attack-success figure and its conditions, benign blocking on SWE-chat, tool-result and stage findings.
• Choose a permission mode — Claude Code documentation, accessed October 30, 2026. Starting modes, the headless default, repeated-block thresholds.
• Configure auto mode — Claude Code documentation, accessed October 30, 2026. Config scopes, classifyAllShell, deny rules before the classifier.
• Configure permissions — Claude Code documentation, accessed October 30, 2026. Rule order and the limits of Bash rule matching.
• R-Judge: Benchmarking Safety Risk Awareness for LLM Agents — arXiv, first posted 2024-01-18; PDF v3 dated 2024-10-05. Table 2 prompt variants.
• ToolEmu: Identifying the Risks of LM Agents with an LM-Emulated Sandbox — arXiv, 2023-09-25. Human-human agreement on safety labels.
• Alice in Warningland: A Large-Scale Field Study of Browser Security Warning Effectiveness — Akhawe and Felt, USENIX Security 2013. Click-through by warning design.
• Artificial Intelligence Risk Management Framework (AI RMF 1.0) — NIST, January 2023. §1.2.1 on measurement and human baselines.
• Doubling down on our commitment to secure vibe coding — Replit, 2025-07-29. Dev and prod separation by default, the rollback restore, and the agent’s unawareness of rollback.
• Vibe coding service Replit deleted production database — The Register, 2025-07-21. Reports the founder’s account: the deletion despite his no-changes instruction, the rollback claim, and a later code-freeze attempt.