Evidence before completion claims

A completion claim is the weakest evidence in the session. Rank what you accept, putting the strongest evidence you can actually get at the top.

Anthropic's documentation asks for evidence instead of assertion: the test output, the command and its output, or a screenshot. Reading it is described as faster than re-running verification yourself and as making an unwatched session usable.¹ It names the failure mode the trust-then-verify gap: "Claude produces a plausible-looking implementation that doesn't handle edge cases."¹

False completion has been measured. One study names it false success, a mismatch between an agent's natural-language claim of completion and the programmatic state of the environment, and finds it accounts for a substantial share of all failures across many model families.² Two cautions travel with that evidence. Rates vary widely between models inside a single benchmark, so this is not a uniform property of agents.² And the suites measured are customer-service and personal-app tasks rather than repository-shaped coding work. No equivalent measurement on pull-request-shaped work was found, and no vendor publishes a false-completion rate for its own agent.

The same study explains why a second model reading the claim is weak evidence. Judges lean on confident closing language and on how many actions were taken instead of on verified state, and plain text-statistics detectors beat the best judge.² Its load-bearing implication is environmental: false success was far rarer in the domain where state could be verified independently of the agent's own report.² Field data agrees that self-reporting is a live problem. A study of more than twenty thousand real coding-agent sessions places progress misreporting in its misalignment taxonomy, and separately reports that 91.49 percent of visible resolutions required explicit user correction, a figure covering every kind of misalignment in that corpus rather than misreporting alone.³

The ladder ranks independent verification highest.

<svg viewBox="0 0 700 296" xmlns="http://www.w3.org/2000/svg">
  <title>Evidence strength ladder, from a bare completion claim up to state verified independently of the agent's report</title>
  <g fill="none" stroke="currentColor" stroke-width="1.5">
    <rect x="24" y="212" width="300" height="44" rx="7" opacity="0.35" />
    <rect x="24" y="158" width="440" height="44" rx="7" opacity="0.55" />
    <rect x="24" y="104" width="510" height="44" rx="7" opacity="0.75" />
    <rect x="24" y="50" width="580" height="44" rx="7" />
    <path d="M 648 246 L 648 58" opacity="0.7" />
    <path d="M 641 70 L 648 54 L 655 70" opacity="0.7" />
  </g>
  <g fill="currentColor" font-family="system-ui, sans-serif" font-size="15">
    <text x="44" y="240">"Done." The claim on its own</text>
    <text x="44" y="186">A model judging the claim</text>
    <text x="44" y="132">Pasted command and its output</text>
    <text x="44" y="78">Independently verified state</text>
  </g>
  <g fill="currentColor" font-family="system-ui, sans-serif" font-size="13" opacity="0.7">
    <text x="24" y="276">weakest</text>
    <text x="596" y="40">stronger</text>
  </g>
</svg>

Figure: A completion claim is weakest; independently verified state is strongest, with model judgement and pasted output in between. In practice this means asking for the command and its output, then running something the agent did not. Anthropic recommends verifying against the running app after the agent's own check has passed, and a fresh-context reviewer that sees only the diff and the criteria, so the author of the work is not its grader.¹

That reviewer is recommended and not measured, which is where this page stops. Anthropic documents its cost as well as its benefit: a reviewer told to find gaps will usually report some even when the work is sound, and chasing every finding leads to over-engineering, so it should be scoped to correctness and requirement gaps.¹ Pasted evidence has a floor too. It identifies which command ran and what it returned, but is not independent proof that the command actually ran, and no source shows that demanding it reduces false claims.