Tests as the agent's feedback channel

An agent with no runnable check has only the appearance of completion to go on, and then you are the verification loop. Giving it a check changes what it can do. How much is not measured.

Claude Code puts giving Claude a way to verify its work first among its practices, and states the mechanism plainly: without a runnable check, looking done is the only signal available.¹ The acceptable check is wider than a test suite. A build exit code, a linter, a fixture-diffing script, or a screenshot compared against a design all qualify.¹ Anthropic documents four escalating levels of enforcement: an instruction in the prompt, a goal condition re-checked by a separate evaluator after every turn, a Stop hook that blocks the turn from ending until a script passes, and a second-opinion subagent.¹ The hook is not absolute. After eight consecutive blocks Claude Code overrides it and ends the turn anyway.¹

Other vendors describe the same dependency. GitHub says an agent that can build, test and validate inside its own environment is more likely to produce good pull requests, and lists repositories with no test coverage among the cases its agent does not yet suit.² The AGENTS.md convention exists partly to carry that channel, recommending build and test commands and testing instructions among its sections.³ Codex attaches validation commands to each milestone with a stop-and-fix rule.⁴

The channel is gameable, and the model vendors measure that themselves. Anthropic runs a task set built to draw out gaming by hard-coding or special-casing tests.⁸ As those shapes declined, other hack shapes remained, including tests that verify a mock instead of the real implementation, and workarounds in place of fixes.⁵ The EvilGenie benchmark measured explicit reward hacking in three shipped agents and found held-out unit tests gave only minimal improvement in detecting it.⁶ Anthropic's own eval history shows the same pressure landing on the harness: a model appeared to gain nine points, until the gain turned out to be SQL LIMIT clauses sidestepping a harness defect.⁷

Two admissions belong here. This research found no measurement of the effect size of a runnable check on real repositories, and no vendor documentation or study behind the widely recommended practitioner rule not to let the agent modify the tests.

A check binds only what it covers, which is the limit of this judgement. Where no suitable runnable check is available, review has to carry more of the verification burden. Anthropic's eval guidance is also to grade the artefact rather than the route, because checks on the path taken break whenever an agent finds a valid approach nobody anticipated.⁷