Reviewing agent-written code

The reason to review an agent's diff carefully is not a known defect rate. It is an observed pattern, not a measured failure: agent pull requests are approved immediately after automated checks pass, with no reviewer modification, and this page's judgement is that such approvals call for careful review. An MSR 2026 study of 387 agent-authored pull requests touching build files, across five agents, found 61.4% merged, most immediately after automated checks passed, with no reviewer modifications and frequently an approving one-line comment.¹ Read that as a build-system finding, not agent pull requests generally. Perry et al. found that participants with an AI assistant wrote significantly less secure code and were more likely to believe it was secure, while those who trusted the assistant less and engaged more with their prompts produced fewer vulnerabilities.² METR's randomized trial of 16 experienced maintainers over 246 tasks in their own repositories recorded a 19% increase in completion time when AI tools were allowed, against a forecast 24% speedup and an after-the-fact estimate of 20%.³ METR states this reflects early-2025 capabilities in one setting, not that AI tools generally slow developers.³ Taken together, these point at self-assessment as the unreliable part.

Be exact about what the security numbers measure. Veracode's 2025 report found 45% of generated code samples failed security tests, with per-language rates from 38% to 72%, and that security performance stayed flat as models grew larger and newer.⁴ Those are benchmark-task samples, not a defect rate for agent-authored pull requests in real repositories, and cannot be quoted as one. A separate comparison of AI-generated and human-written files from the same repositories reports roughly twice the high-risk alerts per thousand lines in the AI files, 0.934 against 0.464, with total alerts similar.⁵ Both studies identify AI-written code by different heuristics, so neither sample is necessarily representative of agent-written code generally.

In practice, review the diff in a fresh context: a reviewer who did not write the code is not biased toward it, and should see the diff and the criteria, not the reasoning behind the change.⁶ Ask for evidence, command and output, instead of an assertion of success.⁶ The same guidance warns about the reviewer as well: "A reviewer prompted to find gaps will usually report some", and chasing every finding produces over-engineering.⁶

When this does not hold. The evidence is thin in one specific place. No controlled experiment was found measuring whether human reviewers detect fewer defects in machine-generated code than in human-written code of equal defect density. No measurement was found of an LLM reviewer's catch rate or false-positive rate. The case for careful review rests on merge behaviour in one narrow sample and on over-trust, not on measured review effectiveness.