Eigma
EN 中 DE
Coding Agents From First Principles 打开 Eigma
目录 Measuring whether throughput improved

Failure modes at scale

Measuring whether throughput improved

The measurement literature on AI coding tools is mixed, and a team that reads one side of it will plan badly. Hold both, and hold each number next to the limitation its own authors state.

The strongest negative result is a randomised controlled trial by METR. Sixteen experienced open-source developers, working 246 issues on mature repositories they averaged five years with, were 19% slower when allowed Cursor Pro with Claude 3.5/3.7 Sonnet, having forecast a 24% speedup beforehand and still reporting a 20% speedup afterwards.1 METR's non-claims travel with that number: it does not show AI fails to speed up most developers, does not generalise beyond software development, and does not rule out more effective ways of using the same tools.1 In 2026 METR changed its experiment design because of selection effects: participants increasingly declined to work without AI and skewed task submissions toward tasks expected to benefit from AI. It now believes developers are likely more sped up than its early-2025 estimate, calls its newer speedup data "only very weak evidence", and treats its figures as a lower bound, warning that the design's validity degrades further as model capability and developer expectations grow.2

The positive results carry limits just as explicit. Google's randomised trial of 96 engineers measured roughly a 21% reduction in time on task with a large confidence interval, and its authors say the effect cannot be assumed to transfer across tools or over time.3 An earlier controlled experiment found a 55.8% speedup with Copilot, but the task was one self-contained HTTP-server implementation in JavaScript, not maintenance of a live system.4 Three randomised field experiments with GitHub Copilot covering 4,867 developers measured a 26.08% increase in completed tasks with a standard error of 10.3% and larger gains among less experienced developers.5 DORA's 2025 survey of nearly 5,000 professionals found adoption positively related to delivery throughput and still negatively related to delivery stability.6 DORA frames AI as an amplifier of existing organisational strengths and weaknesses.7

Two gaps decide how far any of this carries. No randomised trial was found that measures agentic tools rather than autocomplete and chat assistants. The only agentic-tool evidence found is observational: adopters across tens of thousands of engineers merged about 24% more pull requests, and the authors note a merged pull request is not the same as the value it delivers.8 No source measures whether throughput gains survive the rework and change-failure increases DORA reports, so net delivered value is unmeasured. Measure your own delivery rather than importing a figure, and do not set these metrics as targets: DORA warns that doing so invites gaming.9

参考文献

  1. [1]METR, measuring the impact of AI on experienced open-source developer productivity
  2. [2]METR, uplift study update
  3. [3]How much does AI impact development speed? An enterprise randomized controlled trial
  4. [4]The impact of AI on developer productivity: evidence from GitHub Copilot
  5. [5]The effects of generative AI on high-skilled work: three field experiments
  6. [6]Announcing the 2025 DORA report
  7. [7]DORA, balancing the tensions of AI
  8. [8]Observational study of agentic CLI adoption at Microsoft
  9. [9]DORA metrics, the four keys
测验

  1. METR published a productivity estimate for AI coding tools, then later changed its experiment design and revised its own confidence in the number. A team planning next quarter's roadmap should treat that estimate as a ____ rather than a fixed multiplier to build the plan around.

    • lower bound to sanity-check against
    • precise multiplier for scheduling
    • proof the tools slow teams down

    METR now treats its own estimate as a lower bound after revising its design for selection effects, so a team should read it as a floor to sanity-check against rather than a precise planning figure.

  2. What does the reviewed literature not yet establish about these tools?

    • Net delivered value after rework and stability costs
    • That developers report believing the tools make them more productive
    • That a controlled experiment can measure time saved on a self-contained task

    No source in the reviewed literature measures whether AI-linked throughput gains survive the rework and delivery-stability costs DORA reports, so net delivered value stays unmeasured despite estimates on both sides.

评论

还没有评论，来说第一句吧。