Living with it
Tasks it is good and bad at
A score tells you which shape of task somebody measured. It does not tell you whether the thing will help you with yours.
On SWE-bench Verified, where a model is handed a real GitHub issue and its patch is judged by whether it resolves the issue and passes the project's tests, Claude Opus 4 scored 72.5% and Claude Sonnet 4 scored 72.7%.1 On GPQA Diamond, 198 multiple-choice questions in biology, physics and chemistry vetted by subject experts so web lookup alone cannot answer them, Claude Opus 4 scored 76.9% with extended thinking.1 That score is on the Diamond subset. A separate measurement, human trials on the full GPQA set from the same paper, found skilled non-experts with web access reaching about 34%, a measure of question difficulty, not a chatbot-versus-human comparison.2 Both are closed tasks scored by a test suite or an answer key: the shape of task where these high scores were measured, not necessarily the shape these systems do best on overall.
Open-ended paid work has been measured too, and it reads differently. OpenAI's GDPval set collects 1,320 tasks from 44 occupations, each written by a professional averaging 14 years' experience and ranked blind against the human version by another in that occupation. On the 220-task public subset the strongest model in that paper, Claude Opus 4.1, had 47.6% of its deliverables graded as good as or better than the professional's, so the human version was preferred in most comparisons, on tasks averaging seven hours of expert work.3 What lost those comparisons was procedure rather than missing knowledge: Claude, Grok and Gemini lost mainly on instruction-following failures, GPT-5 high mainly on formatting.3
| What was measured | Result |
|---|---|
| Real GitHub issues, patch must pass tests | Claude Opus 4 72.5%, Claude Sonnet 4 72.7%1 |
| Expert-written science questions, extended thinking | Claude Opus 4 76.9%, Claude Sonnet 4 71.5%1 |
| 202 hand-scored legal research queries, May 2024 | Lexis+ AI incorrect or unsupported on over 17%, Westlaw over 34%, general-purpose GPT-4 on 43%4 |
| 220 real work deliverables ranked blind by professionals | Claude Opus 4.1 as good as or better 47.6% of the time3 |
| 246 real coding issues, 16 experienced developers, July 2025 | 19% longer with the AI tools than without5 |
Speed is its own trap. In that trial the developers had predicted AI would make them about 24% faster, and afterwards, having measurably been slower, still believed it had sped them up by about 20%.5
That trial is narrow: experienced developers, large repositories they already knew, early-2025 tooling, and not re-run since. Narrower still, nothing above touches personal writing, ordinary conversation, or advice about your own life. No measurement of those was found, so treat claims about them, positive or negative, as untested.
参考文献
测验
Claude Opus 4 scored highly on both SWE-bench Verified and GPQA Diamond, closed tasks with an external scoring rule. What does that kind of score establish about how the same model would do on open-ended occupational work?
- Little on its own, since open-ended work is judged differently
- That it will also rank above the human professional's version
- That it has the occupational knowledge the job requires
A test suite or answer key scores a different thing than a blind professional ranking. On GDPval's deliverables the same class of model was still judged worse than the human's version most of the time.
A model's high score on a closed, externally scored benchmark does not by itself establish that it will perform as well on open-ended work graded by a human professional in that occupation.
- True
- False
Closed benchmarks score against a test suite or key. GDPval, which grades deliverables blind against a professional's version, found the same class of model preferred less often than the human's work on most tasks.
A high score on a benchmark with a test suite or an answer key does not guarantee similarly high performance on ____ work judged by a professional in the field.
- open-ended
- closed-book
- multiple-choice
Externally scored benchmarks measure closed tasks with a known right answer. GDPval judges open-ended deliverables by blind professional ranking, a different measurement, and found lower performance there.
评论
还没有评论，来说第一句吧。