TokenSave
TokenSave Data · Dataset
Claude Haiku vs Sonnet vs Opus on 25 real bugs: 225 runs scored by the maintainers' tests
On 25 real bugs from 13 open-source projects, Claude Haiku 5.5 fixed as many cleanly as Opus 5.5 (52 and 54 of 75 runs) at about 95% less cost per run. Opus pulled ahead on the hard bugs (16 of 30 clean fixes against Haiku’s 12), mainly because it broke other tests in fewer runs. None of the differences between Haiku and Opus is statistically significant with 3 runs per bug. Where the bug report left a case out, every model missed it.
Measured 2026-10-24 · 225 runs · 25 bugs · 13 projects · Claude Code 2.1.295 · License CC BY 4.0
DOI 10.5281/zenodo.23619927 · Also on Hugging Face · Kaggle · GitHub
Estimate your own Claude Code or Codex bill → See the monthly cost of Claude Code and Codex for your usage: API per task vs Claude and ChatGPT plans. Free, no sign-up.
Key findings
- Overall, Haiku matched Opus. Clean fixes (every maintainer test case passes and nothing else breaks): Haiku 52/75, Sonnet 49/75, Opus 54/75. Resampling by bug, the 95% interval for Haiku minus Opus is −11 to +5 percentage points: no evidence that Opus fixes more bugs, though a gap of about 10 points cannot be ruled out.
- Cost per run at API prices: Haiku $0.019, Sonnet $0.14, Opus $0.38. All 75 Haiku runs cost $1.41 together; the 75 Opus runs $28.53.
- Easy round (15 bugs where cause and symptom sit close together): clean fixes Haiku 40/45, Sonnet 35/45, Opus 38/45, and no run broke the project’s own test suite.
- Hard round (10 bugs with symptom-only reports, many with the cause away from the symptom): clean fixes Haiku 12/30, Sonnet 14/30, Opus 16/30; runs that broke other maintainer tests: 10, 7 and 5. Here the interval for Haiku minus Opus is −33 to 0 points, so Opus may well be better on bugs like these.
- Bug by bug, Haiku and Opus had the same number of clean fixes on 22 of the 25 bugs. Opus had more on nested links in marked (2 of 3 against 0; every Haiku run broke the spec test for that example) and on a markdown-it code-span bug (3 against 1; two Haiku runs broke other tests). Haiku had more on a js-yaml block scalar bug (3 against 1; Sonnet 0). On a zod bug that needed numeric record keys handled in several places, no run fixed everything, but Opus passed 14 of the 24 cases over its 3 runs, Haiku 9 and Sonnet 5.
- What no model found: on three hard bugs every run fixed the reported example and missed the same unreported case: cachetools (a second maintainer test that calls every variant of the decorator through the class), js-yaml (a BOM before each document in a stream, not only the first) and Python-Markdown (every run broke 3 to 5 of the maintainers’ tests for comments after other inline HTML). On the tomlkit underscore bug every run passed the same 20 of 29 cases. Model size did not help.
- Speed: median seconds per run Haiku 66, Sonnet 34, Opus 84.
- This updates our previous dataset. There, on one hard bug in marked, Sonnet and Opus caught an unreported edge case far more often than Haiku. Across 25 bugs that pattern did not hold: the unreported cases here were missed by every model alike.
Hard round: test cases passed per run
Target cases = the maintainers’ tests for the fix that fail before it and pass after it. A red !n means the run broke n other tests in the same test files that pass both before and after the fix.
| Project · bug | Haiku 5.5 | Sonnet 5.5 | Opus 5.5 | Cases |
|---|---|---|---|---|
| cachetools · cachedmethod class access loses the explicitly supplied instance | 1, 1!1, 1 | 1, 1, 1 | 1, 1, 1 | 2 |
| js-yaml · Sequence parsing in version 5 fails (BOM at the start of the file) | 1, 1, 1 | 1, 1, 1 | 1, 1, 1 | 2 |
| markdown-it · Unclosed link/image label suppresses a later inline code span | 2!2, 2, 2!2 | 2, 2!2, 2 | 2, 2, 2 | 2 |
| marked · A link inside link text produces nested links | 3!1, 3!1, 3!1 | 3!1, 4!1, 3!1 | 4, 4!1, 4 | 4 |
| marked · Emphasis is lost in the text of a rejected reference link | 2, 2, 2 | 2, 2, 2 | 2, 2, 2 | 2 |
| more-itertools · numeric_range: len(), in and index() disagree with the items produced, for float steps | 2, 2, 2 | 2, 2, 2 | 2, 2, 2 | 2 |
| pyparsing · match_previous_expr fails on nested and repeated matches | 2, 2, 2 | 2, 2, 2 | 2, 2, 2 | 2 |
| Python-Markdown · HTML comment inside inline raw HTML creates malformed output | 2!4, 2!4, 2!5 | 2!4, 2!3, 2!4 | 2!3, 2!3, 2!4 | 2 |
| zod · Functions are executed twice for object property defaults | 3, 3!1, 3 | 3, 3, 3 | 3!1, 3, 3 | 3 |
| zod · toJSONSchema on record with numeric keys results in invalid propertyNames | 3, 0, 6 | 2, 3, 0 | 4, 5, 5 | 8 |
Easy round: test cases passed per run
Cases written from the maintainers’ tests for the fix. No run broke the project’s own test suite.
| Project · bug | Haiku 5.5 | Sonnet 5.5 | Opus 5.5 | Cases |
|---|---|---|---|---|
| humanize · metric() shows one significant figure too many | 6, 6, 6 | 6, 6, 6 | 6, 6, 6 | 6 |
| js-yaml · forceQuotes quotes non-string scalars and emits explicit type tags | 7, 7, 7 | 7, 7, 7 | 7, 7, 7 | 7 |
| js-yaml · Block scalar with a whitespace-only body fails | 3, 3, 3 | 2, 2, 2 | 3, 2, 2 | 3 |
| jsdiff · diffJson throws on truthy, non-callable toJSON | 2, 2, 2 | 2, 2, 2 | 2, 2, 2 | 2 |
| jsdiff · applyPatch drops a literal carriage return at the end of a file | 5, 5, 5 | 3, 5, 3 | 5, 5, 5 | 5 |
| jsdiff · diffJson serializes arrays with toJSON wrongly | 2, 2, 2 | 2, 2, 2 | 2, 2, 2 | 2 |
| markdown-it · IPv6 address literal brackets are percent-encoded in link destinations | 3, 3, 3 | 3, 3, 3 | 3, 3, 3 | 3 |
| more-itertools · seekable(..., maxlen=0): peek() and bool() drop items | 3, 3, 3 | 3, 3, 3 | 3, 3, 3 | 3 |
| more-itertools · ichunked(..., 0) raises islice's ValueError on a nonempty iterable | 3, 3, 3 | 3, 3, 3 | 3, 3, 3 | 3 |
| PapaParse · unparse crashes on a null cell when skipEmptyLines is true | 4, 4, 4 | 4, 4, 4 | 4, 4, 4 | 4 |
| qs · stringify: a lone surrogate swallows the next character | 19, 19, 19 | 19, 19, 19 | 19, 19, 19 | 19 |
| qs · comma: true is ignored with charset 'iso-8859-1' and interpretNumericEntities | 11, 11, 12 | 11, 11, 12 | 11, 12, 11 | 12 |
| tomlkit · Emitting underscores in integers | 20, 20, 20 | 20, 20, 20 | 20, 20, 20 | 29 |
| tomlkit · Deleting a slice from an array corrupts the rendered output | 3, 3, 3 | 3, 3, 3 | 3, 3, 3 | 3 |
| tomlkit · Parse errors report the wrong line and column for CRLF files | 3, 3, 3 | 3, 3, 3 | 3, 3, 3 | 3 |
Totals by round and model
Full fix = every target case passes. Clean fix = full fix and no other test broken. Cases passed = share of all target cases over all runs.
| Model | Runs | Cases passed | Full fix | Clean fix | Runs that broke tests | Cost/run | Median s |
|---|---|---|---|---|---|---|---|
| Easy · Haiku 5.5 | 45 | 91% | 40 | 40 | 0 | $0.011 | 48 |
| Easy · Sonnet 5.5 | 45 | 88% | 35 | 35 | 0 | $0.095 | 27 |
| Easy · Opus 5.5 | 45 | 90% | 38 | 38 | 0 | $0.26 | 47 |
| Hard · Haiku 5.5 | 30 | 72% | 18 | 12 | 10 | $0.030 | 154.5 |
| Hard · Sonnet 5.5 | 30 | 69% | 19 | 14 | 7 | $0.22 | 76.5 |
| Hard · Opus 5.5 | 30 | 82% | 21 | 16 | 5 | $0.57 | 167.5 |
| All · Haiku 5.5 | 75 | 87% | 58 | 52 | 10 | $0.019 | 66 |
| All · Sonnet 5.5 | 75 | 84% | 54 | 49 | 7 | $0.14 | 34 |
| All · Opus 5.5 | 75 | 88% | 59 | 54 | 5 | $0.38 | 84 |
Statistical tests
Fisher’s exact test on clean fixes per run; a paired sign-flip permutation test on the share of cases passed, pairing runs of the same bug and repetition; and a 95% interval for the difference in clean-fix rate from resampling whole bugs, because runs on the same bug are not independent. With 25 bugs, a real 10-point difference would usually go undetected; showing that two models are equal within ±10 points would take several hundred runs per model.
| Comparison | Clean fix, Fisher p | Cases, paired p | Clean-fix difference, 95% interval |
|---|---|---|---|
| Easy · Haiku vs Opus | 0.76 | 0.376 | 0 to +13 pts |
| Easy · Haiku vs Sonnet | 0.26 | 0.063 | 0 to +29 pts |
| Easy · Sonnet vs Opus | 0.59 | 0.185 | −18 to 0 pts |
| Hard · Haiku vs Opus | 0.44 | 0.092 | −33 to 0 pts |
| Hard · Haiku vs Sonnet | 0.80 | 0.870 | −17 to 0 pts |
| Hard · Sonnet vs Opus | 0.80 | 0.061 | −23 to +7 pts |
| All · Haiku vs Opus | 0.86 | 0.497 | −11 to +5 pts |
| All · Haiku vs Sonnet | 0.73 | 0.078 | −4 to +15 pts |
| All · Sonnet vs Opus | 0.48 | 0.006 | −16 to +1 pts |
Method
- Bugs: 25 bugs fixed upstream between 12 August and 20 October 2026 in js-yaml, jsdiff, qs, PapaParse, markdown-it, marked, zod (JavaScript and TypeScript), humanize, tomlkit, more-itertools, cachetools, Python-Markdown and pyparsing (Python). The commit before each fix is in the CSV.
- Starting point: the source at the commit before the fix, with dependencies installed and a fresh git history, so the fix could not be read from git. The prompt was the bug report reduced to the symptom, the reproduction and the expected result; cause analysis and proposed fixes were removed.
- Agent: Claude Code 2.1.295 on one Windows 11 laptop, headless (
-p), effort medium, acceptEdits, with read-only shell commands, the project’s test runner and its build allowed; package installs, network and web tools denied. 3 runs per bug and model, each in a fresh folder. - Scoring: only the run’s changes to the library source were kept (tests it wrote or edited were dropped), then the maintainers’ tests for the fix were run against it. Easy round: hidden cases written from those tests plus the project’s full test suite. Hard round: the fix’s test files, comparing every test before and after the fix.
- Cost is Claude Code’s own per-run figure at API prices; the runs were made on a subscription.
Limitations
- 3 runs per bug and model, 25 bugs. The differences between models are within chance; see the intervals above.
- The permissions differ from dataset 24: here every run could run the project’s tests.
- The tomlkit underscore report was written by us from the upstream issue, which had no reproduction.
- On Windows, zod’s CLAUDE.md (a symlink) did not extract, so the zod runs had no project instructions.
- Timing tests (quadratic, linear, performance) were not counted as broken, because they fail at random under load.
- In the hard round, “broke other tests” means tests the maintainers added or kept in the fix’s test files, many of them edge cases near the bug that the run could not see.
- The projects are public, so the models may have seen similar code in training. All fixes are from August to October 2026.
- Most easy-round bugs are small: a few lines in one function.
How to cite
Free to use, quote and republish with attribution under CC BY 4.0. Please link to this page.
TokenSave (2026). Claude Haiku vs Sonnet vs Opus on 25 real bugs: 225 runs scored by the maintainers' tests [Data set]. Zenodo. https://doi.org/10.5281/zenodo.23619927
BibTeX
@misc{tokensave_claude_haiku_vs_opus_real_bugs_2026,
author = {{TokenSave}},
title = {Claude Haiku vs Sonnet vs Opus on 25 real bugs: 225 runs scored by the maintainers' tests},
year = {2026},
publisher = {Zenodo},
doi = {10.5281/zenodo.23619927},
url = {https://doi.org/10.5281/zenodo.23619927},
note = {Dataset, CC BY 4.0}
}
Embed the chart
Related
- Where Claude Haiku falls behind Opus (dataset 24)
- Vibe coding by Claude model
- Claude effort levels: tokens, time and cost
- Estimate your own Claude Code or Codex bill
Questions or corrections: contact@jonhisking.com