TokenSave Data · Dataset

Raising effort didn't catch more edge cases, and Haiku got nearly the same result for 87% less: 504 Claude Code runs on 21 real bugs

Telling Claude to think about the cases a bug report leaves out, or raising effort to high, did not make it pass more of those cases. Across 504 Claude Code runs on 21 real bugs, no condition moved the share of unreported maintainer test cases passed by more than 4 points (pre-registered threshold: 20). The edge-case prompt did raise the cost per run, by 60% for Opus and 134% for Haiku with the test-reading line. Haiku 5.5 passed 84% of the unreported cases against Opus’s 88% at about 87% less cost per run.

Measured 2026-10-29 · 504 runs · 21 bugs · 12 projects · Claude Code 2.1.295 · License CC BY 4.0

DOI 10.5281/zenodo.23796306 · Also on Hugging Face · Kaggle · GitHub

Key findings

• The prompt and effort did not help. Share of unreported cases passed (mean per bug), baseline / effort high / edge-case prompt / edge-case prompt + read tests: Haiku 5.5 84% / 82% / 84% / 86%; Opus 5.5 88% / 86% / 91% / 84%. The 95% intervals from resampling bugs: Haiku effort high −5 to +1 points, edge-case prompt −4 to +4, with test reading 0 to +5; Opus −9 to +2, 0 to +7 and −12 to +2. None reaches the 20-point threshold set before the runs; small effects of a few points cannot be ruled out.
• But they cost more. Mean API cost per run, same order: Haiku $0.050 / $0.081 / $0.10 / $0.12 (up to +134%); Opus $0.51 / $0.62 / $0.82 / $0.81 (up to +60%). Effort high alone added 60% for Haiku and 22% for Opus. Median time per run more than doubled with the edge-case prompt: Haiku 121s to 299s, Opus 120s to 271s.
• Clean fixes barely moved. Runs where every maintainer case passed and nothing else broke, out of 63 per cell: Haiku 34, 33, 36, 37; Opus 38, 38, 39, 37. Runs that broke other maintainer tests: Haiku 9, 8, 7, 5; Opus 5, 3, 6, 4.
• Haiku against Opus. Over all 252 runs each, Haiku cost $0.088 per run and Opus $0.69; clean fixes 140 and 152. Opus was ahead by a few points in most cells and broke other tests in fewer runs; Haiku did it for about 13% of the cost.
• The one hint of an effect: on the 8 harder bugs carried over from dataset 25, Opus with the edge-case prompt passed 83% of unreported cases against 74% at baseline (interval +2 to +17 points). Adding “read the existing tests first” took it back to 75%. We read this as noise until it repeats.
• What decided the result was the bug, not the prompt. On a js-yaml bug (a BOM before every document in a stream) no run in any condition passed the unreported case. On 12 of the 21 bugs every condition passed every unreported case for both models. Most of the movement came from a few bugs: a zod JSON Schema bug, where Haiku rose from 73% to 89% with the edge-case prompt and test reading, and a marked nested-link bug, where Opus went 2, 0, 3 and 1 out of 3 across the four conditions.
• Test reading sometimes hurt. With “read the existing tests first”, Opus failed the unreported case on a Python-Markdown backtick bug in all 3 runs (2 of 3 passed at baseline) and missed one on a cachetools mock bug it otherwise always passed.

Totals by model and condition

Unreported passed = share of the maintainer test cases the bug report does not show, averaged per bug (the pre-registered primary outcome). It includes cases that already pass before the fix, which a careless fix can break. Unreported targets = only the unreported cases that fail before the fix. Interval: 95%, from resampling whole bugs.

Bug by bug: unreported cases passed

C0 baseline, C1 effort high, C2 edge-case prompt, C3 edge-case prompt + read tests. 3 runs per cell. Cases R/U = test cases the report shows / leaves out.

The four conditions

• Baseline: the bug report cut down to symptom, reproduction and expected result, effort medium. Same prompts as dataset 25 for the 8 carried-over bugs.
• Effort high: the same prompt with --effort high.
• Edge-case prompt: the report plus “Before changing code, list the inputs and variants this report does not mention but the fix must also handle. Write tests for them first, then fix.”
• Edge-case prompt + read tests: the above plus “Read the existing tests for the code you change before you start.”

Method

• Pre-registered on 2026-10-25 before any run, with four dated amendments, all before the first run: PREREG.md. The case labels (reported or not) and the check files were frozen with sha256 hashes.
• Bugs: 21 bugs fixed upstream between August and October 2026 in cachetools, Python-Markdown, anyio, python-dateutil, zod, fastify, p-queue, undici, js-yaml, marked, qs and tomlkit. Each has at least one maintainer test case the report does not show. Parent and fix commits are in the CSV.
• Agent: Claude Code 2.1.295 on one Windows 11 laptop, headless (-p), acceptEdits, the project’s own tests allowed, package installs and network denied. 3 runs per bug, model and condition, each from a fresh copy.
• Scoring: only the run’s changes to library source were kept; tests it wrote were dropped. The maintainers’ tests for the fix were run 5 times before and after the fix, and any test that changed between repeats was dropped. Each remaining test case was labelled reported or unreported before any run.
• Cost is Claude Code’s own per-run figure at API prices; the runs were made on a subscription.

Limitations

• Many new bugs were easy for both models: 12 of 21 bugs had every unreported case passed in every condition. That leaves little room for a prompt to help, so the result applies to bugs like these and says less about very hard ones.
• 3 runs per cell and 21 bugs. Effects of a few points cannot be ruled out; effects of 20 points can.
• One prompt wording per condition. A different wording, or a different agent, might do better.
• The anyio bugs were scored on Windows and the rest on Linux, with the same scorer and the same pre-fix test baseline method.
• The scoring setup was rebuilt from the published code after our cloud workspace was reset; re-scoring the 228 Haiku runs scored before the reset gave identical numbers.
• The projects are public, so the models may have seen similar code. All fixes are from August to October 2026.

How to cite

Free to use, quote and republish with attribution under CC BY 4.0. Please link to this page.

BibTeX

@misc{tokensave_claude_edge_case_prompts_2026,
  author    = {{TokenSave}},
  title     = {Raising effort didn't catch more edge cases, and Haiku got nearly the same result for 87\% less: 504 Claude Code runs on 21 real bugs},
  year      = {2026},
  publisher = {Zenodo},
  doi       = {10.5281/zenodo.23796306},
  url       = {https://doi.org/10.5281/zenodo.23796306},
  note      = {Dataset, CC BY 4.0}
}

Embed the chart

Related

• Claude Haiku vs Opus on 25 real bugs (dataset 25)
• Claude effort levels: tokens, time and cost
• Where Claude Haiku falls behind Opus
• Estimate your own Claude Code or Codex bill

Questions or corrections: contact@jonhisking.com