−39% agent cost vs Induction's published −37%

On tau3-banking, Context Mode cut agent cost by 39.3% (95% range 27.1% to 49.5%). Induction publishes −37% at medium effort. The pass rate did not drop in a significant way: 36 against 33 of 170 pairs.

The models differ: Induction used gpt-5.6-sol, we used Claude Haiku 5.5. Read the two sets of numbers side by side, not as one test.

Cost change

Context Mode on Haiku 5.5: −39.3%, 95% range −49.5 to −27.1, 170 task pairs. Induction on gpt-5.6-sol: −37% (medium), −60% (xhigh), −67% (max). Induction gives no range, so its bars have no whisker. In dollars, ours is the total agent cost of 170 task pairs ($8.51 to $5.16); theirs is the cost of one run of 97 tasks ($59 to $37, $130 to $52, $225 to $74). Each pair of bars is scaled to its own direct cost.

Pass rate

Context Mode: 21.2% direct (36 of 170), 19.4% through Context Mode (33 of 170). The difference is −1.6 points (95% range −6.8 to +3.7). Induction: 28.6% to 31.3% at medium, 40.0% to 39.7% at xhigh, 43.8% to 47.1% at max.

Side by side

	Model	Tasks	Trials	Cost change (95% range)	Pass rate	Raw data public	Plan written first	Rerun
Context Mode	Haiku 5.5	96	2 seeds, 170 pairs	−39.3% (−49.5 to −27.1)	21.2% to 19.4%	Yes: release	Yes: plan	Steps below
Induction	gpt-5.6-sol	97	Not stated	−37% (medium), −60% (xhigh), −67% (max); no range	28.6% to 31.3%, 40.0% to 39.7%, 43.8% to 47.1%	No	No	None listed

Induction's figures are from its tau3 page and method page, read 2026-11-01. The columns for Induction say what those pages show.

Read the numbers with these notes

The upper end of our 95% range (−27.1%) is above Induction's −37%, so the ranges overlap.
The gateway's own meter reads 3.0% to 3.3% higher than the client sum. On that meter the change is −37.4%.
Most of the saving comes from fewer calls above 100k prompt tokens, which Anthropic prices higher. At one flat price for every call the change is −9.7%.
Seed 301 direct stopped at 74 of 96 tasks (spend cap).

Agent cost: the tokens Anthropic returned on each agent call, at list price. The user simulator is counted apart.

Proof

Plan commit0442a3a1, pushed 2026-11-01 12:42:42 UTC. Read the plan.

First paid call2026-11-01 12:45:48 UTC, three minutes after the push.

ResultsRESULTS.md, commit 1a547af6. Data file: JSON.

Harnesstau2-bench, commit 122e2432, domain banking_knowledge, bm25 retrieval.

Raw filesRelease tau3-confirm-1101. Checksums: SHA256SUMS.txt.

File	Size	sha256
direct-s300.json.gz	10.1 MB	17cb530a11fb91a60670c381491c137386e7dd38e1aadeb994f4a2f96fc495f1
direct-s301.json.gz	7.2 MB	36e543c16f864f308a147f1c86293b96596b9da27e564ece447863cb69e5febf
gateway-s300.json.gz	11.0 MB	b4c9a7d6663f09d1d450c898faa60bf2dbab53f97382f30ad6c53b9da0da8c36
gateway-s301.json.gz	11.2 MB	c2e4fe8eaa742c0e5f98b6c03d32222aadffacd7d238c051c542368e150bc3c2

Run it again

Download the raw files, check them against the checksums, and recompute the result with the analysis script that was in the plan commit. It uses a fixed random seed, so you get the same numbers.

mkdir -p raw && gh release download tau3-confirm-1101 --repo mksglu/cm-ci-lab --dir raw
shasum -a 256 -c raw/SHA256SUMS.txt
gunzip raw/*.json.gz
curl -fsSLO https://contextmode.ai/benchmarks/tau3/confirm-1101/analyze.py -O https://contextmode.ai/benchmarks/tau3/confirm-1101/summarize.py
python3 analyze.py --d300 raw/direct-s300.json --g300 raw/gateway-s300.json --g301 raw/gateway-s301.json --d301 raw/direct-s301.json

Get started free All benchmarks