tau3-banking: -39% agent cost, measured in the open
On tau3-banking, an agent benchmark, Context Mode cut the agent's cost by 39.3%, 95% range -49.5% to -27.1%. Agent cost fell from $8.51 to $5.16 over 170 task pairs. The pass count moved from 36 to 33, which is not a significant drop. We wrote the plan before the first paid call, and the raw files and checksums are public, so you can rerun the analysis.
TL;DR
- Result: -39.3% agent cost, 95% range -49.5% to -27.1%.
- Quality: pass 36 vs 33 of 170, difference -1.6 points, range -6.8 to +3.7.
- Public: plan, results, raw files, checksums and the analysis script.
- Side by side: Context Mode vs Induction.
How it was run
tau3-bench is a public benchmark from Sierra Research. We used the banking_knowledge domain with BM25 retrieval, 96 tasks, at tau2-bench commit 122e2432. The agent and the user simulator were both Claude Haiku 5.5. We ran two seeds, 300 and 301. One task, task_102, was left out because it calls OpenAI.
Each task ran twice: once direct to the provider, and once through the Context Mode gateway. The setup was the same in both arms. Only the base URL differed. The gateway arm ran on its own funded tenant, and the quota guard confirmed no request was paused.
The plan came first
We pushed the plan, the analysis script and the task list to the repo at 2026-11-01 12:42:42 UTC. The first paid call was at 12:45:48 UTC. The plan names the question, the metric, the stop rule and the spend cap. Read it here: PLAN.md.
The cap was $16 and the plan said the direct arm of seed 301 would probably stop early. It did. It stopped at 74 of 96 tasks, so seed 301 uses the 74 tasks that finished in both arms. We filled in nothing.
The result
| Direct | Gateway | |
|---|---|---|
| Agent cost, 170 pairs | $8.51 | $5.16 |
| Tasks passed, of 170 | 36 | 33 |
By seed: seed 300 (96 pairs) -42.0%, range -52.9% to -27.6%. Seed 301 (74 pairs) -35.4%, range -50.7% to -16.1%.
Cost is the tokens Anthropic returned on each agent call, at list price. The user simulator is counted apart.
What to read with it
- The gateway's own meter reads 3.0% to 3.3% higher than the client sum. On that meter the change is -37.4%.
- Most of the saving comes from fewer calls above 100k prompt tokens, which the provider prices higher. At one flat price for every call, the change is -9.7%.
- The gateway was deployed twice during the runs. Both changes touched only the try page path. The results page lists them.
The full list of deviations from the plan is in RESULTS.md.
What is public
- The plan and the results page, linked above.
- The raw tau2 result files for each arm and seed, in a release. Keys are shown as
***. - The checksums: SHA256SUMS.txt.
- The analysis scripts, analyze.py and summarize.py.
Rerun the analysis
This uses the raw files and no API key. It checks the checksums, then runs the same script we ran.
mkdir -p raw && gh release download tau3-confirm-1101 --repo mksglu/cm-ci-lab --dir raw
shasum -a 256 -c raw/SHA256SUMS.txt
gunzip raw/*.json.gz
curl -fsSLO https://contextmode.ai/benchmarks/tau3/confirm-1101/analyze.py -O https://contextmode.ai/benchmarks/tau3/confirm-1101/summarize.py
python3 analyze.py --d300 raw/direct-s300.json --g300 raw/gateway-s300.json --d301 raw/direct-s301.json --g301 raw/gateway-s301.jsonOther published numbers
Induction publishes a tau3 result too: -37% at medium effort, -60% at xhigh and -67% at max, on gpt-5.6-sol and 97 tasks. We read it on 2026-11-01. The models differ: they used gpt-5.6-sol and we used Claude Haiku 5.5, so the two sets of numbers are side by side, not one test. When we read their page it showed no trial count, raw files or plan. The comparison page has all three of their rows with our range beside them.
The upper end of our range, -27.1%, is above -37%, so the ranges overlap. We do not claim more than that.
Try the CI side
The same gateway cuts CI agent cost by 20% to 23% per run on five jobs. Pick a job on the try page, or run the lab tests in your own repo with the template.