# Default effort: Context Mode −39%, Induction −37%

On tau3-banking, Context Mode cut agent cost by 39.3% (95% range 27.1% to 49.5%) at its default effort. Induction publishes −37% at medium effort. The pass rate did not drop in a significant way: 36 against 33 of 170 pairs.

The models differ: Induction used gpt-5.6-sol, we used Claude Haiku 5.5. Only the medium row is near our setting. Read the sets side by side, not as one test.

## Cost change

### Same tier: default effort

### Higher reasoning effort (Induction only)

### Same tier: default effort

### Higher reasoning effort (Induction only)

These higher-effort runs spend 2 to 4 times more per run ($130 direct at xhigh, $225 direct at max, against $59 at medium). The saving grows with the size of what is re-sent. We run the same model, Haiku 5.5, at higher reasoning effort next.

Context Mode on Haiku 5.5, default effort: −39.3%, 95% range −49.5 to −27.1, 170 task pairs. Induction on gpt-5.6-sol: −37% (medium), −60% (xhigh), −67% (max). Induction gives no range, so its bars have no whisker. In dollars, ours is the total agent cost of 170 task pairs ($8.51 to $5.16); theirs is the cost of one run of 97 tasks ($59 to $37, $130 to $52, $225 to $74). Each pair of bars is scaled to its own direct cost. Induction figures: [source](https://www.induction.ai/docs/benchmarks/tau3), read 2026-11-01.

## Pass rate

Context Mode: 21.2% direct (36 of 170), 19.4% through Context Mode (33 of 170). The difference is −1.6 points (95% range −6.8 to +3.7). Induction: 28.6% to 31.3% at medium, 40.0% to 39.7% at xhigh, 43.8% to 47.1% at max.

## Side by side

|  | Model | Tasks | Trials | Cost change (95% range) | Pass rate | Raw data public | Plan written first | Rerun |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| **Context Mode** | Haiku 5.5 | 96 | 2 seeds, 170 pairs | −39.3% (−49.5 to −27.1) | 21.2% to 19.4% | Yes: [release](https://github.com/mksglu/cm-ci-lab/releases/tag/tau3-confirm-1101) | Yes: [plan](https://contextmode.ai/benchmarks/tau3/confirm-1101/PLAN.md) | [Steps below](#rerun) |
| **Induction** | gpt-5.6-sol | 97 | Not stated | −37% (medium), −60% (xhigh), −67% (max); no range | 28.6% to 31.3%, 40.0% to 39.7%, 43.8% to 47.1% | No | No | None listed |

Induction's figures are from [its tau3 page](https://www.induction.ai/docs/benchmarks/tau3) and [method page](https://www.induction.ai/docs/benchmarks), read 2026-11-01. The columns for Induction say what those pages show.

## Read the numbers with these notes

- The upper end of our 95% range (−27.1%) is above Induction's −37%, so the ranges overlap.
- The gateway's own meter reads 3.0% to 3.3% higher than the client sum. On that meter the change is −37.4%.
- Most of the saving comes from fewer calls above 100k prompt tokens, which Anthropic prices higher. At one flat price for every call the change is −9.7%.
- Seed 301 direct stopped at 74 of 96 tasks (spend cap).

Agent cost: the tokens Anthropic returned on each agent call, at list price. The user simulator is counted apart.

## Proof

Plan commit 0442a3a1, pushed 2026-11-01 12:42:42 UTC. [Read the plan](https://contextmode.ai/benchmarks/tau3/confirm-1101/PLAN.md). First paid call 2026-11-01 12:45:48 UTC, three minutes after the push. Results [RESULTS.md](https://contextmode.ai/benchmarks/tau3/confirm-1101/RESULTS.md), commit 1a547af6. Data file: [JSON](https://contextmode.ai/benchmarks/data/tau3-confirm-1101.json). Harness [tau2-bench](https://github.com/sierra-research/tau2-bench), commit [122e2432](https://github.com/sierra-research/tau2-bench/commit/122e2432b82d740e8f313f46f0a071729c404f39), domain banking_knowledge, bm25 retrieval. Raw files [Release tau3-confirm-1101](https://github.com/mksglu/cm-ci-lab/releases/tag/tau3-confirm-1101). Checksums: [SHA256SUMS.txt](https://github.com/mksglu/cm-ci-lab/releases/download/tau3-confirm-1101/SHA256SUMS.txt).

| File | Size | sha256 |
| --- | --- | --- |
| [direct-s300.json.gz](https://github.com/mksglu/cm-ci-lab/releases/download/tau3-confirm-1101/direct-s300.json.gz) | 10.1 MB | 17cb530a11fb91a60670c381491c137386e7dd38e1aadeb994f4a2f96fc495f1 |
| [direct-s301.json.gz](https://github.com/mksglu/cm-ci-lab/releases/download/tau3-confirm-1101/direct-s301.json.gz) | 7.2 MB | 36e543c16f864f308a147f1c86293b96596b9da27e564ece447863cb69e5febf |
| [gateway-s300.json.gz](https://github.com/mksglu/cm-ci-lab/releases/download/tau3-confirm-1101/gateway-s300.json.gz) | 11.0 MB | b4c9a7d6663f09d1d450c898faa60bf2dbab53f97382f30ad6c53b9da0da8c36 |
| [gateway-s301.json.gz](https://github.com/mksglu/cm-ci-lab/releases/download/tau3-confirm-1101/gateway-s301.json.gz) | 11.2 MB | c2e4fe8eaa742c0e5f98b6c03d32222aadffacd7d238c051c542368e150bc3c2 |

## Run it again

Download the raw files, check them against the checksums, and recompute the result with the analysis script that was in the plan commit. It uses a fixed random seed, so you get the same numbers.

```
mkdir -p raw && gh release download tau3-confirm-1101 --repo mksglu/cm-ci-lab --dir raw
shasum -a 256 -c raw/SHA256SUMS.txt
gunzip raw/*.json.gz
curl -fsSLO https://contextmode.ai/benchmarks/tau3/confirm-1101/analyze.py -O https://contextmode.ai/benchmarks/tau3/confirm-1101/summarize.py
python3 analyze.py --d300 raw/direct-s300.json --g300 raw/gateway-s300.json --g301 raw/gateway-s301.json --d301 raw/direct-s301.json
```

[Get started free](https://console.contextmode.ai/login) [All benchmarks](https://contextmode.ai/docs/benchmarks)