# tau3 banking_knowledge: pre-registered replication vs Induction (2026-11-01)

This file, `analyze.py`, `run-all.sh`, `run-arm.sh`, `tasks96.ids` and the helper scripts are pushed to `main` BEFORE any
paid run. The push commit sha and the time GitHub shows for it go in RESULTS.md. Any change after that push goes in the
"Deviations" section of RESULTS.md.

## Question

Does the gateway (dedup on, minimum run 256 B, as live) cut the agent cost of tau3 banking_knowledge by more than
Induction's published -37% (medium, on a different model), without a significant drop in pass rate?

Earlier exploratory result (one seed, direct arm reused from 1030): -43.6% [-58.3, -23.7], pass 17 vs 14 of 96
(`docs/bench/tau3-n2-1101.md`). This run repeats it with fresh direct arms, two seeds, and a fixed analysis.

## Harness (as docs/bench/tau3-n2-1101.md)

- tau3-bench (`sierra-research/tau2-bench`) at commit `122e2432b82d740e8f313f46f0a071729c404f39`.
- Domain `banking_knowledge`, retrieval `bm25`, 96 tasks (`tasks96.ids`; task_102 out), 1 trial per seed.
- Agent and user simulator: `anthropic/claude-haiku-5-5` through LiteLLM. Agent args: `drop_params`, timeout 180 s,
  `cache_control_injection_points` = system message + last message, in BOTH arms. User simulator: direct to Anthropic in
  both arms, no cache injection.
- `--max-concurrency 3 --max-retries 8 --retry-delay 30 --auto-resume`.
- Direct arm: LiteLLM default Anthropic endpoint. Gateway arm: `api_base` `https://gateway.contextmode.ai`, headers
  `x-context-mode-key` (owner key, scrubbed to `***` from every output file) and `x-cm-tenant`.
- Gateway deploy at plan time: `/__version` ends `...-s213-thinking-integrity-s214-interleaved-thinking-banner`. Dedup
  source `crates/cm-hotpath/src/dedup.rs` and `src/code-mode/result-dedup.ts` (`CM_DEDUP_MIN = 256`) as on `main`.
- Anthropic key: `bmksglu` only. A 1-token call before each arm (`keycheck.py`); "Credit balance is too low" stops the run.

## Runs (same day, this order)

| # | arm | seed | tenant |
|---|---|---|---|
| 1 | direct | 300 | - |
| 2 | gateway | 300 | t_bench_c1 |
| 3 | gateway | 301 | t_bench_c2 |
| 4 | direct | 301 | - |

Driver: `run-all.sh`. Spend guard: `spend-guard.sh` (per-call usage at list price, agent + user simulator, all four arms).

## Tenant setup (before run 1)

- `t_bench_c1`, `t_bench_c2`: new, used 0, used_paused 0 at plan time.
- Each: `POST /__quota-admin {op: grant, n: 2000}`, then `POST /__flags {tenant, dedup: true}` (as tau3-dedup-1030).
- Check: `/__flags` shows `dedup: true`; quota snapshot before and after each gateway arm. **used_paused must stay 0**.
  If it grows, that gateway arm is invalid and is reported as such, not used in the claim.
- Lab quota guard `ci-quota-guard.py --need 500` before and after each arm; a round is valid only if its `used_paused`
  did not grow.

## Budget and stop rule

Cap $16 on the bmksglu key (agent + user simulator). Expected spend from earlier runs: direct about $5.8 per seed, gateway
about $3.5 per seed, total about $18.6, which is ABOVE the cap. So the guard will probably stop run 4 (direct s301)
part-way. Fixed rule, set now:

- Tasks start in `tasks96.ids` order (concurrency 3), so a stop leaves about a prefix of the list finished.
- Seed 301 is analysed on the tasks that finished in BOTH of its arms. Unfinished tasks are not filled in or modelled.
- The pooled primary metric uses every finished (task, seed) pair. Seed 300 alone is reported as well.
- No arm is re-run or topped up after the cap, and no other key is used.

## Analysis (analyze.py, fixed)

- Unit: a (task, seed) pair finished in both arms. `infrastructure_error` does not count as finished.
- Cost: per-call usage at Haiku 5.5 list price (`summarize.py`, same as 1030 and n2). Primary metric is AGENT cost.
- **Primary:** pooled agent-cost change sum(gateway) / sum(direct) - 1 over all pairs. 95% CI: paired bootstrap that
  resamples TASKS (a task carries all its seed pairs), 10,000 draws, `random.Random(1)`, 2.5 / 97.5 percentiles.
- **Pass rate:** per seed, exact two-sided McNemar on discordant pairs. Pooled: per-task difference (mean over that task's
  seeds of gateway pass - direct pass), 95% CI from 10,000 task resamples, `random.Random(1)`. A significant drop means
  the CI upper bound is below 0.
- Also reported: per-seed cost change with CI, input-token change, user-simulator cost, calls, long (>100k) calls.

## Claim bar (fixed now)

We may write "beyond Induction's published -37% (medium, on a different model)" only if BOTH hold:

1. the pooled agent-cost CI upper bound is below -37%, and
2. the pooled pass-rate difference shows no significant drop.

`analyze.py` prints this as `BEYOND_INDUCTION_37`. If it is false we report the numbers as they are.

## If the gateway passes fewer tasks

List each discordant task (per seed), and what the gateway changed in that task: dedup pointers, results rewritten and
bytes cut (`replay.mjs`, an offline replay of the same Wasm pass over that task's tool results, min 256 B), and folds
(compaction of shell/grep tools; this harness has no such tools, so we expect 0) from the tenant's `/__ops` snapshot.

## Outputs

`RESULTS.md` with the raw results (gzip, keys `***`, tenants withheld, no local paths), a sha256 manifest, the analysis
JSON, spend, quota snapshots, and a Deviations section. Docs-only push to `main`.