# tau3 banking_knowledge: pre-registered replication vs Induction, results (2026-11-01)

Pre-registration: `PLAN.md`, `analyze.py` and the driver went to `main` in commit
`0442a3a16c08217e99655b97f78825efa6a502db`. GitHub shows that push at **2026-11-01T12:42:42Z**. The first paid call
(arm 1 key check) was at 12:45:48Z. All four arms ran on 2026-11-01 (UTC), in the planned order.

## Answer

| | value |
|---|---|
| Pooled agent-cost change (170 task-seed pairs, 96 tasks) | **-39.3%**, 95% CI [-49.5%, -27.1%] |
| Seed 300 (96 pairs) | -42.1% [-52.9%, -27.6%] |
| Seed 301 (74 pairs, cap stop) | -35.4% [-50.7%, -16.1%] |
| Pass, pooled | direct 36, gateway 33 of 170 |
| Pass difference per task, pooled | -1.6 pp, 95% CI [-6.8, +3.7] (no significant drop) |
| Pass, seed 300 | 17 vs 16 of 96, exact McNemar p = 1.00 (7 vs 6 discordant) |
| Pass, seed 301 | 19 vs 17 of 74, exact McNemar p = 0.80 (9 vs 7 discordant) |
| `BEYOND_INDUCTION_37` | **false** |

The claim bar is not met. The point estimate (-39.3%) is past -37%, but the CI upper bound (-27.1%) is not below
-37%. So we may NOT write "beyond Induction's published -37%". What the data supports: the gateway cut agent cost by
39% (95% CI 27% to 49%) on this benchmark, with no significant change in pass rate.

The pass rate shows no significant drop (bar 2 holds). Bar 1 fails.

## Cost detail (list price, `summarize.py`)

| arm | seed | tasks | agent $ | user-sim $ | agent calls | calls >100k prompt |
|---|---|---|---|---|---|---|
| direct | 300 | 96 | 4.9879 | 0.4710 | 1,842 | 245 |
| gateway | 300 | 96 | 2.8904 | 0.4830 | 1,965 | 102 |
| gateway | 301 | 96 | 2.8300 | 0.5367 | - | - |
| direct | 301 | 74 | 3.5177 | 0.3623 | 1,373 | 190 |

Seed 301 on its 74 paired tasks: gateway 1,490 calls, 100 long calls. Total prompt tokens (input + cache write +
cache read), paired tasks: seed 300 -16.5% (95.3M to 79.6M), seed 301 -2.8% (63.2M to 61.5M). Most of the saving
comes from fewer calls above 100k prompt tokens (long-tier price is 5x): 435 to 202 over both seeds.

Spend: $16.079 at list price over the four arms (cap $16; the guard checks every 20 s, so it stopped $0.08 past the cap).
Key: bmksglu only. Each key check returned 200. No "Credit balance is too low".

## Validity checks

- Lab quota guard (`ci-quota-guard.py --need 500`) before and after every arm: `used_paused` 0 each time
  (`out/driver.log`).
- Gateway tenants: before, used 0, used_paused 0, granted 2,000 each, dedup on. After: tenant A used 1,969,
  tenant B used 2,049, **used_paused 0 for both**. Both gateway arms are valid.
- Live dedup probe before run 1 (`ddprobe.py`): the same 4-result body went from 1,486 prompt tokens
  (`x-cm-compact: off`) to 973 on both tenants, so the pass was active.
- Every finished simulation ended `user_stop`; no `infrastructure_error`. Gateway `/__ops`: 0 errors, 0 overloads
  (`out/ops-summary.json`).
- No `tau2 run` process was running before arm 1.

## Audit (skeptic check, 2026-11-01, after the results push)

- Re-run: the plan-commit `analyze.py` on the gunzipped `raw/` files (SHA256SUMS OK) gives `out/analysis.json`
  exactly (all fields equal except the file names).
- Push before pay: GitHub push event 12:42:42Z; first paid call 12:45:48Z (`out/driver.log`); first simulation start
  12:45:54Z (`raw/direct-s300`). Task ids, seeds (300, 301), model and agent args in `raw/` match `tasks96.ids` and
  `run-arm.sh`; tau2-bench checkout is `122e2432`.
- Cross-check against the gateway's own meter (`out/ops-summary.json`, whole gateway arms, 96 tasks): `/__ops` says
  $2.9852 (s300) and $2.9137 (s301); `results.json` says $2.8904 and $2.8300. The gateway meter is **3.3% and 3.0%
  higher**, and it counts more requests (1,977 vs 1,965; 2,099 vs 2,045). The extra requests are not explained by the
  probe calls alone. The direct arms have no second meter. If the gateway arms are scaled up by the `/__ops` ratio, the
  pooled change is **-37.4%** instead of -39.3%. Read the headline as "about 37% to 39%".
- Price sensitivity: most of the saving comes from the >100k long-context tier (5x price). At one flat price for every
  call (no long tier) the pooled agent-cost change is **-9.7%** (s300 -13.0%, s301 -4.9%). The tier prices are
  LiteLLM's `model_cost` table for `claude-haiku-5-5`; this run did not compare them with an Anthropic invoice.

## Discordant tasks where the gateway failed (direct passed, gateway failed)

"Pointers / rewritten / bytes" come from `replay.mjs`: an offline replay of the shipped Wasm dedup pass (min 256 B) over
that task's gateway-arm tool results. It models what the last request carried; it is not a capture. Folds
(shell/grep compaction) are 0 in every task: this harness has no tool in `CM_COMPACT_TOOLS` (the tools are KB_search,
discoverable-tool calls, account lookups), so compaction cannot run.

| seed | task | tool results | rewritten | pointers | bytes before -> after |
|---|---|---|---|---|---|
| 300 | task_010 | 7 | 1 | 2 | 43,390 -> 38,729 |
| 300 | task_012 | 9 | 5 | 20 | 85,084 -> 66,724 |
| 300 | task_031 | 6 | 0 | 0 | 43,114 -> 43,114 |
| 300 | task_036 | 10 | 3 | 14 | 125,454 -> 91,449 |
| 300 | task_059 | 33 | 25 | 211 | 440,952 -> 220,763 |
| 300 | task_072 | 17 | 6 | 19 | 135,025 -> 117,776 |
| 300 | task_074 | 30 | 13 | 99 | 320,817 -> 210,785 |
| 301 | task_002 | 16 | 12 | 82 | 186,356 -> 131,892 |
| 301 | task_008 | 8 | 5 | 18 | 108,771 -> 88,650 |
| 301 | task_010 | 8 | 2 | 5 | 34,263 -> 28,528 |
| 301 | task_024 | 14 | 10 | 55 | 170,827 -> 120,522 |
| 301 | task_025 | 14 | 12 | 73 | 204,740 -> 119,153 |
| 301 | task_056 | 23 | 16 | 117 | 309,298 -> 157,431 |
| 301 | task_057 | 46 | 32 | 266 | 508,844 -> 228,381 |
| 301 | task_072 | 18 | 7 | 46 | 176,690 -> 123,654 |
| 301 | task_075 | 22 | 13 | 124 | 269,618 -> 142,508 |

Gateway-only passes: seed 300 task_008, 024, 032, 033, 055, 058; seed 301 task_014, 016, 017, 036, 037, 038, 062.
Notes: task_031 (s300) had no rewrite, so its loss is run-to-run noise. task_008 and task_024 failed on the gateway in
s301 but passed on the gateway in s300. task_010 and task_072 failed on the gateway in both seeds: they are the tasks to
read first if we look for a dedup effect. Whole-arm replay: 39.7% (s300) and 39.4% (s301) of tool-result bytes rewritten.

## Files

- `raw/*.json.gz`: tau2 `results.json` per arm and seed. Keys shown as `***`, tenant ids as `<tenant>`, home path as `~`.
- `raw/SHA256SUMS`: manifest (below).
- `out/analysis.json`: `analyze.py` output. `out/replay-*.json`: dedup replay. `out/driver.log`, `out/guard.log`:
  quota, key check and spend lines. `out/ops-summary.json`: gateway `/__ops` per tenant.
- Re-run: `python3 analyze.py --d300 <direct-s300> --g300 <gateway-s300> --d301 <direct-s301> --g301 <gateway-s301>`
  (gunzip the raw files first).

```
17cb530a11fb91a60670c381491c137386e7dd38e1aadeb994f4a2f96fc495f1  raw/direct-s300.json.gz
36e543c16f864f308a147f1c86293b96596b9da27e564ece447863cb69e5febf  raw/direct-s301.json.gz
b4c9a7d6663f09d1d450c898faa60bf2dbab53f97382f30ad6c53b9da0da8c36  raw/gateway-s300.json.gz
c2e4fe8eaa742c0e5f98b6c03d32222aadffacd7d238c051c542368e150bc3c2  raw/gateway-s301.json.gz
```

## Deviations (changes after the plan push)

1. **Seed 301 direct stopped at 74 of 96 tasks** (cap $16, as PLAN.md expected). Missing: task_080 to task_101. Per
   the fixed rule, seed 301 uses the 74 tasks finished in both arms; no task was filled in, re-run or topped up.
2. **Files added after the push:** `pack.py` (scrub, gzip, manifest) and `ddprobe.py` (live dedup check). Neither
   changes the analysis. `ddprobe.py` made 8 small calls on bmksglu before run 1 (under $0.01, not in the $16.079).
3. **`input_tokens_change` in `analyze.py` counts only uncached input tokens** (+6.1% pooled, a tiny share of the
   prompt). The name suggests all prompt tokens. The total prompt-token change above was computed after the fact and
   is labelled as such. The primary metric is not affected.
4. **Folds come from the tool list, not from a live counter.** The `/__ops` snapshot has no compaction counter. Folds
   are 0 because no tool in this harness is in `CM_COMPACT_TOOLS`.
5. The tool wrapper that ran the driver reported exit code 144 twice. The driver, `run-arm.sh` and `tau2 run` kept
   running (checked with `pgrep`); no arm was interrupted except by the planned cap.
6. **The gateway was deployed twice during the runs.** `0ee919035` at 13:29Z (during direct s300) and `085a47955` at
   13:50Z (during gateway s300). PLAN.md named the deploy at plan time. Both changes touch only the `/try` page path
   (`try-gate.ts`, the try tenant in `caller-resolve.ts` and the quota files, `worker.ts` try wiring); no dedup,
   cache or forwarding code for other tenants changed. `/__version` was not logged per arm.
7. **`ddprobe.py` calls have no logged time or output file.** No file in `out/` or in the local run folder records the
   1,486 -> 973 numbers above, so a reader cannot check them from this folder.