What we learned measuring AI agents in CI

November 2, 2026CI6 min read

Measuring cost for an AI agent in CI is easy to get wrong. We got it wrong more than once, and some of those mistakes cost us a day of results. This post lists what we learned, in the order we learned it, and the rules we now run by. Each rule has a file you can open.

TL;DR

• Real repos rerun the same job about every 2 minutes, so a test must rerun too.
• A paused tenant turned our proxy into a plain pass-through with no warning.
• One wrong cache-read price made our own meter read high.
• We now write the plan first, judge with another model, price from provider usage and balance the arm order.

1. Real repos rerun the job every two minutes

Our first tests ran each job once, alone. That is not how CI works. We looked at seven days of real runs. Issue triage in anthropics/claude-code ran 985 times, median gap 1.9 minutes. PyTorch's PR review ran 270 times at the same gap. LobeHub's ran 466 times, 12 minutes apart.

A cache that lives five minutes is warm for all three of the first two. A test that runs one job alone sees only a cold cache and misses the main saving. So we run jobs in sequence, close together, and report the warm runs.

2. A paused tenant turned the proxy into a pass-through

On 2026-10-31 our lab tenant used its 1,000 free requests. The gateway paused it at 10:29:57 UTC. A paused tenant is not blocked: its requests go through unchanged, and the run shows no error. The gateway arm of our first area tests ran as plain pass-through for 1,518 requests. The numbers looked fine, but both arms were the same.

We caught it by looking at request 1. On a working run it is 30% to 42% smaller. On the paused run it was the same size as the direct arm. We threw away that day of results and ran it again on a funded tenant. The notes are in the lab: clean re-run.

The rule now: a quota guard reads used_paused before and after every round. If it grows, the round is invalid. In the latest round it stayed at 0 before and after every phase. The tau3 plan has the same rule in its text (PLAN.md).

3. A wrong cache-read price

Sonnet 5.5 had no row in our price table, so the table fell back to Sonnet 5 by name. That model bills a cache read at $0.20 per million. Sonnet 5.5 bills $0.10. Every read was priced at double.

We found it because our meter and the provider's price page disagreed. We added the Sonnet 5.5 row, and the price page now sets the table. The lesson: when a meter and its source differ, find out why before you trust either.

4. Write the plan before the runs

Choosing the tasks, the model, the stop rule and the analysis after you have seen the numbers lets you find a result. So we push a plan file first. The CI plan was pushed at 2026-11-01 23:39:27 UTC. The first paid run started at 23:40:03 UTC. The tau3 plan went up at 12:42:42 UTC and its first paid call was at 12:45:48 UTC. Anyone can compare those times with the commit history: PLAN-r2.

Tasks come from a seeded draw, so we do not pick them by hand.

5. A judge from a different model

For jobs with no test to run, a model has to judge the output. If the same model family judges its own work, it tends to like it. So an Opus 5.5 judge scores the Sonnet outputs. It sees the two outputs in a random order, set by a seeded coin, and it sees no tool or arm names. For fix jobs we skip the judge: the task's failing tests either pass or they do not.

6. Cost from provider usage, checked against our ledger

A client's "total cost" is a number the client computed. We do not use it as the source. We take the token counts the provider returned, apply the list price, and count every model in the run, side calls too. Then we check two ways.

• Our price times the client's usage against the client's total.
• Our price times the gateway ledger's usage against the ledger's own total.

The target is a gap under 3%. In the latest round the mean gap between client and ledger was 0.86%. Seven of 119 gateway runs were over 3%, and each one is listed with its cause in RESULTS-r2.

7. Balance the arm order

Which arm runs first matters, because the first run may warm a cache the second one uses. Two accounts matter too, because each has its own rate limits. So in each lane the first half of the tasks run the gateway arm on one account and the direct arm on the other, and the second half swap. Arm order is set by a hash of the task, so exactly half the pairs run gateway first.

The rules, in one list

• Run jobs close together, as real repos do.
• Check request 1 size on every round. It shows a pass-through at once.
• Read used_paused before and after.
• Push the plan before the first paid run.
• Judge with a different model, blind.
• Price from provider usage, and reconcile with the ledger.
• Balance accounts and arm order.

Run it yourself

Every file above is public. The template runs the same jobs in your repo, with your own keys.

curl -fsSL https://claude.ai/install.sh | bash
npx @context-mode/cli login
npx @context-mode/cli ci init
npx @context-mode/cli ci key create --repo owner/name --gh

Open the try page, copy the lab template, or read the reproduce guide.