01. Two columns nobody keeps
Every run on this project ends with a report, and every report ends with a token ledger: one row per agent run with model, task, tokens, tool uses and wall time. The first report added two more columns: findings that changed code and findings later refuted. Its reasoning was one line: the ratio of code-changing findings per 100k tokens, per agent type, is the number to steer the review budget by.
A finding that changed code is one an implementer actually acted on. Style remarks nobody applied, and claims that turned out wrong, do not count. The second column counts those wrong claims, so a reviewer that is confidently wrong is visible too.
02. What 28 reports said
| Reviewer | Runs | Tokens | Findings / 100k |
|---|---|---|---|
| Conventions reviewer, every task | 124 | 12.0 M | 1.29 |
| Code reviewer, every task | 125 | 10.3 M | 0.89 |
| Generic task reviewers | 23 | 2.4 M | 0.00 – 0.48 |
| One branch-wide review, end of branch | 19 | 3.1 M | 3.4 – 3.9 |
The two per-task reviewers cost 22.3 million tokens for 247 code-changing findings. In the CI story of our Nx migration the gap was starker: 0.31 per 100k for the task reviews against 4.76 for the final review. In one later run the conventions reviewer spent 434k tokens and changed nothing.
The reason is not that per-task reviewers are bad. It is that a task is small and already gated: type-check, tests, lint parity and a boundaries check have run before the reviewer opens the diff. What is left for a reviewer to find is mostly in how tasks fit together, and only a review of the whole branch can see that.
03. And sometimes the reviewer is wrong
Two reviewers once agreed with each other that two lazy-load sites of the same feature share one injector. A throwaway spec, under a thousand tokens, refuted both. One branch review had eight of its findings refuted on inspection. A reviewer's finding is a claim to verify, not a change to apply, and the refuted column is what keeps that honest.
04. The rule, and the re-tiering
A reviewer under 1.0 code-changing findings per 100k tokens over two consecutive runs leaves the per-task dispatch.
- Mechanical checks moved into a script. Translation-key parity, added comments, a missing
OnPush, AI attribution in a commit message: 70 ms in the verification command, zero tokens, never skipped. - Per-task review only for a new shape. The first task of its kind gets the judgement reviewers; repeats of the shape get one batched pass per wave.
- One branch-wide review, always, scoped to the files that carry logic. Each finding is reproduced as a failing test before it is fixed.
The honest part: the metric was defined on day one and acted on five weeks later. Writing the number down is not the same as steering by it. Put the threshold in the rules the same day you add the column.