Inconclusive

My cost counter showed 513,192 dollars.

The real number was 3,916.

Let me say it right away: this isn't a bill. I'm on a flat plan. It's what my sessions would have cost at API rates, token by token. But 513,000, even as an equivalent, wakes you up.

The bug was dumb. Codex logs give cumulative counters: on each message, the total since the session began. My reader summed them as if each line were a separate message. A single session went from 1,330 dollars to 243,457.

I would have believed that number if it had been ten times smaller. And that's exactly the problem.

Why I measure#

Every agent tool promises the same thing: fewer tokens, cheaper, faster. Foundry runs each role on the model it needs, so on paper I should be saving.

On paper. I wanted to see.

The rule of the game, the same for every campaign: the threshold is written before the measurement, and frozen. After that, it doesn't move, and the result isn't reinterpreted.

The pilot: 7.42%#

First campaign in August. Ten real tickets, with role-based routing. The stated goal: 40% less than if everything ran on the biggest model.

Result: 95.76 dollars against 103.43. That's 7.42%.

Missed by a mile. And I can't even claim that 7.42%: it's the same work, re-priced afterwards at the big model's rate. Nothing says the big model would have needed the same number of round trips. It's not a measured saving, it's cross-multiplication.

Decision: I keep the routing. And I announce nothing.

The local model dream#

This is the one I care about. I have a Mac with 64 GB of memory. The idea: a local model pre-chews the work — summarize a diff, sort logs — and the paid model gets less to read.

Four candidates: Qwen, gpt-oss, Devstral, Gemma. Thresholds frozen in advance: 100% of answers in the right format, and at least 40% less input.

Result, for each of the four: 0 out of 12.

Measured input reduction: 0.32%.

And the worst part is that it says nothing about their intelligence. Digging in, the failure came from the format: every response carried a metadata field that my deliberately strict reader refused. The campaign never measured the quality of what the models wrote. It says so itself in its conclusion.

A detail that stings: on my first version of the test, with a prompt written for the occasion, two candidates scored 12 out of 12. It was going through the product's real path that dropped them to zero.

In between, an entire version died before it began, on an external drive that had stopped responding.

The 48 false positives#

In that same campaign, my classifier output "forged citation accepted" 48 times. Forty-eight security alerts.

None was real. There was nothing to classify, since no candidate had produced a valid response. The classifier, frozen before the measurement, returned "true" on nothing.

I could have fixed it and rerun. I kept the raw output, untouched, and published a correction next to it. If I allow myself to retouch a result that bothers me, I'll allow myself to retouch one that suits me.

October: missed by 0.93%#

I tried again, differently. Five versions in a week.

First the ambitious one: a local model handling a ticket on its own. Five candidates, six tasks. 0 accepted out of 30.

Then the modest one: the local model explores the repository, and the paid model implements with what it found. Ten tasks, done twice, with and without the local scout. Five accepted on each side. A tie on quality.

That left the question: did the paid model consume less? Frozen threshold: it had to come down to 85% of the reference consumption.

It came down to 85.79%.

Missed by 14,485 tokens per task. 0.93%.

I'm keeping the cloud. The threshold was written beforehand, and a threshold you round off when it's inconvenient is no longer a threshold.

All the more since, looking closer, it was even less flattering. Not all tokens cost the same. Weighted by price, the ratio goes from 0.86 to 1.03. With the local scout, it cost slightly more. "Fewer tokens" depends on what a token weighs.

The test that refused to answer#

My favorite. To validate a model change, a small trial: the agent has to read a file and copy out its value.

Not conforming. The agent refused to copy it.

The file was called token.txt. And my own instructions forbid displaying secrets. The model did exactly what I usually ask of it.

The verdict stayed "not conforming". The second trial, with a neutrally named file, passed. Both are in the record.

What I'm left with#

Three reflexes.

Unknown is not zero. When a piece of data is missing, the cell stays empty. Not "0 dollars". Empty.

Threshold first. Written and frozen before the measurement. Otherwise you always find a reading that suits you.

Keep the raw output. The wrong result stays in the record, with its correction beside it.

Is it frustrating? Yes. Honestly, yes. I still haven't managed to integrate a local model that is relevant, and at the right level. I have a machine that could, models that get better every month, and a folder full of "inconclusive".

But I know why. I know to within 0.93% where the line is. And the day a model crosses it, I'll have the protocol to see it — and to believe it.

In the meantime, the only gain I can state isn't in any of my tables: what I ship is better. That one I haven't measured. I see it at every merge.

— Pato