← All writing

Measuring the cost of a finished result

Planning a small workflow comparison that counts retries and review as part of the work.

I wanted to know whether adding free and local models to my AI workflow was actually helping. There were more ways to route work, but I couldn’t point to a reliable figure for how much usage they saved. That seemed like a reasonable thing to measure before putting more work into the setup.

The question gets complicated once an agent needs help. A model can produce an answer without a paid API call, then leave another agent to diagnose it and redo part of the task. Someone still has to check the result. I wanted those errors and retries included in the comparison, because they affect whether the arrangement is practical.

I asked for a small benchmark plan on October 25. The scope was deliberately limited to planning and a place to keep the work. I didn’t want to use up a lot of credits building an elaborate evaluation system before deciding what a useful test would look like.

The plan centred on a completed result with defined acceptance criteria. It would track whether the workflow finished, what needed another attempt, and whether a fallback model had to take over. Elapsed time and human involvement belonged alongside usage. Measured spend could go in when it was available. Unknown costs needed to remain unknown.

That last part matters when subscriptions are involved. A change in a usage allowance doesn’t directly tell me what a task cost in dollars. Neither does counting how many requests went to a local model. Without the surrounding work, those numbers leave out too much of the process I was trying to compare.

The early telemetry work also needed review. One review caught double-counting in token accounting, and the counter still needed checking against real logs. It would have been premature to build a savings claim on top of it.

At this point I had a way to frame a modest experiment, rather than a set of benchmark results. The next useful comparison would need a small workload that both routes could attempt, the same checks at the end, and a record of the help each route needed to finish.

Written by . Follow via RSS ↗