← All writing

Checking the age of the evidence

A stale error changed a workflow diagnosis, and exposed why I need timestamps and attribution alongside usage numbers.

An AI workflow review told me that one of my tools had exhausted its budget. That didn’t match what I was seeing: I still had most of the credit available.

The error was real, but it was old. The review had picked up failures from several days earlier and treated them as the current state. Once the timestamps were checked against a more recent successful call, the recommendation to add credit was withdrawn.

That was a useful example of how an agent can produce a confident diagnosis from the wrong slice of evidence. The log entry alone wasn’t enough. I needed to know when it happened, what had changed since, and whether the tool was working now.

The same problem appears in usage tracking

I was also trying to make model routing decisions from the results of previous tasks. That requires some way to connect a completed task to the usage it incurred. Without that connection, a cheap model’s rate doesn’t tell me much about the cost of getting useful work accepted.

One attempt added automatic token estimates when recording a task. It used the newest transcript associated with the working directory and calculated the change since the previous recording. The report included an important limitation: two threads sharing the directory could be attributed to the same task.

That made the number a rough estimate. It couldn’t support a precise comparison between models or tell me how much a particular task had cost. The fact that the value could be collected automatically didn’t make its attribution reliable.

The report kept that limitation visible instead of treating the new field as a completed measurement system. Reviewer usage still needed separate handling, and the estimate hadn’t yet been exercised on a real task record.

This is the kind of detail I want to preserve as I improve the setup. A dashboard can make an old observation look current, or make an estimate look exact, simply by displaying a clean number. Before I use it to change routing or spending, I need enough information to understand its scope.

For the budget diagnosis, that meant checking a current successful request. For task usage, it meant keeping the uncertainty attached to the estimate and leaving model cost comparisons open until the records could support them.

Written by . Follow via RSS ↗