← All writing

What my AI logs can actually tell me

Looking back through my workflow history showed how much activity I could count, and how much measurement I still needed.

I wanted to look back at how my use of AI had changed. Most of the useful history was spread across conversations about other work, so I started pulling the workflow decisions into one timeline.

The archive available for this review ran from October 9 to November 16. It contained 732 top-level threads and 1,721 delegated child threads. Those numbers describe the archive. A thread might contain a small question, a long investigation, or another attempt at something already underway. I can’t turn that count into a claim about completed projects.

Every top-level thread received an opening screen, with 64 selected conversations read more deeply. The child inventory was metadata, not a claim that every delegated conversation had been reviewed end to end. Some long messages and run histories were truncated by the tools. That leaves useful evidence for a retrospective, with limits I need to keep visible.

The cost question still has gaps

I also checked the local outcome ledger. It had 174 recorded rows covering 97 ticket identifiers. Of those rows, 134 were marked accepted. Several records can refer to the same ticket or different stages, and the tasks were selected in different ways. That makes the ledger useful for investigating particular outcomes, but it isn’t a fair contest between models.

The clearest result was a missing measurement: none of the 174 rows had both coordinator and review paid-token fields measured. I couldn’t calculate a defensible cost per accepted result from that snapshot. Missing usage doesn’t mean a task was free.

That matters because the model call is only part of the work. A cheaper call can need another attempt or a longer review. A more capable model can also spend time coordinating work that a small script could have done. I want the records to include enough of that surrounding work to make a comparison useful.

Preferences I can explain

The conversations support a more modest account of my model choices. I moved toward Opus for orchestration and Sol for an independent review perspective. I tried smaller or lower-cost options on bounded tasks, and revised those boundaries when the results needed more supervision. Effort settings became another experiment to observe rather than a setting I could assume was best.

I can explain those decisions without claiming a universal ranking or a precise amount of money saved. The history also shows me changing my mind, which is part of why I wanted to write it down.

For the next round of measurement, I want an accepted outcome tied to the exact sessions that produced and reviewed it, with retries and missing data kept explicit. That would give me a better basis for deciding where each model earns its place in the workflow.

Written by . Follow via RSS ↗