← All writing

A local model test I decided to stop

One recorded attempt, an uneven comparison, and the decision to park further experimentation.

On October 27, I asked for a local model run to compare with two Gemini attempts at the same creative brief. I wanted timing as well as output. Getting something generated locally was interesting, but I also wanted to see how practical it would be to use regularly.

The recorded local attempt used qwen3-coder:30b. It produced code for a short motion piece, then went through type checking and an attempted render. The result stopped short of a usable video.

Recorded measure Result
Generation wall time 307.9 seconds
Model loading, included in generation time 151.5 seconds
Output tokens 2,419
Type checking errors 6
Rendered video None

The loading time is included in the generation figure, so adding those two rows would count it twice. These are observations from one attempt on my setup. They don’t establish what another local model or a warm model would have done.

There was also a more basic problem with the comparison, one I’d already considered when planning how to measure a finished result. The Gemini runs had an agent feedback loop that could work through issues. The local run received one attempt. Starting from the same brief didn’t make the whole process equivalent. A comparison of the final outputs would include the effect of that extra iteration, and I couldn’t separate it from the model choice.

I could have spent more time making the experiment fairer. That would have meant deciding how many corrections each route received and recording the extra time needed to reach a usable result. Instead, I questioned whether continuing was worth it for the work I wanted to do, then parked the experiment.

I’m keeping the failed result in this account because it’s more useful than quietly dropping it from a comparison. It shows what happened in that specific attempt, including the point where the output failed the checks. It doesn’t support a general ranking of local models against Gemini, or a claim about money saved.

The unresolved question was whether an equivalent local feedback loop could finish the task within an acceptable amount of time. I left that question unanswered when I parked the experiment.

Written by . Follow via RSS ↗