← All writing

Measuring why local AI felt slow

A slow greeting pushed me to separate model loading, instruction processing and queued work from response generation.

A local AI setup should be able to answer a greeting without making me wonder whether anything is happening. On October 15, that simple interaction became a useful way to investigate my configuration.

I wanted a local stack I could interact with directly, and I chose to keep the initial tools separate for simplicity. I also asked for local-only operation. Having a cloud fallback quietly handle a failed request would have made it harder to know what the local setup could do.

The first reports looked encouraging once a model was loaded. Then I tried the conversational workflow and found it too slow. A short direct response and a response through an agent application were taking different paths.

The investigation found several things inside the wait. Loading a model from a cold start took substantial time. The application also supplied a large set of instructions before a short greeting. Other measurements included work queued from an older session. One long result explicitly included both queued work and cold loading, so it couldn’t stand in for normal response speed.

Repeatedly switching models added another problem. The assistant revised the setup to keep one model resident for ordinary conversation and use another for a whole coding task, avoiding repeated swaps. It also worked on avoiding unnecessary tool activity during greetings. Those changes focused the experiment on the slow interaction I had reported.

The assistant later reported a much shorter warm greeting, with no tool calls. I treat that as a useful probe of this setup. It doesn’t tell me which model is generally fastest, and it isn’t a fair comparison with a cold start or a coding task. Automatic delegation still needed verification after a failed tool call.

This changed what I needed from a timing result. I wanted to know whether the model was already loaded, what instructions it received, and whether other work was waiting ahead of it. Without those details, I could spend time changing models while leaving the cause of the delay untouched. The next check had to run through the application I intended to use, under conditions I could describe.

Written by . Follow via RSS ↗