addAI.dev·Field NotesView as Markdown

By Jay·October 2026

Adventures in Local LLMs: How We Made Our AI 2.3x Faster for 81 Cents

#AITools#TechDeepDive#Benchmarks

You know what is good at tweaking a local LLM? An LLM.

We run a coding agent, Pi Agent, every day against a model on local hardware (the same setup we measured in Where Your Context Window Actually Goes). It costs nothing per token and nothing leaves the building. The software that serves the model exposes dozens of settings, and the right values depend on the exact graphics card and model. So we put AI on the job. Claude Opus 5.5 orchestrated the tuning. Pi Agent's real sessions supplied the workload, and Qwen3.6 did the heavy lifting on the physical hardware: every replay and every stress test ran on the server itself. We built the tests together with Opus and supervised every run, the same way we build agents for clients.

The result: the same model on the same graphics card now reads its prompts 2.3x faster, for 81 cents of electricity.

The machine and the problem

We are currently working with a 24-core AMD Threadripper 3960X, 256 GB of RAM and one NVIDIA RTX 3080 with 10 GB of video memory. The model is Qwen3.6-35B-A3B-UD-Q4_K_XL (Qwen3.6-35B-A3B in Unsloth's UD-Q4_K_XL quantization), served by llama.cpp with a 256K-token context window. Compressed to roughly 4-bit precision (most weights squeezed into about 4 bits, with higher precision kept where it matters), it takes about 22 GB.

A 22 GB model does not fit on a 10 GB card. This one still runs well because it is a mixture-of-experts model: of its 35 billion parameters, only about 3 billion do any work for a given token. So most of the expert weights sit in system RAM and the CPU computes them there, while the GPU handles the parts every token needs. Where each piece lives is decided once, at startup. Tuning is mostly about how much of that split you can move onto the GPU before the 10 GB runs out.

How we measured it

A speed test on a short chat prompt says little about an agent that sits at tens of thousands of tokens of context and ends almost every turn in a tool call (a request to run a command or open a file). So we measured on real work:

1. Record. Three real Pi Agent sessions went through a small logging proxy: 71 turns of actual agent traffic.
2. Replay. Every candidate configuration got the same 71 requests (same prompts, same random seed), timed by llama.cpp's own counters.
3. Score. Speed, memory use and whether every tool call the model made was still valid JSON that matched its schema.
4. Stress. The finalists also had to grow a real session to the full 256K tokens, recall facts planted deep in it, handle an image and fail cleanly when pushed past the limit.

A faster config that breaks tool calls is a slower agent.

Temperature and power loggers ran alongside the tests, plus a meter at the wall socket.

The result

Baseline is the configuration we had tuned by hand months earlier, one setting at a time, re-measured on October 14. "Now" is the one the server has run since October 15.

Same 71-turn replay for both. Single runs; writing speed varies about 1-2% run to run, and only 3 turns reached 120K+ tokens. Tool-call totals count the turns where the model chose to call a tool, which varies slightly between runs.

The 2.3x is prompt reading (what engineers call prefill), not writing speed. Writing improved about 6%. For Pi Agent, reading is the part you wait on: every file it opens and every command result it gets back has to be read before the first word of its reply comes out.

A 15,000-token tool result used to mean 29 seconds of silence. Now it is about 13.

Three changes did it

1. Two layers of experts moved onto the GPU. That gained 6% reading and 7% writing speed on its own; the later changes gave a little of the writing gain back. The sweep tried three layers first, hit out-of-memory at startup and the test script fell back to two.
2. The image encoder moved to the CPU. That freed about 1.1 GB of video memory and fixed a crash where any image killed the server on the tuned layout. The cost: a large screenshot now takes about 18 seconds to read instead of about 10, with images capped at 2,048 tokens.
3. Bigger batches and a smaller conversation cache. Reading the prompt in larger chunks is what doubled the speed, and it only fits because the conversation cache moved from 8-bit to 4-bit numbers, which freed the room. Recall deep in a long session held up: 17 of 18 recall checks passed (across three repeat runs) for both settings.

What the testing turned up

The numbers are the payoff. The more useful part is what happened between them.

Finding the edge. When a configuration ran out of memory partway through a run instead of at startup, Opus read the crash out of the server log, changed the test script to treat that as "try the next option" and restarted the sweep.

A new feature, slower. llama.cpp had added a GPU cache for CPU-side experts a week earlier. On this card it cut writing speed by 44%: the GPU worked harder and ran hotter while doing less. A bigger card that can hold more of the cache may see the opposite.

Safe defaults. llama.cpp can size the split itself, and that came out about equal to our starting point because it keeps 1 GB of video memory spare. The first gain came from spending that margin, which is exactly why the deep-context stress test had to prove it was safe.

The real danger. Video memory stayed flat from 123K tokens all the way to the top of the 256K window, because llama.cpp reserves the conversation cache in full at startup. Then a single JPEG crashed the server: the image encoder asks for GPU memory only when a picture arrives.

The context was never the danger. The image was.

Dead ends. Four other speed settings (thread count, two variations on how the model guesses ahead and dropping the reasoning it keeps from earlier turns) all gained nothing. Two batch sizes did not fit at all with the 8-bit cache. Each of those is a setting nobody has to wonder about again.

The 81 cents

The wall meter recorded 5.05 kWh from the afternoon of October 13 to the evening of October 15. Priced at Ontario's Time-of-Use rates, with the split across peak and off-peak hours estimated, that is about $0.81 in Canadian dollars (between $0.49 if it had all been off-peak and $1.03 if it had all been on-peak).

That figure is the electricity energy charge. It does not include delivery charges or tax on the bill. It also covers idle time and setup work in the same window, so it slightly overstates the tuning alone. The tuning itself ran on a Claude subscription we already pay for, on hardware we already have, so electricity was the only new cost.

Between jobs the server now sleeps: it suspends after 30 idle minutes with the model still loaded, and Pi Agent wakes it over the network when it needs it. It came back answering in 20 out of 20 test cycles.

What this means beyond one local server

The insight was in the setup: three things that let an AI agent grind through patient work safely.

• a recording of the real workload
• a scoreboard it cannot argue with (speed, memory and valid tool calls, measured by the server itself)
• guardrails that stop it safely (an out-of-memory fallback and a temperature cutoff for long unattended runs overnight)

With those in place, the tuning came down to patience: dozens of configurations, each replayed against real work, each checked for broken tool calls and memory headroom, with the dead ends written down. Our hand-tuned baseline took weeks of exactly that kind of iterating.

If you find yourself doing something repetitive, an AI is better at it than you are.

It is tireless, and far more thorough than you will be by the twentieth run.

The same three carry over to a business task. Take an agent that drafts replies to booking requests. The recording is last month's real requests. The scoreboard is whether each reply matches your calendar and your price list. The guardrail is that nothing sends without your approval.

That is how we build agents for clients: measured against the work they will actually do, with the safety nets in place before anything runs on its own.

Get in touch