← All writing

Build log

The local model wasn't worse. My context pipeline was.

Aug 14, 2026

I benchmarked local models against paid frontier tiers on real document grunt-work, with sealed answer keys and a deterministic grader. The first verdict blamed the model. The recorded runs blamed my inputs.

My benchmark's first verdict said the paid model was clearly better. That verdict was wrong. My documents were born-digital text, and I was feeding them to every model as images.

The numbers move a lot once you fix that: the cheapest paid tier scored 35% reading images and 92% reading the same class of documents as text. Much of what gets reported as a small model's ceiling is, in my recorded runs, an input-pipeline bug wearing a model-quality costume.

The running example is my offload bench, from June 2026. A home model lab asked whether a local model on my desktop GPU could take over document grunt-work (pull these fields, find these values, check this list). That work otherwise goes to a paid frontier model, the big hosted tier. It's the same lab behind the sealed task-difficulty benchmark and what running local models actually costs. Every task had answer keys sealed before any model ran, and grading was deterministic — a dumb script comparing strings, same inputs, same score, no model judgment anywhere near the result.

Why did 35% become 92%?

Because I stopped photographing text. The test documents were born-digital: created as digital text, never scanned. My first run rendered them as images anyway, forcing every model through a needless OCR step (optical character recognition — turning a picture of text back into text). Fed images, the cheapest frontier tier scored 35%. Handed the same class of documents as extracted text, it scored 92%, and every contestant (two local models, two paid tiers) landed in a 90–94% band.

Honest limits, stated plainly. These were single scored runs on small batches, roughly 8–12 documents per condition, so the small spread inside that band is noise, not a ranking. And the 35%-to-92% pair comes from two comparable-but-different document sets, not an identical A/B swap of the same files.

The contamination went past the image mistake, and it was mine. I had let a competing model grade the results, and I had hand-matched the hardest documents to the strongest model. That first verdict deserved to be thrown out, and it was. The rebuilt rule the whole lab now keeps: the thing being tested never grades itself.

Can a small context window do a whole-document task?

Yes, if you stop asking the model to hold the whole document. The local model's context window (how much text it can hold at once) was about a quarter of the biggest test document. The test was a global cross-check: find every entry in a long document's reference list that the body never actually cites. I chunked the document, had the local model extract citations from each piece, and let plain code do the global comparison. Deterministic aggregation: no model in the reduce step.

Result: 9/9 targets recovered, matching the strong frontier tier's single full-document read. It also flagged three false positives, so the output still needs a downstream deterministic or human pass, not blind trust. The durable lesson holds anyway: a context-window limit is a capacity problem, and capacity problems are engineering problems. Chunk, extract, reduce with code.

Does more "thinking" mean more accuracy?

No, and this is the finding I would least have guessed. The local model has a togglable deliberation mode. Turning it ON made extraction worse: 19/20 dropped to 17/20, including a fabricated date — a confident, made-up value it had read correctly with thinking off. On the chunked global task, thinking mode produced literally empty output: 0/9. Not slower and right. Nothing.

A sibling model that cannot turn its thinking off failed the same way, empty on every chunk, and was structurally disqualified from the pipeline for exactly that reason. Meanwhile the non-thinking mode ran at roughly 8–10× the speed as recorded (an approximate ratio across runs, not one controlled timing) at equal or better accuracy.

None of this would have been visible without two boring decisions: grading was deterministic, and deliberation was measured rather than assumed. A vibes evaluation would have told me the thinking mode "seemed more careful." The scorecard says it fabricated data and then went silent.

So should you switch to a local model?

Not on this evidence, and I haven't. These are historical benchmark lessons under stated limits: single runs, small batches, and almost entirely born-digital documents. The skip-OCR lesson applies only to text you already have as text; genuinely messy scans were never part of that comparison. All-in costs (my time, electricity, the tokens a frontier agent spends reading local output back) were not captured, so I won't call the local route "free." The lab's own egress-boundary file blocks that claim from leaving the bench. No route got promoted.

What survives is the mechanism:

• Don't photograph text you already have as text. A worse input reads as a worse model.
• Capacity limits are engineering problems: chunk, extract, reduce with deterministic code.
• More deliberation is not more accuracy. Measure it; don't assume it.
• None of the above is visible unless the grader is sealed and deterministic before the first run.

This page is one slice of how I know when my AI is right. The question I'd put to you: if you re-fed your worst-scoring task as clean extracted text, would your model ranking survive?

I'm Ali — I run real life-and-work admin on AI agents, then check their work in the open. More at /about.

Published under my standards. Found an error? Tell me — corrections go in the corrections log, dated, never silent.

@online{ali2026contextbottleneck,
  author = {Ali},
  title  = {The local model wasn't worse. My context pipeline was.},
  date   = {2026-08-14},
  url    = {https://alidoes.ai/local-model-context-bottleneck/}
}

Caught something I got wrong? Send it directly. Confirmed corrections go in the corrections log.