alidoes.ai

All writing

Post-mortem

Is a free local AI model actually cheaper? Mine wasn't

I ran local models on my own GPU to make AI automation cheaper without trading away trust. A full audit found neither current quality proof nor a single measured cost, so the lab was rebuilt to fail closed.

Key findings

  • On 2026-07-24 my route register held zero promoted routes: one local route needed revalidation, two were not qualified, one was retired.
  • The all-in cost ledger tracks eight rows per route, from generation to human time. On audit day every applicable row read NOT CAPTURED.
  • Every route now fails closed: unapproved, with no writes to trusted records, until quality proof and full cost are measured on the current runtime.

No. The "free" local pair failed its own audit: no current quality proof, every downstream cost row unmeasured. My route register now holds zero promoted routes until the evidence exists.

A model that costs nothing per call is the easiest place in your stack to hide costs, because nobody audits what they already believe is free. I know because I ran the audit and lost every route I had.

The running example is my model lab: an Ollama server on my own machine, version 0.32.3, carrying gemma4:12b, qwen3.5:9b, and a small vision model. The plan was the obvious one: route bounded jobs to the local pair and let automation get cheaper without trading away trust. The lab's route register is the file that records which model is approved for which job, and what evidence backs the approval.

On 2026-07-24 I audited that register. Qualification state: stale. Promoted routes: an empty list. No local-model schedule was even live when I checked.

What did the audit actually find?

Four routes, none approved, each broken in its own way.

bounded-document-extraction.local-gemma4, the flagship, sits at revalidation-required. Its old benchmark receipts never bound the current runtime and model digest (the digest is the model file's fingerprint; change it and you are testing a different model). The Ollama version had moved since those runs, and the model's exact thinking behavior was never revalidated afterward. Evidence about a system that no longer exists is nostalgia, not proof.

web-source-discovery.local was worse, and this is the judgment I own: run 4 produced six human rubric files and zero signed ones. I built the grading forms and never graded. That is not an eval; it is a prop. The strict fabricated-citation rate failed the zero-fabrication gate, and in run 5 one output cited a URL it never fetched. The audit that did exist proved fetch provenance (that a page was really retrieved), not that any claim was supported by it. A separate adversarial review, one that tries to break a design instead of confirming it, blocked the route's live-fetch loop outright over DNS-rebinding, proxy, redirect, and private-query leakage risks.

web-synthesis.local carries the limitation I now quote at myself: agreement between model outputs is not evidence that a claim is true. Two local models nodding at each other is consensus, not correctness.

The fourth route, a fixed local pair for recurring demand mining, is retired. Its only saved smoke receipt was incomplete and non-qualifying.

Where does a "free" model hide its costs?

In the rows nobody prices. The register keeps an all-in cost ledger for every route, eight rows deep: generation, retrieval, ingestion, retries, verification, adjudication, human time, local compute. On audit day, every applicable row on every route read the same value: NOT CAPTURED.

Generation is the only row people mean when they call a local model free. The other seven are where local quietly gets expensive. Retries pile up when a small model misses the schema (the exact output structure a job demands). Then come verification passes over its output, adjudication (deciding which of two conflicting outputs to trust), the minutes of human review, the cleanup after a failure, and the compute itself. I had measured none of them. "The local route is cheaper" was never a finding in my lab. It was a mood.

The honest part: I still cannot tell you what my local pair costs all-in. Those rows stay empty until a governed run fills them, and this page will not pretend otherwise. What changed is that the system now refuses to treat an empty row as a cheap one.

What does it mean for a route to fail closed?

Fail closed means missing evidence reads as no, never as probably fine. After the rebuild, every route in the register carries canonical_write_allowed: false (canonical records are the single source-of-truth files the system lives by) and a null approval. Promotion needs two gates passed on the current runtime: quality proof bound to the exact model digest, and a cost ledger with all eight rows measured. Until both exist, the work falls back to an already-governed route.

Route promotion gate: quality proof and full cost, or fail closed Route promotion gate: fail closed by default Candidate route local model + task → Gate 1 · Quality proof? bound to runtime + digest yes → Gate 2 · Full cost? all 8 cost rows measured yes → Approved enters the register no ↓ no ↓ Missing evidence → fail closed unapproved · no canonical writes work falls back to a governed route
A candidate route must pass two gates on the current runtime: quality proof bound to the exact model digest, then a fully measured eight-row cost ledger. Missing evidence at either gate fails closed: unapproved, no canonical writes, work falls back to a governed route. Diagram source: this page; maps 1:1 to the route states and all_in_cost rows in _route-register.json.

That inverts the default most local setups run on. A route is not approved until proven bad; it is unapproved until proven good. The models are still on disk (gemma4:12b alone is about 7.5 GB) and the lab still exists. What changed is the burden of proof, the same rule the wider system runs on: you cannot stop an AI from making a mess, but you can guarantee the mess never stays invisible. A free model that skips this rule is not cheaper. It is unpriced.

If you run a local model today, try to fill those eight rows for one real job. Which ones can you actually write a number in, and what does your setup do with the route while the rest stay empty?

Method & data

Method: the model lab's route register as audited on 2026-07-24 — four local routes, their qualification states, evidence lists, and all-in cost ledgers, generalized
Data: patterns and methods only; no names, dollar figures, addresses, case identifiers, or confidential content · Last checked: 2026-08-14

How this was made

AI-drafted, adversarially checked, human-directed. My AI assistant wrote this from the system's own records — the model lab's route register (`_route-register.json`), its per-route evidence and limitation entries, and the deterministic control checks (same inputs, same result) that gate it. A separate AI session then tried to break every claim against those records, and automated privacy and readability gates ran before publish. I direct this pipeline, own every boundary in it, and audit published pages on a rolling basis — if you find an error, tell me and it goes in the corrections log, dated, never silent.

I'm Ali — I run real life-and-work admin on AI agents, then check their work in the open. More at /about.

Published under my standards. Found an error? Tell me — corrections go in the corrections log, dated, never silent.

Cite this

@online{ali2026localcost,
  author = {Ali},
  title  = {Is a free local AI model actually cheaper? Mine wasn't},
  date   = {2026-08-14},
  url    = {https://alidoes.ai/local-ai-model-hidden-costs/}
}

Caught something I got wrong? Send it directly. Confirmed corrections go in the corrections log.