← All writing

Build log

A URL in the fetch log is not a citation

Aug 14, 2026

My web-research harness gave every answer a green GROUNDED stamp if it had fetched the cited page. The stamp measured retrieval, not support — and the gap was enormous.

My benchmark harness stamped answers GROUNDED whenever a cited URL appeared in its own fetch log. A strict re-audit priced that stamp: up to 89% of the citations were fabricated.

The design was not exotic. Every "sources" footer under an AI answer makes two separate claims: this page was retrieved, and this page backs the sentence beside it. Most citation tooling I have seen, mine included, checks only the cheap one.

What was the harness actually checking?

The setting is run 4 of my model lab's web-research benchmark: a preregistered four-pass test, meaning the tasks, grading rules, and pass criteria were frozen and signed before any model ran. Local models answered research questions through a caller-controlled search/fetch loop. My harness, not the model, executed every search and every fetch, and it kept the full trail: exact queries, the result menus the model saw, each fetch attempt, and the text that actually came back.

That trail is real value. It gives every answer complete fetch provenance — a verifiable record of where each retrieved page came from. The mistake was what I did with it next. The grader took each URL a model cited and looked for it in the harness's own fetch log. If the URL was there, the answer got stamped GROUNDED (grounding is the trade's word for tying an answer back to real sources).

Read that check again. It proves the model cited pages from inside the loop rather than inventing URLs outright. It says nothing about whether any cited page supports the claim sitting next to it.

How bad was it under the green stamp?

Bad. A stricter re-audit of the same receipts counted fabricated citations — cited sources that do not genuinely support the claim they decorate. One local model hit 89% with its "thinking" mode on and 87% with thinking off. A second model came in at 42%. The stamp had been content with all of it.

Two boundaries on those numbers, both deliberate. They measure a failure mode, not a model ranking: I am not naming the models, and the run cannot rank anything anyway, for reasons below. And they are historical: the runtime version the receipts bind to is no longer the one installed, so nothing here describes any current setup.

The critical judgment lands on me, not the models. The check was lazy in the most seductive way: it verified the part that was cheap to verify and let the green label imply the rest.

What does a citation actually have to prove?

The lab's correction became a terminology rule, not a patch. URL presence proves fetch provenance only. Before an answer earns the word "cited," a separate supports-claim check must verify three more things: the page actually backs the adjacent claim, the source is authoritative, and the model read it correctly. A search snippet is a lead, not evidence. Two models agreeing does not help either — correlated models repeat the same error, so agreement is not proof.

The receipt folder shows the gap in its own filenames. Supports-claim verdict files exist for passes one and two; for passes three and four they were never produced. The human blind rubrics (score sheets a person fills in with model identities hidden) sit blank. So the run is recorded as non-qualifying, no matter how fluent the answers looked. Failing the zero-fabricated-evidence gate is decisive on its own; prose quality never gets a vote.

Why keep a run you can't use?

Because it measures one failure mode precisely, and that is worth keeping. The frozen receipts show that local models can drive a caller-run search/fetch loop, and they show exactly how hollow a self-awarded green label can be. This is the same shape I keep finding elsewhere: a green check sitting on top of missing records asserts less than its name implies, and nobody reads the fine print until it burns them.

What the run does not do: rank models, prove any local route reliable, or approve anything for production. The live-fetch runner itself is now execution-blocked, partly because a search query can carry private context out of the workspace. Any future attempt starts from an operator-curated, pre-fetched packet of public pages, with the supports-claim check wired in before the first answer is graded. The work sits in the same lab as my local-model cost accounting, and it feeds the standing question behind how I know my AI is right: what does this green label actually assert?

So: if your pipeline prints sources under its answers, go read the code that awards that badge. Mine checked membership in a log and called it grounding. What, exactly, does yours check?

I'm Ali — I run real life-and-work admin on AI agents, then check their work in the open. More at /about.

Published under my standards. Found an error? Tell me — corrections go in the corrections log, dated, never silent.

@online{ali2026fetchednotcited,
  author = {Ali},
  title  = {A URL in the fetch log is not a citation},
  date   = {2026-08-14},
  url    = {https://alidoes.ai/fetched-is-not-cited/}
}

Caught something I got wrong? Send it directly. Confirmed corrections go in the corrections log.