alidoes.ai

All writing

Post-mortem

Why did my AI dashboard say a case was settled when it wasn't?

An AI summarizing a legal folder merged two counterparties and promoted mediation paperwork into a settlement. The most dangerous wrong answer is the one that asks for no attention.

Key findings

  • An AI dashboard merged two different counterparties into one and read mediation paperwork as a finished settlement; later trial and judgment records proved the shortcut wrong.
  • The fix is an evidence hierarchy: filed records outrank drafts, drafts outrank correspondence, and no document below the required tier can change a case's state.
  • Forcing every case to an outcome was the real design flaw; an honest UNKNOWN state is what makes 'settled' believable.

My AI dashboard marked a legal matter I was tracking as settled. It wasn't. The fix: rank every document by evidence tier, and let unproven outcomes stay UNKNOWN.

"Settled" is the most dangerous status an AI can put in front of you. Every other wrong answer creates friction someone might notice. A settled case asks for nothing: no deadline, no follow-up, no review. The error doesn't just sit there. It actively waves you off.

This is the post-mortem of how my dashboard produced that word, and the rebuild that replaced it.

How did an open case become "settled"?

Two shortcuts, stacked. First, the AI summarizing the case folder merged two different counterparties (the parties on the other side of a matter) into one. Once that happened, documents from one thread were read as updates on the other, and the folder looked far more resolved than it was.

Second, it promoted mediation paperwork into an outcome. Paperwork about trying to settle was read as having settled. Later trial and judgment records proved the shortcut wrong: the case the dashboard had retired was still live.

Neither mistake is exotic. A hallucination (a confident, made-up answer) would have been easier to catch. These were plausible readings of a messy folder, which is exactly why they survived a glance.

The verdict I owe myself is harsher than the one I owe the model. I built a status field that forced every case to an outcome, then acted surprised when a summarizer obliged. Ask an AI for a verdict on every folder and it will find one. That's a design flaw before it's a model flaw.

Why does an evidence hierarchy fix it?

Because documents are not equal, and my dashboard had been treating them as if they were.

The rebuild ranks every document in the folder by tier. Filed judgments and other filed records sit at the top: they say what actually happened. Drafts and mediation paperwork sit in the middle: they say what someone proposed. Correspondence sits at the bottom: it says what someone claimed on a given day.

Then one rule does the work: a case's state may only change on the strength of a document at the required tier. A letter can suggest a settlement exists; only a filed record can conclude one. Under that rule the original failure is unreachable. Mediation paperwork is middle-tier evidence, "settled" is a top-tier state, and the promotion is simply not allowed.

I also rebuilt the status field itself as a small state machine (a fixed list of allowed states, plus explicit rules for moving between them). It replaced a free-text label the summarizer could set to anything it liked.

Two readings of one case folder: the shortcut versus the evidence hierarchy One folder, two readings Case folder one matter, mixed papers filed judgment · tier 1 mediation draft · tier 2 correspondence · tier 3

THE SHORTCUT → Any document can set the status → SETTLED (wrong) a settled case asks for nothing

THE REBUILD → Evidence hierarchy filed > draft > letters → OPEN next action known unproven outcomes stay UNKNOWN

One folder, two readings: the shortcut let any document set the case status and produced a false "settled"; the evidence hierarchy (filed records outrank drafts, drafts outrank correspondence) keeps the case open with a known next action and honest UNKNOWNs. Diagram source: this page; maps 1:1 to the rebuild described in this post.

What about outcomes the records don't prove?

They stay UNKNOWN, on purpose. That's the part most dashboards refuse to do.

Rebuilding from the hierarchy did not resolve everything. It restored the state to open, made the next action clear, and left some outcomes explicitly UNKNOWN, because no document at the required tier settled them. That reads like a weaker answer — it's a stronger one. A dashboard that can say "I don't know" is the only kind whose "settled" means anything. It's the same principle that runs through everything else I publish about verification.

The merged counterparties fell out of the same rebuild. Reconstructing state per matter means every document has to be tied to one matter and one party before it counts as evidence for anything. A document that can't be placed doesn't get a vote; it gets a question mark.

One honesty note, because this is the most sensitive story I publish: details are generalized to protect the people involved. No parties, no places, no identifiers, no amounts, no dates, nothing about strategy. "A legal matter I was tracking" is all the specificity you get, and the pattern — merge, promote, contradict, rebuild — is exactly what my incident ledger records.

So here's the question I'd put to your setup: which status in your dashboard, if silently wrong, would ask for no attention at all, and what's the weakest document currently allowed to set it?

Method & data

Method: one incident on a real matter-tracking dashboard — an AI status shortcut later contradicted by filed records, and the evidence-hierarchy rebuild that replaced it, generalized
Data: patterns and methods only; no names, dollar figures, addresses, case identifiers, or confidential content · Last checked: 2026-08-14

How this was made

AI-drafted, adversarially checked, human-directed. My AI assistant wrote this from the system's own records — the incident's ledger row, which states the pattern without any of the particulars. A separate AI session then tried to break every claim against that record, and automated privacy and readability gates ran before publish. I direct this pipeline, own every boundary in it, and audit published pages on a rolling basis — if you find an error, tell me and it goes in the corrections log, dated, never silent.

I'm Ali — I run real life-and-work admin on AI agents, then check their work in the open. More at /about.

Published under my standards. Found an error? Tell me — corrections go in the corrections log, dated, never silent.

Cite this

@online{ali2026wrongstate,
  author = {Ali},
  title  = {Why did my AI dashboard say a case was settled when it wasn't?},
  date   = {2026-08-14},
  url    = {https://alidoes.ai/ai-legal-dashboard-wrong-state/}
}

Caught something I got wrong? Send it directly. Confirmed corrections go in the corrections log.