← All writing

Post-mortem

My AI designed, ran, graded — and won — its own benchmark. I threw the run out.

Aug 14, 2026

The first run of my task-difficulty benchmark was won by the model that designed it, ran it, and graded it. This is the post-mortem: the discard, the sealed rebuild, and the sequel I refused to publish.

A model that scores its own work will always find a way to like it. Mine designed, ran, graded, and won my benchmark, so I threw the entire run out.

The benchmark in question is my task-difficulty benchmark: worker models extract fields from my own real paperwork, and a manager model sizes up each document before the run. Its first version produced a scoreboard I could not use. This is the post-mortem of why, what the sealed rebuild changed, and the follow-up story I refused to publish.

How does an AI end up grading its own benchmark?

By default, not by malice. I asked the manager model (the orchestrator: the model in charge, which sizes up the work, hands documents to worker models, and judges what comes back) to build me a benchmark on my own paperwork. It did everything I asked. It designed the test. It ran the workers. It graded the outputs. And when the numbers landed, its own calls came out looking great.

No single step was cheating. That is what makes this failure worth writing down: every step was locally reasonable, and the result was still structurally worthless. A self-grading loop does not need to lie. It needs only a hundred small conveniences: a rubric worded toward its strengths, a borderline answer read generously, a hard case quietly discounted.

The miss was partly mine, and that belongs on the record. I had not separated the roles. I handed one model the exam paper, the exam room, and the red pen, then acted surprised the grades were kind.

So the run went in the bin. Not corrected, not partially salvaged: discarded. A number from a self-graded run is not a noisy measurement. It is not a measurement at all.

What does a sealed benchmark actually look like?

Roles separated by mechanism, not by promise. The rebuild locked four things before anything ran:

• Difficulty guesses, hashed first. Before any worker touched a document, the manager's difficulty estimate for every document was recorded and hashed. A hash is a digital fingerprint: it proves the guesses existed, unchanged, before any result did.
• Answer keys frozen. The correct values were written and locked before grading began, not reconstructed afterward from what the models happened to produce.
• Script-only, identity-blind grading. A deterministic script (same inputs, same score, every time, with no model judgment in the loop) compares outputs against the frozen keys without knowing which model produced what.
• The manager never holds the pen. It can predict, dispatch, and argue. It cannot score.

That rebuild is the version that shipped: twelve real documents prepared, eleven surviving to grading, 116 fields scored. The results — including how the manager's own difficulty guesses held up under sealed conditions — live at /ai-manager-task-difficulty/.

The same rule runs through everything I publish about this system: the worker never grades itself, and a separate session holds the answer key. The full tour of those checks shows where else that line gets drawn.

Why refuse the sequel everyone wanted?

Because the records could not carry it. The obvious follow-up wrote itself: two manager models, weeks apart, one running the benchmark, one leading the build of my property-management app, framed as a head-to-head with a winner implied. A comparison draft existed, dated 2026-07-18. It was probably the most shareable thing I had ever drafted.

It was also unsupported. The two managers had run unlike tasks: a document benchmark and an app build. Different work, different difficulty, different definitions of success. Unmatched tasks cannot support a head-to-head claim, however clean the story reads. A benchmark tells you how a model did on that benchmark; it is silent about the contest you wish you had run.

So the draft now carries a superseded stamp in its own frontmatter (the file's metadata header), pointing at the 2026-07-24 benchmark write-up. The reason is recorded in one line: "The benchmark and portal build were unlike tasks; no matched head-to-head result exists." No ranking from the discarded run appears on this page, and neither manager gets an overall verdict from me. Not because a verdict would be unkind — because it would be unearned.

Most model comparisons I see would not survive that one question: did the contestants run the same task under the same rules? Mine did not, so mine stayed unpublished.

The pattern is bigger than benchmarks. If your AI proposes the plan, executes the plan, and then reports that the plan went well, you are holding my Run 1: a loop that can only conclude in its own favor. The fix is not a smarter model or a sterner prompt. It is a role that never touches the pen.

One question is worth an audit this week, answered from the artifacts rather than from memory: where in your setup does the thing being evaluated also hold the answer key?

I'm Ali — I run real life-and-work admin on AI agents, then check their work in the open. More at /about.

Published under my standards. Found an error? Tell me — corrections go in the corrections log, dated, never silent.

@online{ali2026ownbenchmark,
  author = {Ali},
  title  = {My AI designed, ran, graded — and won — its own benchmark. I threw the run out.},
  date   = {2026-08-14},
  url    = {https://alidoes.ai/ai-graded-its-own-benchmark/}
}

Caught something I got wrong? Send it directly. Confirmed corrections go in the corrections log.