Proof
Pre-run task-difficulty scorecard
This is the row-level record behind Before an AI agent delegates work, can it tell which tasks are hard? It preserves the historical numbers, rules, and limits of that run.
Question
Could one AI manager estimate how difficult selected-page document extraction would be for the four contestant configurations available to it?
Design
- A deliberately curated set contained 12 extraction cases and 127 required fields. It was not a random or representative sample.
- Contestants received the same selected rendered pages. Ten cases used excerpts; two one-page sources were complete. Whole-document navigation was out of scope.
- Final difficulty estimates were timestamped and hash-recorded before the first contestant output. The recorded SHA-256 still matches. This was an internal chronology record, not a completed external preregistration; the approval and post-run attestation fields remained blank.
- Four contestant configurations attempted every case.
- The historical deterministic scorer (the same output always receives the same score) applied the same scoring function to every output. It normalized formatting and allowed a guarded containment fallback. Model labels remained present during aggregation, so the scoring function was identity-neutral rather than identity-blind.
- One large-format case was render-limited. It was excluded from extraction-skill interpretation, leaving 11 gradeable cases and 116 fields.
- The measured tier below is based on mean contestant accuracy. Thresholds were written down before the run: easy at 90% or higher, medium at 70-89%, hard at 50-69%, and complicated below 50%.
- The measured tier is relative to this contestant set and historical scoring rule, not an objective property of each source.
- Every contestant ran every case. This measured pre-run difficulty estimates, not selective routing or the manager's acceptance decision.
Result
| Case | Pre-run estimate | Measured tier | Mean accuracy | Direction |
|---|---|---|---|---|
| D01 | easy | medium | 89% | harder |
| D02 | easy | hard | 58% | harder |
| D03 | medium | hard | 64% | harder |
| D05 | medium | hard | 56% | harder |
| D06 | medium | hard | 67% | harder |
| D07 | hard | complicated | 48% | harder |
| D08 | hard | complicated | 46% | harder |
| D09 | hard | hard | 50% | same |
| N1 | medium | hard | 65% | harder |
| N2 | medium | complicated | 40% | harder |
| N3 | complicated | medium | 88% | easier |
Gradeable-case summary: 9 of 11 landed harder than estimated, 1 matched, and 1 landed easier.
All-case sensitivity: Including the render-limited case produces 9 harder, 2 same, and 1 easier across all 12. That case was estimated complicated and scored complicated, so excluding it changes the denominator and same-count, not the nine harder cases.
Read the paired article: Before an AI agent delegates work, can it tell which tasks are hard?
Running example
D02 looked simple in the pre-run description: a short forwarded email. It was estimated easy. Across the four contestants it averaged 57.5%, rounded to 58%, which placed it in the measured hard tier. The lowest-scoring contestant returned 1 of 10 required fields correctly under the historical scorer.
The description suggested an ordinary email, while the benchmark demanded field-level precision. One plausible explanation is that the estimate tracked how easy the message looked to read, not the specific worker and pass condition. The run did not isolate the cause.
What this supports
- This AI manager's pre-run estimates were poorly calibrated for this curated contestant-and-rubric configuration.
- The gradeable-case errors were strongly one-directional in this run.
- A difficulty label should be calibrated against the intended worker and a defined acceptance rule before it controls exact-work delegation.
The third item is a risk-control recommendation suggested by the case. The benchmark did not test whether that intervention improves live routing.
What this does not support
- A universal optimism bias in AI managers or a model family.
- A matched comparison between AI-management systems.
- A representative estimate for document extraction generally.
- A whole-document-navigation result.
- An objective definition of source difficulty.
- A completed formal preregistration.
- A claim that live routing or acceptance failed.
- A clean extraction conclusion from the render-limited case.
Verification record
The historical 9/11 summary was recomputed from the stored aggregate results and matches the row-level table above. This scorecard retains the original historical grading rule and numbers rather than silently replacing them with a new analysis.
Release rule: this scorecard and the paired article are reviewed as one release bundle against the timestamped estimate file, the pre-run thresholds, the stored aggregate result, and the historical scorer implementation. The public page reproduces only sanitized case IDs, aggregate scores, method details, and limits. It does not reproduce source documents, private filenames, prompts, answers, or identities.
Published under my standards. Found an error? Tell me — corrections go in the corrections log, dated, never silent.
Caught something I got wrong? Send it directly. Confirmed corrections go in the corrections log.