← All writing

Build log

Before an AI agent delegates work, can it tell which tasks are hard?

Aug 14, 2026

I let an AI manager label document-extraction cases before four contestant configurations ran them. Nine of 11 gradeable cases came back harder than the label — and the worst miss was the file that looked easiest.

Mostly, no — mine couldn't. Before four contestant configurations ran 11 gradeable document-extraction cases, an AI manager labeled each one's difficulty. Nine landed in a harder tier than it estimated.

I was trying to answer a practical question: which document tasks could I give to cheaper models without quietly lowering the quality of the result? I expected the benchmark to rank the four contestant configurations (four model setups attempting the same cases). It broke one step earlier, at the manager's difficulty estimate. Task difficulty is not a label attached to a file. It depends on the worker, the output requirement, and how failure is measured.

The case that made the problem obvious was a short forwarded email, case D02 in the scorecard. The manager marked it easy. The four contestants averaged 58% accuracy on it, and the lowest-scoring contestant returned only 1 of 10 required fields correctly. The file looked easy. The acceptance rule was not.

That is the useful result. An untested difficulty estimate is not enough evidence to delegate exact work.

What does "easy" actually mean?

Three different things, depending on who is looking. The manager saw thin pre-run information: the kind of file, a short description, and limited modality information (what form the content would arrive in). The contestants received selected rendered pages and a field-extraction instruction. The grader asked whether the required values came back under a fixed matching rule.

Those are three views of the same job:

1. The input: how simple the file or request appears.
2. The worker: which model and configuration must do the work.
3. The pass condition: what the answer must contain and what counts as wrong.

The short email looked ordinary from the first view. It was hard for this contestant set under the third. Calling it easy without naming the worker and the pass condition answered the wrong question.

Routing research is circling the same problem. SWE-Router argues that a task description alone can hide whether the underlying work is trivial or extensive, and improves routing by observing a partial trajectory first. LLMRouterBench found that several routing methods did not reliably beat a simple baseline under unified evaluation. Those studies do not validate my small benchmark. They do reinforce testing a router against observed worker performance instead of trusting surface similarity.

What did the benchmark actually measure?

Only the gap between a pre-run guess and a graded result. I deliberately curated 12 extraction cases from my own workspace, with 127 required fields. This was not a random or representative sample. Every contestant received the same selected page images; the test did not measure whole-document navigation.

Before any contestant produced an answer, I saved the manager's estimates with a digital fingerprint (a SHA-256 hash that still matches the file). Every answer then went through the same fixed scoring rules. This was an internal before-and-after record, not a completed preregistration (publicly committing to the plan before running it); the approval and attestation fields stayed blank. The scorer also kept the model labels, so the comparison was not blind.

One large-format case was unreadable at the chosen render resolution. I excluded it from extraction-skill interpretation, leaving 11 gradeable cases and 116 fields.

The tier thresholds were written down before the run: easy at 90% or higher, medium at 70-89%, hard at 50-69%, complicated below 50%. The measured tier came from mean accuracy across all four contestants. It describes performance in this setup, not an objective difficulty level for the source. Within that frame the verdict is still blunt: this manager's pre-run estimates were poorly calibrated, and the errors ran almost entirely in the optimistic direction.

How would I run the routing decision again?

The benchmark did not test live routing, so these are recommendations suggested by the failure, not interventions the run proved.

Define passing before estimating difficulty. For the short email, "read this" sounds easy. "Return these 10 fields under this matching rule, with no unsupported value" is a different job. The pass condition has to exist before the difficulty label means anything.

Estimate for the worker that will actually run. A task can be routine for a strong model and unsafe for a weaker one. If a filename or short request hides the real complexity, inspect a representative page or let the cheaper worker attempt a small pilot before committing the whole job.

Record the estimate, then grade the result independently. Keep the prediction made before the answer exists. Compare it with an output check that does not rely on the manager's original confidence. Over time that produces a calibration record (a running tally of how often the predictions matched reality) instead of a collection of routing anecdotes.

Where is this discipline worth the cost? A rough summary with low failure cost does not need a laboratory; a missing sentence can be fixed when someone notices it. Exact financial, contractual, compliance, or operational fields are different. A plausible wrong value can flow into another system without announcing itself. The stricter the output requirement and the more expensive a quiet error, the less useful an unsupported "easy" label becomes.

What does the proof support — and what doesn't it?

The proof is narrow: a pre-run hash, thresholds written before the run, the stored aggregate result, and a sanitized row-level table reproduce the 9-of-11 comparison. Every estimate-to-result pair and the all-12 sensitivity count are on the proof scorecard. None of it makes the sample representative.

This was one deliberately curated batch, mostly selected-page excerpts, one AI manager, four contestant configurations, and one historical grading rule. The result does not establish a universal optimism bias, compare AI-management systems, or prove that a particular model family is a bad manager. Every contestant ran every case: no router selected a worker, and the manager did not accept or reject a returned answer. The benchmark measured only the gap between a pre-run difficulty estimate and a contestant-relative result.

So, the rule I use now. Before I let an AI agent delegate exact work, I want three things written down: what passing means, which worker is expected to pass, and what independent check will tell me whether it did. The manager can still plan the handoff. It just does not get to turn its own guess into proof. It is the stance behind everything else here, starting with how I know the AI is right: a confident label is a claim, and claims get checked.

Which task would your orchestrator (the model that plans and hands out the work) call easy because it reads easy, under a pass condition nobody has written down?

I'm Ali — I run real life-and-work admin on AI agents, then check their work in the open. More at /about.

Published under my standards. Found an error? Tell me — corrections go in the corrections log, dated, never silent.

@online{ali2026taskdifficulty,
  author = {Ali},
  title  = {Before an AI agent delegates work, can it tell which tasks are hard?},
  date   = {2026-08-14},
  url    = {https://alidoes.ai/ai-manager-task-difficulty/}
}

Caught something I got wrong? Send it directly. Confirmed corrections go in the corrections log.