Measure AI task routing from detection to usable assignment

Key takeaways

• Build a reviewed reference set with clear task boundaries.
• Measure missed work and unnecessary proposals separately.
• Evaluate ownership and conditions after matching a proposed task to genuine work.
• Inspect serious failure types instead of relying only on an average.
• Report results for the tested scope and configuration.

Define the task unit before counting

Decide what counts as an individual task in the evaluation. One message can contain several actions, and several messages can update the same action. Counting messages as tasks without review can distort both the reference and the result.

Ask knowledgeable reviewers to identify the intended action, case, owner and conditions. Resolve disagreements where possible and label genuinely ambiguous cases rather than forcing a convenient answer.

Define how a proposal matches a reference task. The test should distinguish a real new task from an amendment, duplicate or quoted historical request. Keep that matching method consistent across the evaluation.

The method in this article is an original editorial design. Its numerical example is fictional and describes no product's performance.

Separate detection from useful assignment

Detection asks whether the system recognised genuine work. Assignment asks whether the resulting task reaches the right role with the meaning needed to proceed. A proposal can pass the first question and fail the second.

Check the owner, source link, case reference and material conditions separately. A correctly named team may still receive an unusable task if it cannot identify the relevant voyage or if an approval dependency has disappeared.

Record unnecessary proposals as their own failure category. A router that creates work from informational updates or repeated quoted text can burden the queue even if it rarely misses a genuine request.

Do not let a large volume of easy tasks conceal a weakness in a consequential category. Review results by the types of work and exceptions relevant to the intended deployment.

Use a measurement table with clear denominators

State the matching and acceptance rules alongside the scores. A reader should not have to infer whether a detected task counted as successful despite an incorrect owner.

If a denominator is absent in a small test category, report the measure as unavailable for that category rather than manufacturing a percentage. The limitation itself is useful evidence about what the test did not cover.

Work through a fictional routing result

This example is entirely ILLUSTRATIVE. Reviewers identify 50 genuine reference tasks. A fictional router proposes 52 tasks, of which 45 match genuine reference tasks on a one-to-one basis.

There are 52 − 45 = 7 unnecessary proposals and 50 − 45 = 5 missed genuine tasks. Detection precision is 45 ÷ 52 × 100 = 86.54%, rounded. Detection recall is 45 ÷ 50 × 100 = 90%.

Of the matched genuine tasks, 40 also have the correct owner and preserve all required conditions. The other 45 − 40 = 5 matched tasks need correction before they meet the full acceptance rule. End-to-end acceptance is therefore 40 ÷ 50 × 100 = 80% of the reference work.

The unnecessary proposals remain an additional burden; they are not erased by the end-to-end score. These calculations are script-checked using invented inputs and are not a forecast or a measured product result.

The example shows why the stages should be reported separately. A detection result can look stronger than the complete workflow because it does not yet account for incorrect ownership or lost conditions.

Do not compare scores from different evaluations without checking their task definitions and source populations. A set of straightforward requests and a set containing difficult amendments can produce different results even under the same configuration.

Review the consequences behind each failure

Classify missed work by what prevented its detection. It may be embedded in a long message, expressed indirectly or confused with historical context. That helps the team design a targeted test and correction.

For unnecessary proposals, examine whether the system duplicated an existing task, extracted a quoted request or treated information as an instruction. These patterns suggest different changes and different risks of suppressing legitimate follow-up work.

For incorrect ownership, distinguish a routing error from an unclear organisational responsibility. Improving a model cannot by itself settle a business process that has no agreed owner.

Review lost conditions as meaning errors. A task can be actively misleading if it converts a conditional request into an immediate action, even when its department label matches the reference.

Test whether the receiving user understands the task

NISTIR 8312 emphasises explanations that intended users can understand. In routing evaluation, the recipient's interpretation helps reveal whether the output is usable.

Ask the recipient to explain the requested action and any unresolved dependency using the proposed task and its approved evidence. Record where additional assistance is required.

Do not substitute a polished summary for that observation. The evaluator may understand the task because they know the source case, while the receiving operator sees an ambiguous instruction.

Include evidence access in the test. A correct link label is not sufficient if the assigned role cannot open the supporting record or if the link leads to an unrelated version.

Fictional case: recall improves while the queue gets worse

A fictional team changes its routing configuration to capture more possible requests. The next evaluation finds fewer missed tasks, and the team initially treats the change as an overall improvement.

Receiving users report that the queue now contains many informational messages and repeated historical requests. They spend more time deciding which tasks are genuine, and some correctly detected tasks still lack their approval conditions.

The team reviews detection, unnecessary proposals and end-to-end acceptance together. It keeps the useful improvement in finding work while testing a more specific treatment of quoted requests and dependencies. The decision is based on the whole workflow rather than the most favourable metric.

This case illustrates an evaluation trade-off, not an empirical claim about an actual router.

Use results to choose a bounded next step

Report the source scope, reference method, configuration and acceptance rules. Explain unresolved failure categories and which tasks still need mandatory review under the proposed process.

Measure review effort consistently if efficiency is part of the decision. Count the work required to correct, reject and clarify proposals rather than timing only initial assignment.

After a relevant change, repeat the affected cases and ordinary cases that might regress. Keep a separate representative set where practical so repeated tuning does not become mistaken for independent validation.

Use the results to narrow, expand or improve the workflow. A score becomes useful when it informs a specific decision about the work the team is prepared to rely on.

Common mistakes

Avoid reporting recall without unnecessary proposals, counting a detected task as fully usable or changing matching rules between tests. Do not let department accuracy hide lost conditions.

Another mistake is measuring the router in isolation from the receiving team's review burden and understanding.

Where Zeaclub fits

Zeaclub supports agentic, AI-assisted operations with a human in control.

Zeaclub provides a multi-party shared workspace for owners, charterers, brokers and port agents.

Explore Zeaclub

Learn about Zeaclub