Measure shipbroking AI by completed useful work, not output volume
More activity can mean more work for the broker
A shipbroking assistant should be evaluated by the useful work it helps complete, not by how much text or how many candidate records it produces. A US desk can receive more drafts, summaries, and alerts while becoming slower if staff must correct and sort them. Output volume describes activity. It does not establish productivity.
Anthropic's agent evaluation discussion distinguishes an agent's account of its work from the resulting outcome. Google's People + AI guidebook recognizes that output can be confident yet irrelevant to the user's needs. Those ideas support a practical measurement question: did the assistant help the desk reach a correct, useful result with less total effort? The framework and figures below are original and illustrative.
This does not mean speed is unimportant. Faster preparation can be valuable when quality holds and the output arrives at the right stage. The mistake is to count generation speed while excluding review, correction, duplicate handling, and the work created by unnecessary messages.
Define a completed unit of work
Choose a unit that corresponds to a real desk task. Examples include a clarified cargo enquiry, a common-scope quotation comparison, a correct draft ready for review, or a reconciled readiness update. Avoid broad units such as handled voyage unless the workflow actually covers all the necessary work and the team agrees on what completion means.
The unit should include a quality condition. A comparison is complete when it uses a consistent basis and identifies unresolved components, not merely when a table exists. A draft is complete when it accurately reflects the authorized purpose and evidence, not merely when it reaches a word count. This prevents the metric from rewarding superficial completion.
Also define the boundary of the measurement. If the assistant prepares a document and a broker reviews it, include both stages when comparing effort. If the recipient later requests clarification because the draft was ambiguous, record that rework where practical. The more meaningful the business claim, the more complete the measurement needs to be.
A fictional time-saving calculation
Consider a hypothetical US desk processing ten comparable clarification requests. Without assistance, preparation and review take twelve minutes per request, for a total of 120 minutes. With a proposed assistant, initial preparation takes two minutes and human review takes six minutes per request. That totals eighty minutes before rework.
Suppose three assisted requests need an additional ten minutes each because they omit a material question. The assisted total becomes 110 minutes. The net saving is ten minutes, approximately 8.3 percent of the original 120 minutes. Counting only preparation would suggest a much larger improvement and hide the quality cost.
These numbers are fictional, not a benchmark or a Zeaclub performance claim. Their purpose is to show why the measurement boundary matters. The same assistant might perform better after improving the omitted-question cases, or it might be unsuitable for that task. The calculation directs attention to the actual source of lost value.
The desk should also inspect the nature of the rework. A minor formatting correction differs from an omission that causes a counterparty to answer the wrong question. Minutes alone do not capture every consequence. Pair time measurement with a record of material errors and the usefulness of the completed result.
Watch for incentives that inflate activity
Counting drafts can encourage the assistant to produce several versions when one would suffice. Counting alerts can reward repeated notifications about the same unresolved issue. Counting candidates can favor broad lists with weak relevance. Each metric may look productive while increasing the user's sorting burden.
A better approach asks whether the output changes the next action. If an alert repeats information already acknowledged, it may add no value. If a summary restates the entire thread without identifying the unresolved issue, it may save little time. If a draft asks questions already answered in an attachment, it creates avoidable correspondence.
This analysis should not assume that every unused output is waste. Exploratory work can reveal that an option is unsuitable, and that can be valuable. The distinction is whether the work served a defined decision. A rejected candidate supported by a clear reason may be useful; a large undifferentiated list is harder to justify.
Compare like tasks under realistic conditions
Productivity comparisons need reasonably similar task complexity. Ten simple acknowledgements should not be compared with ten complex quotation reconciliations. Group work by the problem it presents and record major differences in evidence quality, ambiguity, and required review. A single average can hide where the assistant helps and where it creates effort.
Include users with the responsibilities the workflow is intended to support. An experienced broker may review quickly because they know the enquiry, while a new user may need more context. That difference is relevant to product design. It should not be concealed by selecting only the fastest reviewer or the most favorable demonstration.
Measure enough examples to see variation and explain the limits of the result. A small trial can identify promising improvements, but it does not establish a general percentage saving across every cargo and desk. Keep the claim proportional to the evidence and continue observing performance as the workflow changes.
Keep commercial opportunity distinct from measured results
Zeaclub's founder describes hundreds of millions of dollars in commercial opportunity pipeline. That is an attributed statement about potential transaction value, not revenue, completed fixtures, or proof of software productivity. A workflow trial needs its own evidence of usefulness, independent of the size of the broader opportunity narrative.
This distinction strengthens a development discussion. A large potential market can justify investigating a problem, while a narrow task evaluation shows whether a proposed solution helps actual users. The two forms of evidence answer different questions. Neither should be substituted for the other in a customer-facing performance claim.
US maritime teams can bring Zeaclub one recurring task and define a fair measurement boundary before testing assistance. Count correct completed work, review effort, and material rework. Then identify the specific change that would improve the result. This turns a broad promise of automation into a practical learning exercise.
For the next trial, stop counting how many outputs the assistant produces and start tracking what users can confidently finish. If the workflow reduces total effort while preserving the quality of the commercial decision, the value will be easier to explain. If it does not, the measurement will show where to improve or narrow the proposed role.
Sources
- Anthropic: demystifying evaluations for AI agents, inspected October 23, 2026.
- Google PAIR: errors and graceful failure, inspected October 23, 2026.
Related articles
- 0639: Budget maritime AI adoption around the review work it creates
- 0640: Check whether maritime automation saves work or moves it to someone else
- 0768: US Broker Performance Reviews: Measure the Decisions the Work Supports


