Build a chartering AI test set that reflects a US desk's real work

Sample the work, not just the clean documents

A useful chartering AI test set should reflect the tasks a US desk intends to delegate, including the quality of information those tasks normally contain. Selecting only complete, well-formatted enquiries can produce a reassuring result that says little about ordinary work. The test set should include the decision stages, ambiguities, and document combinations the assistant will actually encounter.

Google's dataset division guidance emphasizes representative test data and the risk of duplicated examples. NIST's Measure playbook connects evaluation with the intended operating context. Those principles support the original sampling method here. It is a practical framework for evaluating a workflow, not a prescription for training a model or a claim that a small sample proves general reliability.

Begin by deciding what the assistant is supposed to do. Extracting cargo particulars, comparing quotations, and preparing follow-ups are different tasks. A single collection of emails may support several tests, but each test needs its own question and expected behavior. Otherwise the evaluator may reward a system for answering something other than the user's request.

Map the desk's work across several dimensions

Cargo type is one dimension, but it is not enough. A desk may handle similar cargo through several commercial stages with very different information needs. Early enquiries may contain incomplete dimensions. Later proposals may contain exclusions. Final handovers may depend on distinguishing confirmed arrangements from planning assumptions.

Include evidence quality as a separate dimension. Some cases should have clear source documents, while others should contain conflicting updates, missing attachments, or ambiguous units. The purpose is not to make the test artificially difficult. It is to ensure that the assistant encounters the uncertainty it will need to manage in practice.

US geographic context can also change the task. A Gulf export enquiry, a Great Lakes movement, and a Hawaii delivery question may require different distinctions about route scope and timing. The test should preserve the relevant context without pretending that a place name alone captures operational complexity. Select cases because they test a meaningful decision, not simply to fill a map.

A fictional sampling exercise

Imagine a hypothetical US desk that wants an assistant to prepare initial enquiry summaries. Its archive contains many messages from one large machinery project, several routine bulk enquiries, and a smaller number of coastal and island shipments. If the team randomly selects individual emails, the machinery project may dominate because it generated a long thread.

The resulting test could look large while representing only a few distinct commercial situations. A better approach groups related messages into enquiry-level cases, then selects cases across the intended task mix. The machinery project remains useful, but its twenty follow-up messages are not counted as twenty independent examples of performance.

The team also discovers that most selected cases ended successfully. It adds withdrawn enquiries, unresolved enquiries, and cases where the correct response was to request clarification. This prevents the test from teaching the evaluator to expect every answer to end with a viable option. A useful assistant should recognize when the available information cannot support that conclusion.

Finally, the team creates a separate group of difficult exceptions, such as a changed cargo scope and a vessel identity conflict. It reports those results separately from routine cases. The distinction helps users understand both ordinary usefulness and the limits that matter before expanding the assistant's role.

Preserve the facts needed to judge the answer

Redaction should protect confidential information without destroying the logic of the case. If two vessel names are similar, replace them with two similarly distinguishable fictional labels while preserving their different identifiers. If a quantity changes, preserve the change and its order. If the original difficulty involved two delivery endpoints, keep that structural difference.

Record what the assistant is allowed to see. A reviewer may know the eventual outcome, but that knowledge should not silently become the standard for an earlier-stage answer. The test should judge whether the assistant used the evidence available at the assigned moment. Hindsight can make an unsupported guess look better than a careful clarification request.

Also document the expected answer's basis. A reference response should identify the essential facts and acceptable uncertainty. It need not dictate exact wording. If two reviewers cannot agree on the correct interpretation, resolve the ambiguity or classify the case as requiring judgment rather than forcing an artificial single answer.

Keep development examples separate from fresh checks

Examples used repeatedly to improve prompts can become familiar to the system design process. They remain valuable for checking that a known problem stays fixed, but they are weaker evidence of performance on new work. Maintain some fresh cases that were not used to tune the workflow and explain how the sets differ.

Do not claim independence when several cases are slight variations of one enquiry. Variations can be useful diagnostic tests, especially for message order or missing fields. They should be labeled as such. A report should distinguish breadth across commercial situations from repeated testing of one mechanism.

Refresh the collection when the intended work changes. Adding a new cargo category, a new document source, or an external action can introduce failure modes the original set never examined. The test set should evolve with the role, while retaining important older cases for continuity. Expanding access without revisiting the evidence creates an unsupported assumption about generalization.

Make the sample understandable to buyers

A US team evaluating Zeaclub's proposed agentic direction can ask what kinds of enquiries a trial would cover and what it would exclude. That is a more useful question than asking only for an overall accuracy percentage. The founder's stated YC inspiration supports learning from a defined customer workflow, with a test set that reflects the problem rather than a polished demonstration.

A practical starting point is to select redacted enquiry-level cases across routine work, incomplete information, and consequential exceptions. Agree on the task and expected behavior for each before running the assistant. Then report results by category and describe the limitations openly. This makes the evidence useful for a deployment decision and gives the product team a clear path for improvement.

Sources

Related articles