Key takeaways
- Evaluate identity, freshness, qualification and duplicate handling separately.
- Use an independently reviewed reference set with explicit ambiguity labels.
- Include review effort and downstream corrections in the adoption decision.
Define what a usable position means
Write an operational definition of a usable entry before choosing metrics. It should identify the vessel, reported opening expression, information source, source time and relevant conditions. Decide which fields are essential for the desk's actual use and which are helpful background.
Distinguish source fidelity from commercial verification. An automated entry can accurately reproduce an email that contains unconfirmed information. The quality report should show whether the system copied the source correctly and whether the desk's verification status is represented accurately, rather than treating those as the same outcome.
Likewise, define freshness in terms of the source and desk process. A record processed recently is not necessarily based on recent information. Keep processing time out of any metric intended to describe how current the underlying position is.
Build a reference set that includes updates
Select complete message sequences, not only isolated circulars. Position maintenance involves revisions, withdrawals, repeated lists and conflicting sources. A test that evaluates one email at a time may miss whether the system leaves several incompatible current entries for the same vessel.
Have knowledgeable reviewers label the expected record state after each sequence. Record uncertainty when the messages do not resolve the position. The correct answer may be “requires broker clarification”, not a confidently selected opening date.
Keep an annotation guide describing identity matching, date ranges and superseded information. Review disagreements between annotators to improve the guide. Do not hide difficult cases by excluding them after the tool performs poorly; classify them and decide whether they belong within the intended deployment scope.
Use a measurement plan tied to decisions
| Quality dimension | What to inspect | Business decision informed |
|---|---|---|
| Vessel identity | Fields attached to the correct ship | Whether merging can be suggested safely |
| Source fidelity | Meaning retained from the message | Which extraction fields need review |
| Freshness | Current and superseded information distinguished | Whether entries can support a live shortlist |
| Qualification | Conditional wording remains visible | Whether users may treat status as verified |
| Duplication | Repeated messages do not create false supply | Whether list maintenance work is reduced |
| Review burden | Corrections and source searches required | Whether the workflow is practical for the desk |
Keep the dimensions separate in the report. A strong score on easy vessel-name extraction should not conceal poor withdrawal handling. Managers need to see the types of failure that affect commercial use, not only a blended measure that rewards abundant low-impact fields.
NIST's AI RMF discusses choosing metrics and thresholds with human judgment. Apply that idea by having the desk define which errors are unacceptable for its proposed use. NIST AI RMF publication.
Use qualitative test inputs before adding numerical targets
The following inputs are hypothetical evaluation cases, not market information. They illustrate how a desk can specify expected behaviour without inventing industry-wide accuracy targets or promising a particular commercial return.
| Hypothetical input | Expected handling | Failure to record |
|---|---|---|
| A current note above an old circular | Distinguish new text from quoted history | Old opening shown as a new report |
| A vessel with an uncertain opening range | Preserve the range and uncertainty | Specific port or date invented |
| A repeated circular with a changed condition | Show the material change for review | Earlier qualification silently retained |
| Similar ship names from different sources | Keep identity unresolved without evidence | Unjustified merge |
| A reported withdrawal | Route an availability update to the owner | Vessel remains apparently open without a flag |
Use such cases to make expected behaviour discussable with operators and developers. Once the definitions are stable, the team can set its own measured acceptance criteria from a real evaluation set. The criteria should reflect the deployment scope and be fixed before final results are reviewed.
Track errors by consequence and recoverability
Classify errors according to what a user could reasonably do with the entry. A formatting defect that remains obvious has a different consequence from an incorrect opening presented as verified. Also record whether the review interface made the error easy to detect.
Separate omissions from false additions. A missed vessel can reduce coverage, while an invented position can create misleading supply. Both matter, but their operational consequences differ. A metric that counts only extracted entries may fail to notice what the system omitted entirely.
Track corrections after publication as well as before it. A tool may appear accurate because reviewers catch many errors, but those corrections represent work. Conversely, an apparently low correction rate may reflect insufficient review rather than better performance. Use sampled independent checks to interpret the operational data.
Compare quality across the real workload
Break results down by message format, sender pattern, language where relevant and update type. This helps determine whether a successful pilot can be expanded. A parser that handles a familiar table well may struggle with compact prose or forwarded attachment chains.
Keep a stable evaluation set for comparing versions and add new cases in a controlled way. Record changes to the model, parsing instructions, matching rules and review interface. Improvement in one dimension can coincide with regression in another, so compare the full measurement profile.
Do not turn the dashboard into a staff league table without examining workload differences. Reviewers handling ambiguous sources will naturally make more corrections. The primary purpose is to improve the workflow and identify where human attention adds value, not reward easy queues.
Fictional case: a clean list with stale withdrawals
In this fictional case, a desk's automated list looks tidy and extracts most opening fields correctly. Independent sequence testing shows that withdrawal messages are routed to a general inbox and never update the list. The apparent supply therefore includes vessels whose status should be questioned.
The team adds withdrawal handling as a distinct quality dimension and creates a review route for availability changes. It then tests message sequences ending in withdrawals, corrections and conflicting reports. The desk also adds a visible unresolved-status flag while the broker confirms the position.
The resulting improvement is not captured by counting more populated fields. It comes from measuring the lifecycle of a position and the decision a user might make from it.
Common mistakes
Do not use processing time as a proxy for information freshness. Another mistake is reporting only the accuracy of entries the tool chose to extract, which can hide omissions. Evaluate against an independently defined set of expected records and statuses.
Avoid one aggregate score that combines consequential identity errors with minor formatting differences. Finally, do not equate fewer reviewer corrections with better quality without an independent check. Reduced attention can produce the same apparent result.
Where Zeaclub fits
Zeaclub provides a multi-party shared workspace for owners, charterers, brokers and port agents.
Zeaclub offers digitalised procedures across the voyage workflow.
Explore Zeaclub
Educational information only; not legal, financial or trading advice. Contract wording and applicable rules control. Verify current requirements and obtain qualified advice before acting.
Frequently asked questions
What is the most important quality metric?
There is no single measure that covers the whole task. Identity, freshness, qualification and update handling should be visible separately. The desk should identify which failures would make the list unsuitable for its intended use and set review requirements accordingly.
Should ambiguous messages count as errors?
Not automatically. If the source is genuinely ambiguous, correct handling may be to preserve uncertainty and request review. Count it as a failure when the system invents certainty, loses the ambiguity or routes the case in a way that prevents timely clarification.
How should review effort be measured?
Observe the work needed to verify, correct and publish entries, including source searches and reopened records. Keep that effort connected to the type of message and error. This shows whether automation reduces useful work or merely moves it to another person.
When should the quality plan change?
Revisit it when the intended use, message sources or action permissions change. Preserve the earlier definitions and results so comparisons remain meaningful. New metrics should explain a new operational question, not be introduced simply to make a disappointing result look better.