Overview
Use an original evaluation plan that separates record discovery, field accuracy, uncertainty handling and reviewer effort. The decision is whether the workflow produces reliable, reviewable candidates for the desk, not whether it can fill a table quickly.
Key takeaways
- Define what counts as a position record before measuring extraction.
- Judge material fields and relationships, not only text similarity.
- Count missed records as well as incorrect output.
- Test whether uncertainty is preserved or hidden by confident normalization.
- Measure correction effort and commercial usability alongside arithmetic scores.
Define the reference task first
Specify the unit the parser should produce. A message may contain several vessel positions, a correction to an earlier position or commentary that should not create a new record. The reference reviewers need a common rule for those cases before the output is scored.
Define the fields that matter to the use case: vessel identity, opening place, opening window, relevant qualification and source relationship, for example. State how ambiguous or absent information should be represented. An unknown value should not be marked wrong merely because the system declines to invent one.
Keep the reference independent of the parser's output where practical. If reviewers simply accept the generated table as the starting truth, they may miss omissions or inherit its record boundaries. The evaluation should reveal what the system failed to find as well as what it found incorrectly.
Judge the relationships between fields
A parser can copy several phrases accurately while combining them into the wrong position. The opening location may belong to one vessel and the date window to another. Evaluate the record as a linked commercial statement, not just a collection of individually recognizable strings.
Likewise, qualifiers can change meaning. Subject to completion of present employment is different from a confirmed opening. A broad geographic description should not become a specific port without support. The evaluation should identify where normalization changes the commercial interpretation.
Allow legitimate differences in presentation when the meaning remains intact. The test should not punish harmless punctuation or formatting while overlooking a wrong relationship. Write the acceptance rule clearly enough that reviewers can explain why a difference matters.
Count misses and false candidates separately
A system that extracts only the clearest positions may produce accurate-looking output while omitting much of the useful market information. A system that creates a record from every mention may capture more candidates but burden the desk with incorrect entries. Both trade-offs should be visible.
ILLUSTRATIVE: In a fictional test set, reviewers establish 120 relevant position records. The parser outputs 110 candidates, of which 100 match relevant reference records under the stated matching rule. There are therefore 10 incorrect candidates and 20 missed reference records.
The share of output candidates matching the reference is 100 / 110 × 100 = 90.91%, rounded. The share of relevant reference records found is 100 / 120 × 100 = 83.33%, rounded. These are invented test results, not a product benchmark or a suggested acceptance threshold.
Those measures answer different questions. The first describes the output's relevance under the matching rule; the second describes coverage of the reference set. Neither establishes that every field within a matched record is commercially correct, so field-level and usability checks remain necessary.
Build a quality scorecard around consequences
The following table is an original scorecard structure. Choose the actual acceptance criteria with the desk and evaluation owners rather than importing universal thresholds.
| Quality dimension | Question to test | Consequence of failure |
|---|---|---|
| Record discovery | Were relevant positions found without spurious entries? | Missing opportunities or unnecessary review |
| Vessel association | Do the fields belong to the correct vessel? | Misleading commercial comparison |
| Opening information | Are place and window faithful to the source? | Incorrect availability assumption |
| Qualification handling | Is conditional or uncertain wording preserved? | Premature reliance on a position |
| Source traceability | Can the reviewer locate supporting context? | Slow or unsupported correction |
| Update handling | Is a correction linked to the earlier record? | Duplicate or stale positions |
| Review effort | Can material errors be found and repaired efficiently? | Hidden workload despite apparent automation |
Report the categories separately. A strong score on easily copied identifiers should not conceal poor handling of qualifications or updates that drive actual decisions.
Test difficult but representative messages
Include messages reflecting the desk's real workflow: multi-vessel lists, forwarded threads, revisions, mixed formatting, ambiguous windows and partial information. The goal is not to create arbitrary puzzles, but to observe failure modes that could matter in normal use.
Keep a stable reference set for comparison and a separate stream of newly encountered cases for learning. If every test is revised after seeing the system's output, the team may lose the ability to judge whether a change improved the workflow or merely fit the latest examples.
Document the origin and permitted use of the test material. Use the organization's confidentiality and access process, especially where live commercial messages contain information unrelated to evaluation. The quality exercise should not create unnecessary distribution of sensitive source content.
Fictional case: accurate fields, wrong vessel
In a fictional evaluation, a tonnage email lists several vessels in compact paragraphs. The parser correctly captures an opening location and a date window, but associates them with the vessel named in the previous paragraph. A text-focused check marks both phrases as present and therefore correct.
The record-level review catches the error because the commercial statement is wrong. The desk revises the evaluation rule to require correct field association and adds a category for boundary errors between vessel entries. It does not merely penalize the individual copied words.
The corrected scorecard shows why the output was less usable than its cell-level accuracy suggested. Reviewers also assess whether the source link makes the mismatch easy to detect. The case demonstrates that extraction quality depends on preserving relationships, not just recognizing strings.
Measure the work that remains for humans
Record the nature of reviewer corrections and the time or effort needed under a clearly defined test process. Distinguish quick formatting edits from investigations requiring a return to the original sender. A workflow that saves typing but increases uncertainty may not deliver the expected operational benefit.
Also note errors that reviewers fail to detect during ordinary use. A candidate that looks polished can be harder to challenge than visibly incomplete output. The evaluation should consider whether material uncertainty and source context are presented in a way that supports effective review.
Use the results to make a scoped rollout decision. The system may be suitable for candidate preparation in a defined message class while remaining unsuitable for unattended distribution. State the accepted use and the cases that still require manual handling.
Common mistakes
One mistake is measuring only how many fields are filled. Completeness without faithful meaning can produce confident but misleading positions.
Another is reporting a single average that hides missed records, wrong associations and lost qualifications. Keep the failure categories visible so the desk can judge their practical importance.
A third is evaluating only messages used to tune the system. Preserve an independent comparison process and representative new cases so the result reflects more than familiarity with the test examples.
Where Zeaclub fits
Zeaclub supports agentic, AI-assisted operations with a human in control.
Zeaclub provides a multi-party shared workspace for owners, charterers, brokers and port agents.
Explore Zeaclub
Educational information only; not legal, financial or trading advice. Contract wording and applicable rules control. Verify current requirements and obtain qualified advice before acting.
Frequently asked questions
Is a matched record automatically correct?
No. Matching establishes that the candidate corresponds to a relevant reference entry under the stated rule. Its fields, qualifications and source relationships still need assessment for the intended commercial use.
Should uncertain source information count as an error?
Define the expected handling in advance. Preserving uncertainty can be correct, while inventing a precise value may be a material failure. The reference should distinguish absent information from extraction mistakes.
What is a sensible acceptance threshold?
Set it for the actual use, error consequences and human review process. The illustrative percentages are not recommendations. A workflow preparing internal candidates may need different criteria from one distributing positions externally.
How should results guide rollout?
State which message classes and tasks the evidence supports, what review remains necessary and which failure modes require improvement. Avoid turning a limited test result into a claim that all tonnage correspondence can be handled reliably.