Recap Extraction Quality: Measure Meaning, Completeness and Version Accuracy

Key takeaways

• Evaluate complete commercial expressions, not isolated headline values.
• Include source-version selection and omission detection in the test plan.
• Keep ambiguous terms unresolved when the evidence does not support certainty.

Define the extraction task precisely

Decide whether the system is extracting a selected recap or also selecting the authoritative recap from a negotiation chain. Those are different tasks and need separate evaluation. A perfect extraction of the wrong document is still unusable for the intended handover.

List the commercial expression groups to be captured: parties, vessel, cargo, dates, places, payment basis, subjects and document references as applicable. Define how values relate to their qualifiers. A quantity and its option language should be assessed together rather than counted as unrelated successes.

Specify what the output should do when information is absent or ambiguous. Appropriate abstention can be a correct result. If a referenced rider has not been supplied, the system should identify the dependency instead of completing the record from general knowledge.

Build reference answers with commercial reviewers

Create an evaluation set from permitted recap examples and associated attachments. Include amendments, compact broker shorthand, forwarded correspondence and recaps with unresolved subjects. Keep the full context needed to judge the intended extraction task, but avoid exposing unrelated correspondence unnecessarily.

Have qualified reviewers annotate the expected output using a shared guide. Record the selected source, relevant phrase and expected status for each material expression. Where the source itself is unclear, document the uncertainty rather than asking reviewers to guess what the parties probably intended.

Separate disagreements about extraction from disagreements about legal interpretation. The test can verify that wording is preserved without deciding its legal effect. Cases requiring contractual judgment should be labelled for escalation and evaluated on whether the system routes them appropriately.

Assess meaning at several levels

At field level, check whether names, places and values match the source. At expression level, check whether options, conditions and units remain attached. At document level, check whether the record is complete enough for its stated purpose and reflects the correct source version.

Also examine cross-field consistency. The output may correctly identify a loading port in one section while assigning that port as the redelivery location elsewhere. These errors can survive isolated field scoring because each value appears somewhere in the source.

NIST's discussion of AI trustworthiness recognises that failures have different potential consequences. Use that principle to distinguish commercial meaning changes from minor presentation defects in your own evaluation plan. NIST AI trustworthiness discussion.

Use a qualitative recap test table

The following examples are hypothetical test inputs, not actual fixtures. They illustrate expected handling and the defects a review should detect without assigning invented industry benchmarks.

Use this table to agree what constitutes an error before running the evaluation. Reviewers should know whether a normalised abbreviation is acceptable and when preserving the exact phrase is necessary. That consistency makes comparisons between versions more meaningful.

Add real failure cases to a controlled challenge set after review. Keep their origin and expected behaviour documented so a future improvement can be tested against the same question rather than a remembered anecdote.

Measure omissions and unsupported additions separately

An extraction can appear accurate if the evaluator checks only populated fields. Compare the output with the complete reference answer to identify missing conditions, subjects and attachments. A missing qualification may matter more than several correctly extracted addresses.

Track unsupported additions explicitly. These include invented payment dates, assumed clause editions and inferred subject status. A system that produces fewer blanks may be less useful if it fills them with plausible but ungrounded information.

Assess abstentions by their usefulness. A vague warning on every field creates review fatigue; a precise statement that an incorporated rider is unavailable directs attention effectively. Evaluate whether the system explains the uncertainty and points to the source passage requiring a decision.

Include the reviewer and downstream handover

Measure the work needed to validate the output, including opening source documents, resolving field associations and correcting status. The review interface affects practical quality. A source-linked expression can be quick to check, while an equally accurate value without provenance may require repeated searching.

Test the handover produced from approved data. Confirm that unresolved items remain visible and that exported instructions do not discard qualifications. Quality can be lost after extraction if a downstream template simplifies a nuanced field into a misleading label.

Keep evaluation results tied to the tested configuration and schema. A change to field grouping or normalisation can affect commercial meaning even if the underlying model is unchanged. Recheck material workflow changes against the reference set and inspect any new failure categories.

Fictional case: a strong score hides missing options

In this fictional case, a chartering team evaluates a recap extractor by counting correct names, ports and headline values. The initial result looks encouraging. A commercial reviewer then notices that quantity options and conditional date language are frequently omitted from the generated handover.

The team changes the reference guide to assess complete expressions and adds a separate omission review. It also checks the exported handover, where some qualifications were being removed after extraction. The next evaluation distinguishes parsing errors from template errors.

The revised measurement plan gives the adoption decision a better foundation. It does not claim that every recap can be interpreted automatically; it shows whether the system preserves the information humans need to make the actual commercial decisions.

Common mistakes

Do not count a value as correct when its unit, option or condition has been lost. Another mistake is testing extraction from a selected document while marketing the result as reliable selection of the final agreement from an entire negotiation chain.

Avoid excluding ambiguous examples after results are known. Label their expected escalation behaviour instead. Finally, do not stop evaluation at the database fields; check the handover and exports that colleagues will actually use.

Where Zeaclub fits

Zeaclub offers digitalised procedures across the voyage workflow.

Zeaclub provides 24/7 operating systems and worldwide coverage.

Explore Zeaclub

Learn about Zeaclub