US AI Voyage Estimates: Test Evidence Selection With Conflicting Inputs

Test which evidence enters the estimate

A US maritime team evaluating AI-assisted voyage preparation should test evidence selection separately from arithmetic. A system can calculate perfectly from the wrong quotation, an obsolete instruction, or a rate with a different scope. The resulting total may look plausible enough to survive a superficial review.

Google researchers' 2025 conflict-RAG study distinguishes types of conflicting retrieved information and evaluates how systems respond. Google research publication. A separate IJCAI study examines retrieval-augmented fact-checking under conflicting evidence of different credibility. IJCAI CONFACT study. Neither study validates a maritime product. They support making conflict handling an explicit evaluation problem.

The original test design below asks a narrower commercial question: did the proposed workflow select the evidence that is applicable to this estimate, and did it preserve the uncertainty that cannot be resolved from the packet?

Build cases around a decision, not a document pile

Choose a particular input the estimator needs, such as a service price for a named shipment scope or a quantity from the latest authorized cargo instruction. Define what makes a source applicable before presenting the documents to the system. Date alone is rarely a sufficient rule.

For each case, record the expected source, the relevant passage, the unit, the applicable scope, and any unresolved condition. Have a knowledgeable reviewer establish that reference answer. If two reviewers disagree about the commercial interpretation, resolve the test definition or label the case genuinely ambiguous rather than forcing a convenient answer.

Use fictional or properly authorized materials. The purpose is to examine behavior, not to expose confidential counterpart communications. Preserve realistic conflicts while changing sensitive identities and values where necessary. A simplified clean packet can be useful initially, but it should not be the only evidence used to judge performance.

Keep the reference answers separate from the system's input. Otherwise the test may measure the ability to repeat a supplied solution rather than retrieve and interpret the relevant commercial evidence.

Challenge different reasons for disagreement

Include an older specific quotation and a later general rate sheet. The correct choice may remain the specific quotation if its stated validity and scope apply. A rule that always selects the most recent document would fail that case.

Include a revision that changes quantity but leaves price unchanged, and another that changes an exclusion while preserving the headline amount. The workflow should identify which field changed rather than replacing the whole estimate indiscriminately. Also include a rate expressed in a different unit so that the system must preserve the denominator before any calculation occurs.

Add an apparently authoritative document that concerns another vessel, customer, or shipment period. Its credibility does not make it applicable to the current enquiry. Evidence selection needs both source quality and scope fit.

Finally, include a packet that genuinely lacks a decisive instruction. The desired result may be a targeted clarification request, with the conflicting evidence shown. Treating that response as a failure would reward unsupported certainty and teach the evaluator to prefer the wrong behavior.

Use a fictional quotation conflict

Imagine an invented voyage packet containing an original service quote of $90,000, a shipment-specific revision of $100,000, and a later general advertisement showing prices from $80,000. The revision explicitly applies to the requested cargo and includes a particular service that the original excluded. The advertisement does not confirm applicability to the shipment.

Under those fictional facts, the expected input is the $100,000 revision with its stated inclusion. Choosing $80,000 because it is later or cheaper would not be supported. Choosing $90,000 and then adding an estimated service cost would also ignore the supplied revision unless the task specifically asked for a historical comparison.

Now remove the revision's statement identifying the cargo. The correct behavior may change to asking whether it applies. The test should reflect that difference. A good evaluation can distinguish resolving a clear conflict from noticing that the necessary link is absent.

Do not score only the final amount. Require the source identifier and the passage supporting the selection, along with the relevant scope. A coincidentally correct number without the correct basis is weak evidence that the workflow will behave reliably on the next case.

Score selection and scope together

Suppose a fictional evaluation contains sixteen cases. The system selects the correct source in fourteen and preserves the correct scope in twelve. It satisfies both conditions in ten. Source selection is therefore 87.5 percent, scope preservation 75 percent, and joint success 62.5 percent.

The joint result matters because the estimator needs both. A dashboard highlighting only the 87.5 percent source-selection figure would understate the remaining commercial review burden. Show the cases that failed and the reason, not just the headline percentage.

Separate clear-answer cases from ambiguity cases. In the latter, measure whether the system identifies the missing decision and asks an appropriate question. An abstention is useful only if it explains what is unresolved; a generic refusal does not necessarily help the user progress.

Keep the evaluation small enough to inspect initially, then broaden it with representative examples. Repeated cases with only changed names can inflate the test count without testing new reasoning. Include different conflict types, document layouts, and commercial scopes relevant to the intended US desk.

Connect the result to a bounded product decision

A weak retrieval result suggests improving the evidence-selection workflow before adding more elaborate calculations. A strong result on one carefully controlled packet does not establish universal accuracy. Define what the proposed deployment would encounter and whether the test covers those situations adequately.

Zeaclub's founder-described YC inspiration can be expressed through this kind of focused learning: start with a real user problem, test the difficult cases, and improve the proposed workflow from observable failures. That is inspiration, not YC affiliation or evidence of a shipped retrieval capability.

Bring Zeaclub a small authorized conflict set and the commercial rules that determine applicable evidence. Ask the team to show the selected passage, preserved scope, and unresolved question for each case. The useful outcome is confidence in the basis entering an estimate, with clear limits on what the test has established. Arithmetic quality can then be evaluated on inputs whose meaning has survived retrieval.

Sources

Related articles