Review individual claims, not the overall impression
Start with answers that a user challenged. Split each answer into checkable claims: a dimension, material requirement, revision date, approval status or statement about a document. Record the source needed to support each claim and whether that source was actually available to the model.
FActScore assesses factual precision by checking atomic claims against a reliable source. Our practical adaptation is a claim ledger for your workflow, with supported, contradicted and not-established states. Keep those states separate from whether the prose is clear. A useful answer can state that the available files do not establish a fact. A citation should point to the relevant evidence; a valid-looking document identifier is not enough to establish support.
Determine whether the answer was possible
Replay a failing task with the exact retrieved passages, file versions and tool results the model received. Compare that context with what the reviewer used. If the reviewer had a later drawing or a separate acceptance record, the model may have been asked to answer a question its input could not settle.
Then supply the relevant evidence in a controlled replay. If the answer improves, investigate retrieval, extraction, permissions or version selection before changing training. If the model still contradicts the supplied material, inspect instruction behavior and target quality. Keep sensitive or unavailable evidence out of evaluation prompts that simulate ordinary user access. An evaluation should represent what the deployed assistant can actually see, while recording what additional evidence a reviewer used to assess it.
Look for training targets that reward unsupported certainty
Sample the records closest to the failure. Does a target contain an approval statement without an associated approval source? Does it silently fill blank fields? Were polished synthetic answers accepted without checking whether the provided context supports their claims? Record these as specific preparation defects rather than labeling the entire dataset bad.
Review whether source facts, inferred conclusions and illustrative placeholders were distinguished during annotation. A domain expert should be able to explain the basis for the target using the permitted evidence. Do not add private reasoning narratives as a substitute for that evidence. Concise source references, explicit assumptions and a documented review status are more inspectable. Keep native project files distinct from prepared response targets so corrections can be traced back to their origin.
Test an answerable question beside an unanswerable one
Illustrative example: a user asks whether a revised ventilation design was approved. The available package contains the revised model and a comment asking for another check, but no acceptance record. A confident answer that says “approved” is unsupported. The existence of the revised model establishes an output, not approval.
Create a matched task where an applicable acceptance record is supplied, and another where it is absent. Ask reviewers to specify the acceptable response for each. The second response might identify the missing evidence and ask for the review record. Do not require a blanket refusal when the assistant can answer part of the question. Check whether it distinguishes what the files show, what remains unknown and what additional record would resolve the uncertainty.
Train missing-evidence behavior as an explicit experiment
Prepare reviewed examples that include complete evidence, incomplete evidence and conflicting versions. Define when the assistant should answer, qualify a statement, request a source or hand the decision to a reviewer. Those policies should fit the product; generic uncertainty phrases can be unhelpful if they replace an answer supported by the files.
Compare the candidate with the baseline on both supported answers and necessary abstentions. Record unsupported claims, supported claims omitted and unnecessary refusals. Synthetic cases can help create controlled missing-source variations, but inspect them for accidental clues and unrealistic language. Test the resulting behavior on separate real workflows. Changing temperature or adding a cautionary prompt is also an experiment, not proof that unsupported factual behavior has disappeared. Use the same evidence access when comparing alternatives.
Make a release decision using an evidence checklist
- Does each material claim point to available, applicable evidence?
- Can the assistant identify when a required record is absent?
- Are revised files and validation records linked to the same version?
- Does a cautious response remain useful when part of the answer is supported?
Common question: will more expert text solve hallucinations? Expert material can improve the context and examples available for preparation, but professional tone alone does not establish factual support. Buy or prepare records around the missing evidence type, such as requirements, version references or review comments, where available. Define linking and expert checks separately. Release only against a stated acceptance rule and preserve an escalation path for decisions the records cannot support.