Check the input the model actually received
Begin with the serialized request sent to the model, not the document shown in the product interface. Verify whether extraction preserved the relevant clause, table heading, footnote or drawing reference. Check any retrieval filtering, context truncation and intermediate summarization. The interface can show a complete file while the model receives a transformed subset.
Link the failing output to an exact source location and the corresponding position in the model input. If the evidence is missing or detached from its qualifier, fix that path first. Fine-tuning the model to use a requirement it never received tests the wrong hypothesis. Keep this input audit reproducible for future cases.
Separate finding the clause from applying it
Ask the model to identify the evidence needed for the decision before generating the final deliverable. Review whether it found the correct requirement and understood its scope. A correct quotation followed by a violating output suggests an application problem. An absent or unrelated passage points toward evidence selection or extraction.
Do not use the model's own explanation as the sole judge. Compare its cited evidence with a reviewed source reference and assess the resulting action independently. Record omitted requirements, misread exceptions and incorrect combinations as different errors. That distinction directs the next experiment: context presentation may help evidence selection, while task examples may be needed to test how conditions are applied together.
Test position on the model you deploy
Lost in the Middle found position-sensitive context use in its experiments. Treat that finding as a testable hypothesis, not a statement about every current model.
Create matched versions of a reviewed task in which the relevant clause appears earlier, centrally or later. Preserve its wording, meaning and other task conditions. Keep document length comparable, and check that reordering has not altered cross-references or precedence. Run these variants with your deployed prompt and checkpoint. A position effect in this controlled comparison provides a narrower diagnosis than a general complaint that the model cannot handle long documents.
Vary irrelevant context without changing the rule
Compare the complete document with a reviewed evidence-focused version, then add plausible but irrelevant sections. Preserve the requirement and the context needed to interpret it. This tests whether surrounding material affects the decision. It also checks whether your supposedly helpful short version has accidentally supplied an answer that the original task did not make explicit.
Illustrative example: a brief describes general room conditions, with a separate exception for equipment rooms. The task concerns an equipment room. A correct answer must apply the exception rather than copy the general rule. Vary nearby unrelated room descriptions while keeping the exception intact. Score both the selected evidence and the resulting requirement application.
Make requirement coverage inspectable
For complex outputs, test a requirement ledger: an identifier, source reference, applicable condition and output element for each relevant clause. Ask the workflow to report unresolved or conflicting conditions explicitly. Review the ledger against the source, then check whether the final artifact satisfies its entries.
This is a proposed workflow experiment, not proof that adding a ledger solves the task. The ledger may omit the same difficult clause or record an incorrect interpretation. Evaluate that stage separately. Keep document versions visible so an old requirement is not treated as current. Where precedence between revisions is unclear, the correct workflow may require clarification rather than a confident merged interpretation.
Compare context changes before training changes
Test clearer source segmentation, explicit references, evidence extraction or a focused retrieval step against the existing full-context approach. Use the same reviewed tasks and assess requirement coverage, final correctness and review effort. A shorter context is useful only if it retains the conditions required for the decision.
If the model still misapplies requirements when correct evidence is provided, scope a training pilot around that behavior. Include conditional rules, competing clauses and incomplete-evidence cases with reviewed targets. Keep independent projects for evaluation. Avoid treating a collection of long documents as a ready-made dataset for requirement following; extraction and task construction determine which behavior the examples actually teach.
Use source coverage as a product acceptance criterion
Define what the workflow must establish before an output is accepted. For a consequential requirement, that might include a source reference, a version check and a reviewed link to the output element. Choose the checks according to the task rather than adding a citation requirement that the model can satisfy with irrelevant text.
Report the failure stage and the limits of the tested fix. A position-sensitive result supports a context-use diagnosis in the evaluated setup; it does not establish a universal context length limit. Additional professional documents can broaden your tests, but improvement depends on preparation and the intervention. Buy the sample that helps distinguish the remaining hypotheses and measure the behavior your users need.