Define the decision behind the field
Start with the action your application takes after receiving a structured response. Does a quantity trigger an order? Does an approval field move a design into the next stage? Does a priority change what an engineer reviews first? Those actions establish the meaning of correctness.
A field called status can be syntactically perfect while representing the wrong stage of work. Write a plain language contract for each consequential field: what evidence supports it, which conditions make it true, and what should happen when evidence is missing. The response format is one part of that contract. The business decision is another.
See a passing response that should fail
Consider an illustrative building-controls task. A brief asks that leak sensors report to the building monitoring system. The assistant returns an object with room_id, sensor_type and integrated fields. Every field has the correct type, and integrated is true.
The supplied drawing, however, only shows the sensor location. It provides no integration reference. The response passes its schema but asserts a relationship that the evidence does not establish. A better target records the proposed sensor, identifies the missing reference and leaves integration unconfirmed. The training example should reward that distinction rather than teach the model that completing every field is always success.
Give the schema a clear job
The JSON Schema object reference describes constraints on properties, required fields and additional properties. Our practical inference: those checks alone cannot establish source fidelity.
Use your schema to catch avoidable integration problems: missing fields, unexpected keys, incompatible types and invalid enumerations. Version it with the consuming application. Explicitly decide how unknown, absent and not applicable values differ. Otherwise, an annotator may choose an arbitrary value simply because the format requires one. A carefully defined unknown state can carry more useful information than a false appearance of certainty.
Add semantic checks that can explain failure
For each critical field, design a separate check grounded in the task. Confirm that the referenced object exists, the cited file version is applicable, units match, and the claimed relationship appears in the available evidence. Where deterministic checking is possible, use it.
Other judgments need a qualified reviewer. Give that reviewer a focused question and the relevant source, rather than a generic request to approve the response. Record whether the failure comes from extraction, interpretation or missing evidence. That classification helps you choose between changing a parser, improving examples and asking the user for clarification. The remedies have different costs.
Keep draft outputs separate from reviewed targets
An automatically extracted object is a preparation artifact. It becomes a training target only after the acceptance process you agreed for that task. Store both stages without allowing their labels to blur. A file produced during a project is also not automatically a validated answer to a later AI prompt.
Useful records pair the human instruction with its source context, proposed response and review outcome where available. If no review outcome exists, label that gap. Commissioning expert checks can add value, but the resulting annotations should be identified as new preparation. Do not present them as historical decisions made by the original project team.
Measure more than the parse rate
Track three outcomes separately: responses accepted by the schema, claims supported by the available evidence, and decisions acceptable for the intended workflow. A model can improve the first while making no progress on the others.
- Include cases where a required reference is absent.
- Include conflicting values and outdated file versions.
- Check the model’s handling of units and object identity.
- Count unnecessary assertions as well as missing answers.
Use this as an original test design checklist, not a ready-made benchmark. The actual expected outcomes depend on your product, sources and review process. Keep the test projects separate from the training material.
Can a stronger schema replace review?
A stronger schema can encode more rules, and some business conditions can be checked automatically. It still needs the correct facts to operate on. If a model confidently invents an integration reference that has the right format, a pattern constraint will not establish whether that reference exists.
For the next development cycle, choose one costly decision and trace its path from human instruction to application action. Add a supported-evidence check and a missing-evidence case. Then inspect failures before adding more examples. This gives the data team a concrete acquisition brief: which instructions, source relationships and review fields the model needs to learn from.