Ask what the label is supposed to represent
A label such as correct can refer to factual support, compliance with a brief, usability of a file or acceptance by a customer. Those are different judgments. Write the intended question beside the label definition and identify who has authority to answer it. A domain reviewer can assess a design requirement without being the person who approved the actual project.
Our practical approach is to separate criteria before aggregating them. Score whether the output follows the stated instruction, whether the format is usable and whether available checks support the result. Keep documented approval status separate. The Data Cascades study describes downstream problems arising from data issues. The annotation protocol below is a proposed operational method, not a result established by that study.
Confirm that reviewers saw the same context
For a disputed record, compare the actual source bundle shown to each reviewer: initial human input, later instructions, artifact versions and available review notes. Record which documents were missing or inaccessible. If one person used an updated brief and another saw the original, their disagreement may be evidence of a version problem rather than inconsistent expertise.
Ask each reviewer to identify the source and criterion behind the judgment, using concise evidence references. Do not demand lengthy private reasoning transcripts to establish review quality. Check whether the instruction omitted a constraint the expert assumed from ordinary professional practice. If that constraint matters, decide whether the product is expected to infer it or receive it explicitly. Keep this decision in the rubric rather than quietly altering the target.
Classify the disagreement before resolving it
Useful categories include missing context, conflicting source versions, different acceptance criteria, genuine alternative solutions and reviewer error. Let annotators mark uncertainty or insufficient evidence instead of forcing a binary judgment. Record the category with the disputed criterion, not just an overall disagreement flag.
Then decide the appropriate response. Missing context calls for evidence or an unknown state. A rubric conflict calls for a clarified rule and review of affected records. Valid alternatives call for a representation that accepts more than one outcome. A demonstrable source-reading error calls for correction. Majority voting can summarize judgments, but it should not substitute for deciding which question the training label represents. Agreement obtained by hiding ambiguity can produce a confident model that learns the wrong boundary.
An output can satisfy one criterion and fail another
Illustrative example: a task asks for a room layout with a desk and a clear entry path. One reviewer accepts a drawing because the stated path is clear. Another rejects it because they expect access to a maintenance panel that the supplied brief never mentions. Neither judgment can be interpreted without the intended context.
Check whether the panel requirement exists in an accessible project document and belongs to the same version. If it does, add the relevant reference to the task. If it does not, decide whether professional review should introduce it as a new constraint. Preserve that as a change, rather than rewriting the original human input. The original drawing remains a result artifact; the reviewer’s comment becomes separate evidence about a criterion it did or did not satisfy.
Calibrate the rubric using disputed records
Reviewers should first label a shared calibration set independently. Discuss disagreements by criterion, revise ambiguous definitions and label a fresh set to check whether the clarification transfers. Keep a rubric version, reviewer role, evidence references and adjudication status with each prepared record. This makes later corrections inspectable.
Include cases that lack formal validation and cases with multiple acceptable outputs. Agree on how to express unknown approval, partially met requirements and alternative designs. An automatic checker can help with file structure or explicit constraints, while a domain reviewer handles context-dependent judgments. Keep the scope of each check visible. Synthetic calibration examples can isolate one ambiguous rule, but do not assume agreement on those examples proves that reviewers handle messy professional records consistently. Inspect a separate source-based sample before expanding annotation.
Define review deliverables before buying annotation
- What precise question does each label answer?
- Which sources and versions will every reviewer receive?
- How will uncertainty, alternatives and disagreements be recorded?
- Who can adjudicate a disputed criterion, and what evidence is required?
Common question: should we keep only examples with unanimous labels? That can remove useful hard cases without fixing the underlying ambiguity. Decide which examples support a defined training objective and which require a different task, additional context or an unresolved status. Ask an annotation provider to deliver the rubric and review evidence alongside prepared targets. Source-file licensing does not automatically include expert labels or validated task records. Measure the usefulness of the resulting records in a scoped model pilot, keeping evaluation projects separate from annotation-driven training preparation.