Define the product failure before interpreting the curve
Our practical approach starts with a failure ledger. For each held-out task, record the original human input, relevant context, generated output and the reason a reviewer rejected it. Separate missing requirements, unsupported claims, unusable file structure and incomplete work. A single overall quality score hides which behavior changed.
Compare the base model and selected checkpoints using the same prompt, retrieval context, tool access and generation settings. If those differ, the experiment cannot isolate the trained model. Record the effective configuration rather than the configuration someone intended to run. Establish a customer acceptance rule before looking for the checkpoint that appears best. Otherwise, the team can keep redefining success around whichever output looks most polished.
Inspect what actually contributes to the loss
Open several serialized training records, including long and short tasks. Display the text, role boundaries, token positions and loss mask together. Confirm which tokens are targets. Depending on the training setup, copying input text or predicting repeated boilerplate may contribute more to the reported average than the part customers care about.
Inspect truncation after tokenization. A record that originally contained requirements and a completed result may lose an important constraint or the final output when it is shortened. Check whether the instruction template, end markers and tool messages match the intended deployment format. Do not change masks blindly: first write down the intended learning objective, then verify that the implementation represents it. Keep this inspection with the dataset version so later runs can reproduce it.
Check which records dominate the average
Group the records by task type, project, source format, instruction length and output length. Count examples and target tokens separately. A small number of long documents can occupy a large share of training without representing a large share of customer requests. Repeated exports or templated paragraphs can also distort the apparent coverage.
Compare task-level evaluation with per-group loss where your training system exposes it. Look for groups whose text becomes easier to predict while constraint adherence deteriorates. Before adding examples, inspect whether duplicated targets, easy formatting tasks or one project family dominate the batch stream. A practical correction is a deliberately scoped sampling experiment. Change one weighting decision, record what it changes, and evaluate the same held-out tasks rather than assuming balance automatically improves the product.
A lower-loss answer can still miss the job
Illustrative example: an assistant turns a technical brief into a structured requirements list. Most training outputs contain a familiar introduction and the same field names. The customer, however, needs a changed accessibility requirement preserved in the correct field. A new checkpoint produces cleaner formatting but omits that change.
Inspect whether the changed requirement is present in the model input, survives tokenization and appears in the target. Then compare the checkpoints on that requirement separately from formatting. If the source record contains only the final brief, do not invent the missing conversation to explain the revision. Real document edits can support a narrower task. Linking human input, revisions and result files requires explicit preparation; available validation records should remain a separate evidence layer.
Compare checkpoints without contaminating the decision
Use a development set to select checkpoints and keep a separate final test for the release decision. Group related files and revisions by project so a familiar answer does not quietly appear on both sides. When coverage permits, include tasks with missing information, changed instructions and unfamiliar document structure.
Track several observable outcomes: accepted task completion, missed constraints, correction work and valid output structure. Include earlier checkpoints as well as the last one. If later training loses useful behavior, investigate the data mix and training settings with a controlled comparison. Preserve a rollback candidate. Synthetic examples can help test rare instruction changes, but success on those examples needs confirmation against independently reviewed professional workflows. Do not use more synthetic variations as the only evidence that the original failure is resolved.
Decide what evidence justifies another run
- Can the team show exactly which tokens the objective trains?
- Do project-level evaluation tasks represent the rejected customer work?
- Are the important constraints visible in both inputs and reviewed targets?
- Did one controlled change improve acceptance without damaging another task group?
Common question: should we simply stop training earlier? An earlier checkpoint is worth comparing, but it does not diagnose a malformed target, missing context or wrong acceptance rule. Choose the checkpoint that meets the agreed task criteria, then test the suspected cause. If the evidence is missing from the corpus, scope additional source material or domain review before another large run. The useful purchase is the evidence that closes that gap, not an arbitrary increase in file volume.