Separate file growth from learning coverage
A source collection can grow through new projects, additional exports, revised documents or repeated templates. Those additions have different implications for training. The question is what new behavior an accepted record asks the model to learn.
Report volume alongside a coverage inventory. Useful dimensions might include task type, discipline, instruction complexity, missing information, change type and review status. Choose dimensions that affect your product’s decisions. A thousand files do not necessarily represent a thousand distinct situations, and a short technical change can contain more valuable variation than a large export. Coverage needs a task definition to become meaningful.
Distinguish duplication from legitimate recurrence
Research on training data deduplication reports benefits from removing repeated material and reducing train-test overlap. Our recommendation is to examine duplicates at several levels.
Exact byte copies are straightforward candidates for consolidation. Near-identical text may require more care. Recurring requirements can express a genuinely common workflow, while a tiny change can alter the correct decision. Treat recurring patterns as information about frequency, rather than automatically deleting every repetition. Establish whether a record is another copy, another instance of the same decision, or evidence of a different decision. Those categories support different sampling choices.
Protect the change that matters
Consider an illustrative pair of briefs. Both describe the same lighting-control system. One requests dimming in shared spaces; the other excludes dimming in an emergency circuit. A similarity score may place the documents close together, yet the exception changes the model’s required behavior.
Compare the instruction and relevant outcome context together where those links exist. Ask whether differences alter constraints, entities, priority or acceptance. Preserve meaningful variants while grouping repeated background material. If the relationship to an outcome has not been verified, label that limitation. Document similarity alone cannot establish that two training tasks have the same correct answer.
Build a coverage table around failures
Take recent product failures and map them to needed examples. Did the model ignore an exception, assume a missing reference, confuse disciplines or treat a proposal as an approved result? These observations can define coverage categories more usefully than counting file extensions.
For each category, record distinct source families, candidate records, accepted examples and known gaps. Source families might group related projects and their revisions. Include the review status so a category does not appear covered merely because a relevant filename exists. This table turns “we need more data” into a request for specific decision patterns and supporting context.
Keep related material out of both splits
When preparing evaluation, examine connections between files as well as direct copies. A different export of the same model, a later brief revision or a copied template may share substantial task information with training records.
- Group related source files before assigning splits.
- Check exact and near-duplicate content across groups.
- Inspect whether shared context exposes the expected answer.
- Record uncertain relationships for review.
This is an original split-design checklist. Its purpose is to make the evaluation question credible: can the model handle a new task instance? A clean filename list alone does not answer that question.
Ask for distinct coverage in the next batch
Give a supplier a request framed around missing behaviors. For example, ask for instructions containing competing constraints, supported examples of requirement changes, or files from a discipline that your current preparation pipeline has not covered. Confirm which fields actually exist before planning annotations.
Then compare the proposed batch against the current inventory. Separate genuinely new source families from additional versions of material already held. Repeated workflows may still be useful for balance, but make that choice deliberately. Optional extraction or expert annotation should have its own deliverables and acceptance criteria. A more varied source package and a prepared task dataset are separate commercial items.
Should common workflows be downweighted?
Sometimes common tasks deserve substantial representation because they dominate customer use. In other cases, their repetition consumes preparation effort without improving the product. Decide using the intended workload and observed errors, rather than a universal rule about duplicates.
Before the next purchase, review one heavily represented workflow and one weakly represented failure mode. Count their distinct decision patterns, not just their records. Run a bounded comparison when changing the mix, and retain the dataset manifest. This gives your team a defensible reason to expand supply: improve coverage where it matters, maintain representative frequency and avoid paying repeatedly for the same evidence.