Identify what changed in the workflow
Do not schedule another data purchase simply because time passed. Start with evidence that the product’s task distribution changed: a new customer segment, additional engineering discipline, different source format or revised approval process. Define the behavior your existing data no longer represents.
A building assistant that previously extracted room requirements may now need to interpret changes across disciplines. That creates a different acquisition brief. More copies of the earlier workflow may not help. Capture the new inputs, decisions and acceptance criteria before requesting supply. Continuous data work should follow a changing product need, not become a recurring volume target without a task.
Give every source update a version
Maintain an immutable record of each received package, including its source identifiers, delivery date, file inventory and known relationships to earlier material. A later source version should not silently replace the file used to prepare an earlier training record.
Link prepared examples to the specific source versions and preparation process that produced them. If the team changes its extraction method or review rubric, record that separately from a source change. This makes a result reproducible: you can determine whether a behavioral difference followed new evidence, a new transformation or a different acceptance decision. Otherwise, “latest dataset” becomes an ambiguous explanation for every change.
Use documentation as an operating record
Datasheets for Datasets proposes documenting dataset motivation, composition, collection and recommended uses. Our recommendation is to maintain that documentation with each release.
Add a practical change record: what was received, what was excluded, which task categories gained accepted examples, and which gaps remain. Identify the source languages and review stages represented. Describe whether new records contain authentic historical instructions, newly prepared tasks or both. These distinctions help product and procurement teams assess an update. Documentation should explain the delivered package rather than promise that every future batch will contain the same fields.
Make update acceptance a separate gate
Receiving source files is one event. Preparing candidate records is another. Accepting reviewed training examples is a third. Keep these stages distinct so a supply commitment does not turn every incoming artifact into training material by default.
For each update, verify that files open, relationships are usable and the selected task remains supported. Review the intended handling and licensing scope as appropriate for new material. Set aside unresolved records. If a source update changes an answer, record why and which earlier examples are affected. A new filename alone does not establish an improved target, and a later draft is not automatically an approved outcome.
Reserve fresh material for evaluation
Keep a portion of newly available, appropriately scoped material outside training preparation for a fresh evaluation. Group related projects and revisions together. Otherwise, the model may encounter the same underlying task through an earlier version and make the fresh test appear easier.
Compare performance on the established evaluation and the new workload. The former checks whether existing behavior deteriorated; the latter tests the change you are trying to address. Record failure categories and review disagreement. This proposed evaluation process needs task-specific expected outcomes. The existence of new files does not imply that a ready benchmark or completed validation report is available.
Agree on an update brief, not just a cadence
A supply discussion should cover what the next delivery aims to add and what can actually be provided. Set practical expectations for formats, source fields, task categories, selection method and preparation options. Include the handling of incomplete or delayed material.
- Define the source release identifier.
- Specify candidate and accepted record counts separately.
- Identify review responsibilities and acceptance checks.
- Describe known gaps and changes from prior deliveries.
Frequency and volume can be discussed commercially, but the content brief makes them useful. Confirm available supply before committing to a schedule. Custom preparation, annotation and review should remain explicit work items.
Do fresh sources require immediate retraining?
No fixed update frequency fits every product. Some new information belongs in retrieval or an application rule. Some changes require better examples of behavior. Other updates may add no useful coverage. Evaluate the gap before choosing the intervention.
Use a release review to decide: what changed, which source evidence supports the change, what preparation is needed, and how will improvement be measured? Then accept the relevant examples and preserve a fresh test set. A supplier becomes a useful long-term partner when the relationship supports those decisions with scoped, documented updates. Continuous supply creates options; it does not guarantee that every delivery should trigger a training run.