Look for the friction your generator never saw
Consider an illustrative building-layout task. A generated request asks for a revised seating plan. A real task may also involve an unchanged access route, a late client correction, an obsolete drawing, an unresolved dimension and a reviewer who accepts only part of the revision.
If those conditions never enter the generator’s context, it may produce many variations of an easier problem. The dataset grows while the important gap remains. Ask what observable customer difficulty each generation recipe is designed to represent.
Real sources are not automatically complete or correct. Their value comes from inspectable context and outcomes, followed by preparation and review. The useful comparison is task coverage and evidence, not a label alone.
Where synthetic examples can help
Synthetic data can provide paraphrases, controlled variations and examples for a clearly defined skill. The Self-Instruct paper reported instruction-following improvements using generated examples with filtering. That is evidence of usefulness in its setting, not a guarantee for a particular commercial workflow.
A practical approach is to generate variations around a reviewed task: change the wording, introduce a known ambiguity or vary an allowed input format. Have a reviewer check whether the expected answer still follows from the revised input.
Record the source task, generation method and review status. A paraphrase linked to a real task and a wholly invented scenario are different kinds of evidence. Keep that distinction visible when selecting training and evaluation records.
Three ways a generated dataset can give false confidence
Invented completeness. The generator supplies a missing dimension or approval because a complete answer is easier to write. Your deployed assistant will not necessarily have that information.
Shared blind spots. If the same model generates examples and judges answers, the evaluation may reward its preferred style or assumptions. Independently reviewed task checks can expose failures a fluent answer conceals.
Missing consequences. A generated target may treat a draft, a technical check and a client approval as interchangeable. In a real workflow, those statuses determine what action is permitted next. Include status and decision evidence where the task depends on them.
Does synthetic data always cause model collapse?
No. The training setup matters. A Nature study on recursive generated data demonstrated degradation under the conditions it examined. A separate study of accumulating real and synthetic data found that retaining real data alongside generated generations avoided collapse in its experiments.
These findings do not justify predicting collapse from every use of synthetic fine-tuning data. For a founder, the immediate questions are more concrete: what distribution does the dataset cover, what source information is retained, and does the resulting system work on independent customer tasks?
Do not choose a universal real-to-synthetic ratio from a headline. Test the mixtures relevant to your model, task and preparation process.
Compare data mixtures without contaminating the test
Prepare evaluation tasks from separate permitted projects. Define checks before inspecting candidate outputs. Keep source-linked paraphrases and revisions within the same split; otherwise a supposedly unseen task can repeat training content.
Compare a reviewed real-source set, a generated set and a scoped mixture where practical. Document differences in task coverage and preparation effort rather than attributing every change to origin alone. Keep the model configuration and evaluation process consistent.
Include missing information, contradictory revisions and clarification cases. Measure how much work remains for a human reviewer, not just whether the response looks plausible. Our pilot checklist provides a structure for the comparison.
Ask suppliers what happened outside the generated answer
Request source context, revision links, output status and the basis of any review label. Ask which records were generated, transformed, reconstructed or directly extracted. An honest answer about unavailable evidence is more useful than a complete-looking record built on assumptions.
For professional project data, inspect one chain from requirement to revision to result. Determine whether it represents the business task you need and whether the proposed license covers your intended preparation and model use.
YourSOTA provides a starting point for that discussion through professional architecture, construction and smart-building source material. Complete conversations and validation evidence are scoped by availability; they are not implied by the presence of a project archive.
Can synthetic data replace all domain data?
That depends on the task and the evidence you can independently validate. If generated examples cannot represent the decisions, exceptions or acceptance rules your product relies on, you need a way to obtain that knowledge. Real workflow material and qualified reviewers can supply a basis to test those assumptions.