Define a slice by the decision that fails
Group failures around a concrete behavior: applying an exception, preserving a constraint after revision, selecting the approved version or requesting missing information. Labels such as engineering data or difficult examples are too broad to guide a mixture decision. A slice should tell the team what the model must do differently.
Give each slice a reviewed evaluation set and a business interpretation. Some errors require correction time; others block the task entirely. Keep ordinary task coverage visible too. A mixture that improves a rare case while damaging frequent requests may be unsuitable. Define the trade-off before looking at results, so a convenient headline score does not become the decision rule.
Measure the distribution after sampling and packing
Inspect what reaches the trainer after filtering, deduplication, sampling and sequence packing. A small collection of long examples can consume substantial token exposure. Repeated short records can dominate the number of training presentations. File totals do not reveal either pattern.
Record example counts, token contribution, repeat exposure and coverage of distinct projects or scenarios for each slice. Inspect near-duplicates as well as exact duplicates. Many paraphrases of one easy decision may add volume without adding meaningful variation. Compare the planned mixture with the realized batch distribution, including records dropped for length or validation failures. Otherwise the experiment may test a different mixture from the one your team intended.
Borrow the experimental question, not a published ratio
DoReMi studies domain reweighting for language-model pretraining. That research is not a validated fine-tuning recipe for your domain task.
For your own pilot, ask whether changing exposure changes the behavior that matters. Compare a baseline mixture, a targeted addition and a reweighted candidate under a controlled training budget. Keep the model initialization, evaluation and serving setup consistent. Record how budget is controlled, because equal numbers of records do not necessarily imply equal training tokens or compute. The purpose is to identify a useful direction for your product, not to reproduce another setting's proportions without its assumptions.
Use synthetic examples for a defined coverage job
Synthetic examples can help vary wording, exercise a known schema or isolate a particular rule. State what they are intended to add, then check whether the targets remain correct. Track their generating process and their relationship to source scenarios. Variation should not introduce contradictions that a reviewer would reject.
Professional records can expose constraints or dependencies that your generation procedure did not include. They can also contain incomplete context, inconsistent terminology or errors of their own. Evaluate both sources against the same task contract. The useful distinction is which reviewed behavior each contributes. Neither synthetic origin nor professional origin establishes that a record deserves greater weight in your training run.
Add contrastive cases that change the correct decision
Illustrative example: routine room-assignment examples use the stated capacity directly. A costly failure arises when a later exception restricts a particular room. Useful additional records should test that exception and its boundary, rather than merely restating ordinary assignments in more elaborate language.
Prepare cases with and without the exception, preserving the other relevant conditions. Include situations where the correct response is to clarify because the available evidence is insufficient. Review the targets and keep related variants together when splitting data. This creates a focused test of the decision boundary. Repeated exposure to one rare case can still encourage memorization, so include held-out projects or independently reviewed scenarios that require the same behavior in different context.
Choose the mixture using slice results and uncertainty
Run candidates against the frozen ordinary-task suite and the expensive-error slices. Report the number and coverage of reviewed cases alongside the results. Inspect individual paired outcomes, especially when the evaluation set is small or outputs vary across runs. A change in an aggregate score can hide which decisions improved and which became worse.
Track preparation effort and correction cost with task success. If the targeted mixture improves only examples that closely resemble its additions, expand the evaluation before committing. If it increases refusals or unnecessary clarification, inspect those behaviors separately. Avoid repeatedly reshaping the training mixture around the same evaluation errors without keeping a fresh holdout for the final choice.
Turn an uncovered slice into a sourcing requirement
Ask for material that could support the missing task, including the human requirement, applicable exception and relevant artifact version where available. Specify whether you need raw sources, reviewed task records or additional evidence to construct an evaluation. An extra archive helps only if preparation can expose the decision your current data misses.
Inspect a representative sample before setting an expansion target. Confirm availability, linking limitations, preparation cost and permissions for the intended use. Then test the added slice with the same controls used for earlier candidates. A mixture is a revisable experimental choice. Expand it when the evidence supports better coverage and acceptable product behavior, rather than when the dataset has simply become larger.