Write one question and a decision rule
Choose a skill your product repeatedly needs. For an engineering assistant, that might be extracting requirements accurately or preserving an earlier constraint after a user changes the request. Describe the current failure and how a reviewer will recognize a successful response.
Agree on the decision rule with the people who will approve a larger purchase. You may need better task success, fewer serious errors or less manual review without an unacceptable increase in latency or preparation cost. Pick criteria that matter to your product instead of a score that happens to be easy to calculate.
Do not set a universal target from another company’s benchmark. The task mix, model, prompts and operating constraints in your pilot determine what a result means.
Select the sample around coverage, not archive size
Ask for material covering ordinary cases, known failure cases and the variation your product must handle. Identify the languages, formats, disciplines and histories needed. Specify whether you want native sources, prepared examples or both.
A small, coherent sample can expose a missing pipeline dependency quickly. A larger archive may still lack the instruction, the correct output version or the review evidence required for your task. Record unavailable components explicitly.
For planning, estimate the number of independent tasks your team can inspect and evaluate within its budget. Explain how the sample was selected. Keep the selection criteria in the pilot brief so a successful narrow sample is not mistaken for proof about every project in the wider corpus.
Freeze the baseline and hold out related projects
Record the baseline model, prompts, tool setup and the context it receives. If the baseline lacks relevant source material, test improved retrieval or prompting before attributing a failure to insufficient training. The RAG versus fine-tuning guide explains that diagnostic step.
Group related project documents, revisions, exports and prepared records before assigning training and evaluation splits. Keep your evaluation set outside the preparation feedback loop used to improve training examples. Check for repeated templates across the groups.
Use the same evaluation tasks and scoring method for the baseline and candidate. Record other changes separately. Otherwise, better retrieval or a revised prompt can be confused with a benefit from the newly purchased training data.
Track preparation as part of the experiment
Log the work between receipt and usable examples: opening files, extracting content, resolving attachments, translating, pairing versions, annotation and human review. Count rejected examples and record why they were rejected.
Separate repeatable pipeline work from one-off setup and manual corrections. This gives you a more useful estimate of the next delivery than dividing the entire pilot budget by the number of files. Confirm whether the supplier’s quote includes the preparation your team actually needs.
A simple working measure is total preparation cost divided by accepted examples. Review it alongside coverage: inexpensive examples are not useful if they miss the target skill. Also track the cost of scoring and correcting model outputs, since that effort affects your product’s operating economics.
Use a scorecard that exposes the failure
Score the outcomes at the task level and keep examples of errors. Where possible, have reviewers compare outputs without being told which run produced them. Define the rubric and resolve reviewer disagreements in a recorded process.
| Measure | What it helps you decide |
|---|---|
| Task success | Does the model complete the requested behavior? |
| Constraint violations | Which requirements are missed or changed? |
| Unsupported answers | Does the model claim facts the inputs do not establish? |
| Human review effort | How much work remains before the response can be used? |
| Latency and operating cost | Can the improvement fit the product’s delivery constraints? |
| Preparation yield | How many inspected records become accepted examples? |
Inspect task subsets as well as the overall result. A gain on easy extraction tasks can hide unchanged performance on the revisions your customers care about.
Decide what the next delivery should contain
Document the baseline, candidate, task mix, sample size, failures and limitations in a short decision memo. A promising pilot supports a scoped next purchase; it does not guarantee the same outcome across new languages, disciplines or models.
The NIST AI Risk Management Framework emphasizes evaluation in the intended context. Our practical recommendation is to use that context to define the next delivery: more examples of the failing skill, broader project coverage, improved preparation or a different experiment.
Agree on licensing and delivery terms for the selected scope. Keep update requirements and acceptance checks explicit. If the pilot does not support expansion, preserve the findings; knowing why the data does not fit is still a useful buying decision.