Map the change at the workflow level
WILDS studies distribution shifts that occur in real applications. Our practical starting point for enterprise AI is a workflow map, not a promise that one benchmark predicts deployment. Compare the demonstration and customer across document formats, vocabulary, task sequence, tool availability and acceptance rules.
Ask which stage first differs. A request may use unfamiliar terminology, the system may retrieve the wrong revision, or a valid output may fail the customer’s required approval format. Record these separately. Include who supplies the input, who checks the result and what context each person assumes. Two firms can request the same deliverable while expecting different evidence before accepting it. Do not collapse that distinction into a generic domain label.
Replay an authorized customer task with exact context
Obtain an appropriately scoped example of the failed workflow and preserve the input, accessible sources, tool results and rejected output. Use the customer’s ordinary access conditions in the replay. A reviewer should document the rejection reason, including any additional information they used to judge the result.
Then vary one factor at a time: provide a corrected extraction, the applicable revision or an explicit acceptance rule. Observe whether that change resolves the failure. If the assistant succeeds only when a reviewer supplies context absent from production, repair context delivery before claiming a training problem. If it fails despite complete context, investigate task behavior, tool use and target quality. Keep results tied to that bounded experiment rather than generalizing from one unusually clean demonstration.
The same request can mean a different sequence of work
Illustrative example: a demo assistant receives a single approved brief and produces a requirements checklist. A customer asks for the same checklist, but their current workflow includes an original brief, a later change notice and comments that are still awaiting approval. The assistant treats every comment as an accepted requirement.
The failure is not necessarily unfamiliar vocabulary. It may be a missing distinction between proposed, accepted and superseded instructions. Ask the customer’s domain reviewer to identify which source determines status and how unresolved comments should appear. Build a task that tests that distinction explicitly. If the package lacks approval evidence, preserve an unknown state. A finished source file cannot settle an acceptance decision that depends on a separate, unavailable record.
Build a coverage matrix before collecting more examples
List the task families the customer actually needs and the evidence each requires. Mark coverage for initial human input, follow-up instructions, result versions and validation records. Also record source language, software version, template family and extraction quality. Missing cells should direct the next sample request.
Choose examples that expose the demonstrated failure, including ordinary tasks where the product already succeeds. Real professional workflow evidence can reveal revision practices and acceptance boundaries that a polished synthetic scenario omits. Synthetic variations can still help test isolated format or terminology changes. Keep them labeled and compare their usefulness on independently reviewed customer tasks. More files from the same familiar workflow may expand volume without addressing the shift that caused the new customer to reject the result.
Choose an adaptation that matches the observed gap
If source text is missing or misread, test extraction and retrieval changes. If the assistant confuses status categories, test a clearer prompt and reviewed task examples. If it cannot create the required native output, inspect tool interfaces and file validation. Fine-tuning is one candidate intervention, not the default explanation for every deployment failure.
Separate a customer-specific configuration from an improvement intended for the whole product. Hold out customer projects when evaluating adaptation, and replay existing customer workflows to look for regressions. Record latency, review work and successful task completion alongside model scores. A domain reviewer should assess decisions that depend on professional context. Agree on limited rollout conditions and a rollback path. Success in that pilot supports expansion within its measured coverage, not an assurance about every enterprise workflow.
Turn the next sales demo into a more useful test
- What changed in sources, tools, task order and acceptance?
- Which controlled replay isolated the failing stage?
- Does the proposed data cover that stage and its missing evidence?
- Have both new and established workflows been reviewed after adaptation?
Common question: should we train a separate model for each customer? That decision depends on the observed failures, deployment constraints and operational cost. First test whether configuration, retrieval or a shared task representation addresses the difference. Ask for a sample spanning the customer’s workflow boundaries before committing to a large corpus. Define source licensing, task preparation and domain review separately, with explicit acceptance criteria for each. The aim is an inspectable adaptation decision rather than another demonstration chosen because it happens to work.