Separate accepted arguments from an accepted outcome
Define distinct checks for the tool call. Did the arguments match the schema? Was the operation permitted for this user and task? Did it target the right entity and version? Did the resulting state satisfy the request without violating other constraints? Passing the first check leaves the remaining questions open.
Log these outcomes separately. A schema error suggests a different preparation need from an unauthorized update or a change to the wrong revision. Avoid labeling every successful API response as a positive training example. The tool may have executed a harmful or irrelevant decision faithfully. Review the decision and the state transition before using that trajectory as a target.
Give each evaluation task an initial and target state
Specify the state before the agent acts, the user's goal and the allowed changes. Include the facts needed to distinguish eligible and ineligible entities. For a workflow that modifies records, inspect the final state rather than relying on the assistant's final message.
tau-bench evaluates tool agents through user interaction, domain policies and final database state. Your domain still needs its own fixtures and acceptance rules.
Record which conditions must remain unchanged, not just the requested new value. A task can appear complete while the agent has modified an unrelated object. Where deterministic checking is possible, compare those invariants with the allowed target state. Keep human review for conditions the checker cannot establish.
Test whether the agent acts on the right version
Illustrative example: an agent is asked to add a review note to an approved drawing revision. A newer draft exists. The agent selects the most recently modified file, submits a valid call and receives success. It has changed the draft rather than the approved revision named by the workflow.
Make approval status, revision identifiers and selection criteria visible in the task. Test cases where chronological order and permitted target differ. Review how the agent resolves ambiguity and whether it verifies the selected entity before writing. Professional source files can provide realistic version context, but preparing an executable example also requires defined tool behavior, initial state and a reviewed target action.
Teach when a tool should not be called
Prepare cases where the user goal is incomplete, the operation requires confirmation or the agent lacks authority. Define the expected clarification or escalation behavior. A dataset that rewards a call on every request may teach action even when waiting is correct.
Separate application policy from facts found in project documents. A source requirement does not by itself authorize an agent to publish, delete or approve a record. Review both the policy and the task evidence when labeling the next action. Include cases where a legitimate user request conflicts with the available operation or requires a different workflow. Score inappropriate action independently from failure to finish an allowed task.
Exercise stale reads, retries and partial completion
Use controlled fixtures in which a read is followed by a changed version, a write returns a conflict or a response leaves execution status uncertain. Evaluate whether the agent refreshes state, checks the result and avoids duplicating a completed action. Define recovery behavior according to your API contract rather than expecting the model to invent it.
Record expected state transitions for each recovery path. Distinguish a safe retry from a repeated non-idempotent action. Include tool errors in reviewed examples, together with the context that makes the correct response clear. Successful straight-line traces cover only one part of the workflow. A reliable test also inspects how the agent behaves when its assumptions stop being true.
Build training records around observed decisions
A usable trajectory record should identify the user input, applicable policy, state available at each step, tool schema version, call arguments and returned observation. Its review metadata should explain why the next action is appropriate and how final-state correctness was established.
Do not infer the whole trajectory from a final project artifact. A drawing may show an outcome without revealing the API calls, intermediate states or authorization decisions that produced it. If you construct an illustrative trajectory from source context, label it accordingly and validate it in the intended environment. Keep original source references and preparation details separate from observed execution traces, so the buyer can review what each record actually demonstrates.
Choose a fix using the failing layer
Compare the baseline with targeted interventions: clearer schema descriptions for argument errors, improved state retrieval for stale decisions, explicit policy context for permission mistakes or reviewed task examples for poor action selection. Keep fixtures and acceptance rules fixed while testing each change.
Report valid-call rate, permitted-action behavior, final-state success and recovery failures separately. Repeat tasks where execution is variable and inspect unintended changes. An improved syntax score cannot justify release if business outcomes deteriorate. When sourcing additional project data, identify the missing context it must supply. Native materials can ground a preparation pilot, but reviewed executable trajectories need separate work, and their value must be demonstrated in your actual tool environment.