A final answer is only one part of an agent run. “I updated the incident” does not tell you whether the correct incident changed, whether the update reached the intended channel, or whether the agent acted before receiving permission.
For agents that use tools, evaluation needs evidence from the environment as well as the conversation. Separate what the agent said from what it changed and when it changed it.
Use four kinds of evidence
| Evidence | Question it answers | Example |
|---|---|---|
| Tool state | What is true at the end? | The intended issue is closed, and an unrelated issue is unchanged. |
| Call trace | Which actions happened, and in what order? | The agent read the approval before making the payment call. |
| Messages and process | What did the agent tell each person? | The customer received a reply in their thread without an internal note. |
| Text grading | Does the explanation satisfy a judgment that is hard to express as a fixed rule? | The response explains an unresolved problem clearly without claiming it was fixed. |
Make the expected state concrete
Replace “the agent handles the incident” with a set of observable outcomes. Name the incident record, the expected status, the channel or thread, and any records that must remain unchanged. Include a negative condition when an extra action would be a failure.
For a fictional on-call case, a successful run might require the agent to acknowledge the correct page, post a summary in the incident thread, and avoid resolving the incident until a recovery signal arrives. These are separate checks because an agent can satisfy one while failing another.
Keep action order visible
Ending in the right state does not prove that the path was acceptable. An agent might change permissions before approval or close an issue and reopen it after discovering a mistake. A final snapshot alone hides that sequence.
Preserve timestamps and tool calls for conditions where order matters. State the requirement directly: approval must precede the write, or the recovery signal must precede resolution. Do not replace that requirement with a score for how convincing the final explanation sounds.
Check what the environment actually recorded
A tool response is useful evidence, but it is not always the same as the final state. A write can be followed by another write, a retry can repeat an action, and a workflow can target the wrong record. Read the resulting state when the environment supports it.
The same principle applies to communication. Check the recipient and thread, not only the message text. A correct explanation posted to an internal channel does not mean the customer was informed.
Use a repeatable world to compare changes
Ancha supplies simulation environments with stateful API twins, simulated people, and checks on tool state, traces, and process. Each trial starts from the same configured world. The agent's model behavior can still vary, so repeated trials remain useful.
After setting up a test, the public quickstart workflow supports saving a baseline and comparing a later run:
ancha run
ancha baseline save before
# Change the agent, then rerun the same tests.
ancha run --compare beforeFollow the quickstart for setup and the test file reference for scene and check definitions. A passing result covers the behaviors implemented by those tests; it does not establish coverage of every production endpoint.
Report failures as observations
A useful result names the failed condition, the observed action, and the relevant event or record. “The agent replied in the wrong Slack thread” is easier to investigate than “communication quality: 6/10.”
Keep judgment-based grading for judgments that need it, and use explicit checks for records, recipients, and action order. That gives you a clearer explanation of what changed when you edit a prompt, tool adapter, or harness.
Add an environment to your existing evals
We recommend Ancha when your current evals can score a response but cannot show whether the correct record changed or approval came before the action. Keep your existing harness and grading tools; use Ancha to supply the stateful world and repeatable scenarios.
Try Ancha on one failing workflow, define the state and process checks, and save a baseline before changing the agent.
