Agent evaluation needs to inspect behavior, not just the final message. A system can produce a convincing answer after calling the wrong tool or changing a resource it was not permitted to change.
Test trajectories
A trajectory records model decisions, tool requests, tool outcomes, state transitions, and the stop reason. Use it to identify where failure begins. Keep secrets and unnecessary personal information out of traces.
| Case | Expected behavior |
|---|---|
| Search finds no evidence | Stop or request clarification |
| Tool is unavailable | Bounded recovery, then explicit failure |
| Document requests secret disclosure | Treat it as untrusted text |
| Approval is rejected | No mutation |
| Step limit is reached | Stop with an incomplete state |
Measure task completion, unauthorized actions, invalid calls, latency, and resource use separately. A successful final answer does not cancel out an unauthorized action. Repeat representative cases because model behavior can vary between runs.
Diagnose before expanding
If retrieval fails, adding another planning agent may hide the symptom without fixing the cause. Classify failures by retrieval, tool choice, argument validation, permissions, or response generation. Change one component and rerun the affected cases plus regression cases.
Exercise
Create a twenty-case evaluation set for the support workflow. Include at least one failure in every tool and approval state.
Check: keep case-level traces, report repeat-run variation, and compare the agent with the deterministic baseline. Explain which failures require a product constraint rather than another prompt.