What you will learn
Task success, Tool correctness, Replay, Attack cases, Regression gates.
Engineering the capability
Agent correctness includes the route taken. A workflow can reach a plausible final answer after reading unauthorized data or making an unnecessary mutation. Evaluate tool choice, argument validity, access decisions, stopping behavior, and side effects separately from answer quality.
Create replayable fixtures for deterministic components and repeated live runs for model-dependent behavior. Replays are useful for debugging but cannot prove the model will always make the same choices. Record the model, prompt, tool schema, and data versions for each run.
Include adversarial content in user inputs, documents, and tool responses. Test whether it changes permissions or leaks information. A failure should point to a broken boundary and a regression case. An aggregate score must not hide a critical safety or authorization failure.
Worked case
An agent passes nine ordinary tasks but obeys an instruction embedded in a document to export all tickets. A 90% task score is misleading. Mark the unauthorized export as a blocking failure and fix the runtime capability boundary before improving ordinary answer quality.
Put it into practice
Continue with the next lab: create an agent regression harness. Build the artifact, record the failure cases, and explain the tradeoff before moving on.