Make quality changes visible before rollout.
1. Representative datasets
Include normal tasks, ambiguous requests, unauthorized requests and adversarial documents. Keep held-out examples separate from prompt development.
2. Task-level scoring
Measure factual support, tool correctness, schema validity and user effort separately. Aggregate scores should not hide a critical safety or authorization failure.
3. Change attribution
Version prompts, models, retrieval configuration and tool contracts. Change one major variable at a time when diagnosing a regression.
Worked scenario
A new retrieval setting improves average answer quality but exposes stale restricted documents. A segmented evaluation catches the unacceptable failure.
Apply it
Define release thresholds before comparing two configurations on the held-out set.
Check your understanding
Your release gate can reject an attractive demo that fails important edge cases. Explain the decision and show evidence from your implementation or design. If you cannot demonstrate it yet, revisit the relevant section before continuing.