What you will learn
SLOs, Tracing, Alert ownership, Runbooks, Postmortems.
Engineering the capability
Define service objectives around the workflow: successful draft generation, response time, and availability of a manual fallback. Avoid alerting on every transient model error if users still receive a valid result; alert on conditions that require action. Every alert needs an owner, a diagnostic starting point, and an escalation path.
Use a request ID to connect application, retrieval, and integration events. Retain useful metadata while minimizing sensitive payloads. A customer incident may be an identity change, expired secret, source schema update, or network rule, not just a model regression.
A runbook should help someone other than the builder determine impact, contain the problem, recover, and verify normal operation. A postmortem records the timeline and contributing conditions, then assigns specific improvements. “Be more careful” is not an actionable prevention measure.
Worked case
An alert reports a spike in failed drafts. Traces show document retrieval succeeds but the ticket connector returns access denied after a role change. Restarting workers will not fix the permission mapping. The runbook should point to identity configuration and preserve manual access while the owner resolves it.
Put it into practice
Continue with the next lab: run an incident tabletop. Build the artifact, record the failure cases, and explain the tradeoff before moving on.