What you will learn
State machines, Checkpoints, Replay, Concurrency, Compensation.
Engineering the capability
An agent that runs for minutes cannot rely only on process memory. Persist task state at meaningful transitions with a version number. A checkpoint should identify completed operations and pending work without requiring the model to infer what happened from prose.
After a crash, replay may repeat the last operation. Use durable operation IDs and idempotent effects so recovery does not duplicate work. A database transaction can atomically record local state, but a remote action requires reconciliation or an outbox-style delivery design. Exactly-once behavior across arbitrary systems is not achieved merely by writing “completed” to a row.
Concurrent workers can resume the same task. Use a lease or compare-and-set transition to ensure one worker owns the next state change. Define cancellation and compensation separately: stopping future steps does not reverse an email already sent or a record already created.
Claim a state transition atomically
import sqlite3
connection = sqlite3.connect(":memory:")
connection.execute("CREATE TABLE jobs (id TEXT PRIMARY KEY, state TEXT, version INTEGER)")
connection.execute("INSERT INTO jobs VALUES (?, ?, ?)", ("job-1", "pending", 0))
connection.commit()
def claim(job_id, expected_version):
with connection:
result = connection.execute(
"UPDATE jobs SET state = 'running', version = version + 1 "
"WHERE id = ? AND state = 'pending' AND version = ?",
(job_id, expected_version),
)
return result.rowcount == 1
assert claim("job-1", 0)
assert not claim("job-1", 0)
The conditional update prevents two claims of the same pending version. A production worker still needs a lease, recovery policy, durable storage, and idempotent remote effects. This example demonstrates a local transition, not exactly-once distributed execution.
Worked case
A workflow updates a ticket and crashes before recording success. On restart, querying the remote operation ID can reveal the update already occurred. Blindly replaying the model conversation and running the tool again risks a second update. Recovery should use durable facts rather than regenerated guesses.
Put it into practice
Continue with the next lab: recover an interrupted agent task. Build the artifact, record the failure cases, and explain the tradeoff before moving on.