PrerequisiteAI 202 recommended, plus Modules 1 to 6 of this course
0 / 6 completed
Sync my progressJoin a session to keep progress across devices and share it with your instructor
Learning objectives
Explain observability and the difference between a trace, a log, a metric and an event history
Describe a run as a record of state transitions, decisions, tool calls, verification results, human actions, errors and recovery actions
Trace an error back to where it began using error provenance
Choose reliability metrics such as completion rate, retry rate, escalation rate, recovery rate, latency, cost and step count
Define operational boundaries and safe stop conditions: step, retry, cost and time limits, allowed actions and required approvals
Complete Activity 7 by designing a minimal observability plan and drafting a Reliability & Observability Scorecard
What to observe
10 min
Why observability: runs, steps, events, logs and metrics
A harness that verifies, retries and recovers is only trustworthy if people can see what it did. Observability is the ability to understand what happened inside a run from the records the system leaves behind. Without it, a failed run is a black box, and a run that seemed to succeed cannot be checked.
Core question
When the system has a problem, can we know what happened, why it happened, and at which step?
The vocabulary of observability (simulated examples)
Term
What it is
Simulated example
Run
One execution of the workflow from trigger to final status
One run that processes a batch of incoming documents
Step
One unit of work inside a run
The "extract fields" step
Event
Something that happened at a point in time
Step 4 started; verification failed; a person approved
Event history
The ordered list of events for a run
The full timeline of that run, in order
Trace
The connected path of a run: which steps ran, in what order, with what result
Step 1, then 2, then a retry of 3, then a human review
Log
A recorded message or detail about something that happened
A tool returned a timeout error at 10:42
Metric
A number measured across runs or steps
Retry rate this week
Run status
Where the run stands now
Running, waiting for human, failed, completed
Info
Human, AI, Human: a person decides what must be observable, the harness records it, and a person reads the records and remains responsible for acting on them.
Key takeaway: A run leaves a trace, events, logs and metrics, and a status. Decide what to record before something fails.
10 min
The observability model and error provenance
A useful run record holds more than the final answer. It holds the path taken and the reasons along the way. Six kinds of information cover most needs, and together they let a person replay the story of a run.
Observability model: what a Run records
Run
State transitions
Agent decisions (decision trace)
Tool calls (tool-call history)
Verification results
Human actions (human intervention history)
Errors
Recovery actions
Info
Text version of the diagram: a Run is the container. Inside it the harness records the state transitions, the agent decisions, the tool calls, the verification results, the human actions, the errors and the recovery actions. Each item is attached to the run and to the step where it happened.
Error provenance means knowing where an error came from, not just that it appeared. A wrong final value may have started as a bad tool result three steps earlier that passed an unchecked step. To trace it, you need each error linked to its step, its inputs, the tool call involved and the decision that used it.
Give every run a run ID and every step an identity, so records can be joined
Record the state before and after each transition
Record which tool was called, with what inputs and what result, and keep sensitive values out of logs
Record who or what made each decision, including human approvals and overrides
Record the error, the step it occurred at and the recovery action taken
Info
Example only: a stateful framework such as LangGraph records graph state as execution moves from node to node, which can feed a trace. The concepts here do not depend on any one framework.
Check yourself
A run ended with a wrong value. What information best lets you find where the error began?
Key takeaway: Record state transitions, decisions, tool calls, verification, human actions, errors and recovery, all tied to a run and a step.
Reliability measures
9 min
Reliability questions and metrics
Reliability is how consistently a workflow does what it should. You cannot improve it by feeling; you need a few questions and the numbers that answer them.
Did the workflow complete?
How many retries occurred?
Where did failure happen?
How often was human escalation required?
Did recovery succeed?
How much did the run cost?
How long did it take?
Did the system stay within its boundaries?
Reliability metrics and what they show
Metric
What it tells you
Completion rate
How often runs reach a successful final status
Failure rate
How often runs end in failure
Retry rate
How often steps had to be repeated
Escalation rate
How often a person had to step in
Recovery rate
How often a failure was recovered instead of ending the run
Latency
How long a run or step took
Cost
What a run consumed overall
Step count
How many steps a run took, which can reveal loops
Check yourself
Which metric most directly shows how often a failure was handled instead of ending the run?
Key takeaway: Ask the reliability questions, then pick the metrics that answer them.
9 min
Reading reliability metrics together
A single number can mislead. Reliability evaluation reads metrics together and asks what the pattern suggests about the design.
Patterns and what they may mean (simulated examples)
Pattern
A possible reading
What to check
High completion, high retry rate
Runs finish, but only after many repeats
Cost and latency; a flaky tool or a weak verification gate
Low escalation rate
The system works well, or problems are hidden
Whether verification and escalation triggers are too loose
High step count with no progress
A loop that is not bounded well
Loop conditions and step limits
Low recovery rate
Failures are not classified, or fallbacks are weak
The failure taxonomy and fallback paths
Rising cost per run
Retries or context growth are increasing
Retry policy and context compaction
Warning
A metric is only useful if someone will act on it. Decide in advance what value is a problem, who is told and what they should do. Do not collect numbers that nobody reads.
Info
Boundary of this course: this module chooses what to observe and measure at the design level. It does not teach monitoring products, dashboards or general DevOps practice.
Check yourself
A workflow has a very low escalation rate. What is the most careful conclusion?
Key takeaway: Read metrics as a set, and define in advance what counts as a problem and who acts.
Operational boundaries
9 min
Operational boundaries and safe stop conditions
An operational boundary is a limit the system is not allowed to cross without a person. Boundaries keep a long-running or failing run bounded in cost, time and impact. They are decided before the run, and the harness enforces them during it.
Operational boundaries to define
Boundary
What it limits
Simulated example
Max steps
How long a run can go on
Stop after 40 steps
Max retries
How many times a step may be repeated
At most 3 retries per step
Max cost
How much a run may consume
Stop when the budget for one run is reached
Max runtime
How long a run may take
Stop after 30 minutes
Allowed tools and actions
What the run may do at all
Read-only tools unless a person approves
Required approvals
What needs a person first
Any irreversible or externally visible action
Stop conditions
When the run must halt safely
Limit reached, repeated failure or a safety violation
When a limit is reached, stop safely and record why, rather than continuing and hoping
A safe stop leaves the run in a known state that a person can inspect and, if appropriate, resume
Reaching a limit is a signal to escalate to a named person
Prohibited actions stay prohibited even if the agent seems confident
Warning
Do not design a harness that lets high-impact actions continue unchecked because a limit was never set. A run without limits is a run nobody is controlling.
Check yourself
A run reaches its maximum retry limit on a step. What should the harness do?
Key takeaway: Set step, retry, cost and time limits, allowed actions and approvals in advance, and stop safely when a limit is reached.
Practice and next steps
8 min
Activity 7: What Should We Observe?, and your scorecard
You will be given a simulated agentic workflow. Design a minimal observability plan for it, and decide how you would judge whether it is reliable.
1Define the run What is one run, what is its ID and what final statuses can it have?
2Choose what to record State transitions, tool calls, verification results, errors, recovery actions and human interventions: which are the minimum?
3Choose the metrics Pick a few reliability metrics and say what value would be a problem.
4Set the boundaries Step, retry, cost and time limits, allowed actions, required approvals and stop conditions.
Done
Deliverable of this module: a Reliability & Observability Scorecard. It lists what you record, the metrics you watch with their thresholds, the operational boundaries and who is told when one is crossed.
Warning
Privacy reminder: use simulated or already-public scenarios. Do not enter company secrets, client personal data or unauthorized sensitive information, and keep sensitive values out of the logs you design.
In Module 8 you will place this scorecard on the Agent Harness & Reliability Canvas, in the Observability, Reliability Metrics and Operational Boundaries sections.
For a minimal observability plan, which is the best starting set to record?
Key takeaway: Define the run, record the minimum needed to explain it, choose metrics with thresholds and set boundaries, and record them in your Scorecard.
