AI 302 · Module 5

Retry, Failure Classification & Recovery

When an agent fails, should the system retry, try another way, hand over to a human, or stop?

Start Section 1

Duration60 min
LevelArchitect
PrerequisiteAI 301 recommended, plus Modules 1 to 4 of this course

0 / 7 completed

Sync my progressJoin a session to keep progress across devices and share it with your instructor

Learning objectives

Distinguish failure, error and exception, and classify a failure using the eight-type Failure Taxonomy
Explain timeout, retry, retry policy, retry limit and the backoff concept
Recognize when retry is unsafe, such as after a payment, a destructive action, an external notification or an irreversible transaction
Use the Recovery Decision Model to choose retry, fallback, re-plan, compensation, escalation or safe stop
Complete Activity 5 and produce a Failure & Recovery Policy

Failures and their types

9 min

When an agent fails, what should the system do?

In Module 4 you designed gates that check work before it moves on. Now a different situation: a step did not just produce a weak result, it failed. A reliable harness does not hope failures are rare. It decides in advance how each kind of failure is handled.

Core question

When an agent fails, should the system retry, try another method, hand over to a human, or stop?

Key terms
Term	Plain meaning
Failure	A step did not achieve what it was supposed to achieve
Error	A signal that something went wrong, such as an error code or message
Exception	An unexpected condition that interrupts normal execution
Recovery	What the system does after a failure to get back to a safe, useful state

Simulated example

A simulated report agent calls a data tool, gets no answer, then receives a table with missing columns, then drafts a summary that contradicts the table. Three different failures, and each one needs a different response.

Info

This module teaches concepts that apply to any agent framework. A framework can implement them, but your design comes first.

Key takeaway: Failures are expected. Decide in advance how each kind is handled, instead of improvising during a run.

9 min

Failure taxonomy: not all failures are alike

The right response depends on the type of failure. A failure taxonomy gives a team a shared vocabulary so that the recovery choice follows from the type.

Failure taxonomy

Transient: a brief problem that may pass on its own
Tool / Network: a tool or connection failed or timed out
Missing Data: required input is absent or incomplete
Invalid Output: the result is malformed or breaks the expected format
Logical / Planning: the plan or reasoning itself is wrong
Permission: the agent is not allowed to do this
Safety / Policy: the action would break a safety rule or policy
Human Rejection: a person reviewed the work and declined it

Info

Text version of the diagram: a single node called Failure branches into eight types. Transient, Tool / Network, Missing Data, Invalid Output, Logical / Planning, Permission, Safety / Policy and Human Rejection. Each type sits at the same level, so each can lead to a different recovery path.

Failure types with simulated examples
Type	Simulated example
Transient	A service is briefly busy and answers a moment later
Tool / Network	The data tool does not respond before the time limit
Missing Data	The input file has no date column
Invalid Output	The agent returns text where a table was required
Logical / Planning	The plan orders two steps that depend on each other the wrong way round
Permission	The agent tries to write to a folder it may only read
Safety / Policy	The draft would include personal data that policy forbids
Human Rejection	The reviewer declines the draft

Check yourself

An agent tries to write to a location it is only allowed to read. Which failure type is this?

Permission, so retrying the same action will not helpTransient, so waiting and retrying is the answerMissing Data, so the agent should fetch more data

Key takeaway: Name the failure type first. The type, not the error message alone, points to the right recovery.

Timeout, retry and its limits

9 min

Timeout, retry policy and the backoff concept

Some failures are worth trying again. A timeout stops a step from waiting forever. A retry tries the step once more. A retry policy is the written rule for when, how often and how long to retry, so that retrying is a design decision and not a reflex.

Retry concepts
Concept	Plain meaning
Timeout	The longest a step may take before it counts as failed
Retry	Running the same step again after a failure
Retry limit	The maximum number of attempts before giving up
Backoff (concept)	Waiting a little longer between attempts, so a struggling service is not flooded
Retry policy	The rule that combines which failures may be retried, the limit and the waiting pattern

Simulated example

Policy for a read-only data lookup: retry only on Transient or Tool / Network failures, at most 3 attempts, waiting a little longer each time. After the third failure, stop retrying and move to the recovery decision.

Warning

Without a retry limit, a failing step can loop forever, waste effort and hide the real problem. Always set a limit and decide what happens when it is reached.

Check yourself

Why does a retry policy need a retry limit?

To stop endless attempts and force a decision about what happens nextBecause every failure disappears after three attemptsBecause retries are always harmless, so a limit is optional

Key takeaway: A retry policy names which failures may be retried, how many times and with what waiting pattern.

9 min

Retry is not always safe

Retrying a read is usually harmless. Retrying an action that changes the world can repeat the change. If the first attempt actually succeeded but the confirmation was lost, a retry does the thing twice.

Do not retry blindly after
Action	What a blind retry could cause
Payment submission	The same payment is made twice
Destructive file action	A second deletion removes something else, or fails in a confusing way
External notification	The recipient gets duplicate messages
Irreversible transaction	A change that cannot be undone is repeated

Info

Link to Module 3: an idempotent step gives the same result when run twice, so retrying it is safe. For steps that are not idempotent, first check the current state, or use an idempotency key, or ask a human, before any retry.

Simulated example

A simulated agent sends a notice to a customer list and the call times out. Did the messages go out? Unknown. The safe move is to check what was sent, not to send again.

Check yourself

A call that submits a payment times out, and it is unclear whether it went through. What is the safest next step?

Check the real state, and involve a human if it is still unclear, before any retryRetry immediately so the payment is not lostRetry three times, since three is the usual limit

Key takeaway: Retry only what is safe to repeat. After payments, destructive actions, notifications or irreversible changes, verify state first.

Recovery, escalation, practice and deliverable

8 min

Recovery decision model: fallback, re-plan and compensation

When retrying is not allowed or has run out, the system follows a decision model. It asks three questions in order.

Recovery decision model

Failure occurs
Retryable? If yes, apply the Retry Policy
If no, is there an Alternative Path? If yes, use Fallback or Re-plan
If no, can a Human resolve it? If yes, Escalate
If no, Stop Safely

Info

Text version of the diagram: from a failure, first ask whether it is retryable. If yes, follow the retry policy. If not, ask whether an alternative path exists. If yes, use a fallback or re-plan. If not, ask whether a human can resolve it. If yes, escalate. If not, stop safely.

Recovery concepts
Concept	Plain meaning
Fallback	A simpler or safer alternative method for the same goal
Re-plan	Change the plan itself because the original approach cannot work
Compensation	An action that offsets an earlier completed step, such as issuing a correction when the step cannot be undone

Info

Example only, not required: a stateful framework such as LangGraph can express recovery as state-aware resume or an alternate transition. The concepts above apply to any framework.

Check yourself

A retry limit has been reached and a simpler backup method exists. What does the decision model say?

Use the fallback or re-plan before escalating or stoppingStop immediately without looking for alternativesKeep retrying past the limit

Key takeaway: Ask in order: retry, alternative path, human, safe stop. Compensation offsets a completed step that cannot be undone.

8 min

Escalation and safe stop

Escalation hands the problem to a person who can decide. Safe stop ends the run in a state that is clear and does no further harm. Both are correct outcomes, not embarrassments. A system that can stop safely is more trustworthy than one that always presses on.

Escalation versus safe stop
Outcome	When it fits	What it must include
Escalate	A human can resolve the issue, such as approve a permission or supply missing data	What failed, what was tried, the current state, and a clear question for the person
Safe stop	No alternative exists and no human can help now, or continuing risks harm	A stable end state, a record of where it stopped, and no half-finished changes left hidden

Simulated example

A simulated agent hits a Safety / Policy failure: the draft would contain personal data. There is no fallback that keeps that data. The system stops the step, saves the state and tells the owner why, instead of trying to work around the rule.

Warning

Human, AI, Human: for a Safety / Policy failure or a Human Rejection, the agent must not retry around the decision. A person defined the rule or made the call, and a person stays responsible.

Check yourself

A reviewer rejects an agent output. What should the harness do?

Treat it as a decision to respect, then re-plan with the feedback or stop, not silently retryRetry until the reviewer acceptsIgnore the rejection and continue

Key takeaway: Escalate with a clear summary, or stop safely with a clear state. Neither is a failure of design.

8 min

Activity 5: Retry, Recover, Escalate or Stop? and the Failure & Recovery Policy

Take a simulated agentic workflow, or the one you designed in earlier modules. List the ways it can fail, classify each failure, then design the response.

List failures
Write 6 to 8 things that could go wrong in your workflow.
Classify each one
Pick a type from the Failure Taxonomy for every item.
Design the retry
For retryable failures, set the retry limit and waiting pattern. Mark steps that are not safe to retry.
Add fallback and escalation
Name a fallback path where one exists, and who is asked when a human must resolve it.
Define the stop condition
Say when the run stops safely and what state it must leave behind.

Warning

Privacy reminder: use simulated or already-public information. Do not enter company secrets, client personal data or unauthorized sensitive information.

Info

Deliverable: a Failure & Recovery Policy. It lists each failure with its type, retry policy, fallback, escalation and stop condition. Keep it for your Agent Harness & Reliability Canvas.

Go to the AI 302 practice: In the "Retry, Recover, Escalate or Stop?" activity, classify failures and design the response.
Continue to Module 6: Next: how does an agent keep working when its context keeps growing?

Check yourself

Your policy row is for a step that sends an email to customers. What must the row say about retry?

Not safe to retry blindly: check what was sent first, or ask a humanAlways retry up to the limitRetry forever until it works

Key takeaway: Classify each failure, set retry limits only where safe, add fallback and escalation, and define a safe stop.

Save as PDF

Retry, Failure Classification & Recovery

Learning objectives

Failures and their types

When an agent fails, what should the system do?

Failure taxonomy: not all failures are alike

Timeout, retry and its limits

Timeout, retry policy and the backoff concept

Retry is not always safe

Recovery, escalation, practice and deliverable

Recovery decision model: fallback, re-plan and compensation

Escalation and safe stop

Activity 5: Retry, Recover, Escalate or Stop? and the Failure & Recovery Policy

What next?