Checkpointing, Resume & Durable Execution

8 min

The run stopped halfway. Do we start over?

In Module 2 you described a run as state moving through steps and transitions. Real runs get interrupted: a service restarts, a time limit is reached, a person closes a session. The core question of this module is: if a run stops halfway, how does the system continue safely instead of starting over?

• Restarting from the beginning wastes finished work
• Restarting can also repeat actions that already happened in the outside world
• Continuing safely needs saved state and a clear idea of what is already done
• Where and how to save is a design decision made in the harness

Key takeaway: An interruption is normal. The design question is how the run continues from a known point without losing work or repeating actions.

9 min

The checkpoint model: snapshot, persistence, recovery point

A checkpoint is a saved record of the run at a chosen point, so the run can continue from there later. The saved record is a snapshot of the run state. Saving it somewhere that survives the interruption is called persistence. The point you can return to is the recovery point.

Text equivalent: the run alternates between doing work and saving a snapshot. After an interruption the run can return to the most recent checkpoint instead of the beginning. Checkpoint 2 comes after the external action, so it records that the action already happened.

Key takeaway: A checkpoint saves a snapshot of run state, persisted so that the run has a recovery point to return to.

9 min

Where to checkpoint: not every point is safe

You could save state after every tiny action, but that is costly, and more importantly not every point is a safe checkpoint. A safe checkpoint is a point where returning to it gives a run that is correct and consistent. Five questions help you decide.

Key takeaway: Place a checkpoint where the state is consistent and the result of any external action is confirmed and recorded.

9 min

Resume, restart and the risk of duplicate execution

After an interruption the system can restart or resume. Restart begins again from the first step. Resume continues from a recovery point using the saved state. Resume saves work, but replaying steps can still repeat their side effects.

This module teaches idempotency as a concept for reasoning about replay. If an action is not idempotent, the design should record that it happened, check the record before acting again, or place a human gate in front of it (Module 4).

Key takeaway: Resume saves work, but replay can repeat side effects. Reason about idempotency and record completed actions.

9 min

Durable execution, with an example framework mapping

Durable execution is the ability to survive an interruption and continue from a known valid state. It is a property of the design, not of one product. A durable design has saved state, a clear execution identity, safe checkpoints, and a defined resume behavior.

Text equivalent: while running, the system saves snapshots at safe checkpoints. After an interruption it finds the run by its identity, restores the latest valid snapshot, checks which actions were already recorded, and then continues instead of restarting.

Key takeaway: Durable execution means continuing from a known valid state after interruption. Frameworks can implement it, and the concepts stay framework independent.

11 min

Practice: Where Should We Checkpoint?

Activity 3 is called Where Should We Checkpoint? First work the example, then do it for your own workflow. Please do not enter real company secrets, client personal data or any unauthorized sensitive information in this exercise.

1. 1List the steps
   Write the steps of your workflow in order.
2. 2Place checkpoints
   Mark where to checkpoint, and use the five questions to justify each place.
3. 3Mark unsafe replay points
   Circle steps whose side effects would repeat on replay, and note how to prevent that, for example an action record or a human gate.
4. 4Define state and resume
   List what state must be stored, and write what the run does when it resumes: which snapshot it loads and what it checks first.

• Go to the Practice page: Where Should We Checkpoint?
• Next: Verification, Human-in-the-Loop & Control Gates (Module 4)

Key takeaway: Human-AI-Human: a person decides which actions are safe to replay, the harness saves and resumes from known state, and a person reviews the outcome and stays responsible.