PrerequisiteCompleted Module 3 (Technical & Data Review), with a technical and data re-verification checklist
0 / 4 completed
Learning objectives
Design monitoring that lets the team observe how the pilot is actually running, without over-collecting sensitive data.
Sort common errors into an explicit error-handling strategy rather than a single blanket "retry" mechanism.
Design clear human-handoff trigger conditions and flow so uncertain or out-of-scope cases are safely transferred.
From prototype to a monitorable, hand-off-ready pilot version
90 min
Build a monitorable version
IL 201's prototype only needs to prove a value hypothesis; IL 301's pilot version must also be observable: when something goes wrong, the team needs to know what happened, how big the impact is, and whether to invoke stop authority. The challenge in monitoring design is balancing "recording enough information" against "not over-collecting sensitive data."
The minimum information set monitoring design must record
Item
Description
Execution version and time
The code/config version and timestamp for each run
Input summary (without full personal data)
Metadata such as input type and length, avoiding logging full sensitive content directly
Confidence and decision
Whether the AI judged itself confident enough to proceed, or triggered a human handoff
Result and human review
The final output, whether a human edited it, and how much
Exceptions and error codes
Mapped to the error-type table in Module 4
Warning
Common mistake: storing the entire input and output as plaintext logs for debugging convenience. Monitoring and debugging needs should use de-identification or masking and stay within the data scope re-verified in Module 3: monitoring convenience is not a reason to widen data collection.
Key takeaway: Monitoring exists so the team can see anomalies before they escalate, not so they can dig through logs after the pilot ends; what gets recorded is still bound by the data scope.
90 min
Error-handling taxonomy
"The AI made a mistake" is not a description you can act on: it has to be classified first before a matching handling strategy can be designed. A single retry mechanism cannot possibly fit every error type.
Error types and matching handling strategies
Type
Example (SYN-CS-001)
Handling strategy
Tool/system failure
A knowledge-base lookup times out or the service is down
Retry once; if it still fails, hand off to a human: never silently skip it
Missing data
A missing_context case where too little information is available to answer
Explicitly flag insufficient information, attach whatever partial information is known, and hand off to a human
Conflicting data
A conflicting_sources case where multiple documents contradict each other
Do not pick a side; list the points of conflict and hand off for human judgment
Low-confidence output
The AI's self-assessed confidence is below threshold
Do not send it directly; attach the confidence score and hand off for human sign-off
Out-of-scope request
A user requests data or an action outside the approved scope
Refuse and log it; never widen scope just because the user insists
Info
These five error types roughly map to SYN-CS-001's six case categories (normal, missing_context, conflicting_sources, outdated_source, no_answer, restricted_access), so the case categories can be reused directly as the basis for the error-handling taxonomy.
Key takeaway: Unclassified error handling usually ends up as either "always retry" or "always let it through": neither is safe.
90 min
Design the human-handoff mechanism
Human handoff is not the vague idea of "if the AI can't do it, throw it to a human": it needs explicit trigger conditions, a clear interface design, and a trackable response time, or handoff often becomes a black hole nobody actually handles.
Required elements of a human-handoff design
Item
Description
Trigger conditions
For each error type in the Module 4 taxonomy, state exactly when handoff is triggered
Handoff interface
What the human sees must include the AI's reasoning, confidence, and cited sources: not just the words "needs human review"
Response time
The expected response time for each handoff type (e.g. an out-of-scope request needs an immediate response, a missing-data case can join the regular queue)
Ownership
Who makes the final call after handoff, and who it escalates to next if it still cannot be resolved
An inadequate handoff design
The system shows "This case cannot be handled automatically, please assist manually," with no context on the AI's reasoning or urgency.
An adequate handoff design
Reasoning
Shows "This is a conflicting_sources case: documents KB-014 and KB-027 disagree on the return-period policy."
Confidence and known partial information
Attaches an AI-prepared summary of both documents' key points, so the human does not have to start from scratch.
Urgency
Flags whether this case is high-risk (e.g. involves money or personal data), determining queue priority.
Ownership and timing
Assigns the on-shift senior staffer to respond within 30 minutes, auto-escalating to a supervisor if overdue.
Key takeaway: A good human-handoff design lets the human understand "why me," instead of just showing an alert they cannot act on.
90 min
Sprint integration exercise
1Exercise: assemble a pilot architecture and governance-controls draft (using SYN-CS-001) Integrate the first three slides into two concrete deliverables.
2Deliverable 1: architecture draft Draw (or describe as a bulleted list) the pilot version's data flow: input -> knowledge-base lookup -> AI judgment -> confidence routing -> human handoff or auto-reply -> monitoring log.
3Deliverable 2: data/governance controls list List the minimum monitoring information this architecture will collect, its data-scope approval basis (linking to the Module 3 re-verification checklist), and the retention period.
4Self-check Confirm every error type (tool failure, missing data, conflicting data, low confidence, out-of-scope request) maps to a clear human-handoff trigger.
Why is a human-handoff interface that only shows "This case cannot be handled automatically, please assist manually" an inadequate design?
Key takeaway: The output of a solution sprint is not "a working demo": it is an architecture and data/governance-controls document a governance reviewer can actually examine.
