PrerequisiteCompleted Module 4 (Solution Sprint), with a monitorable version, error handling, and human-handoff design
0 / 4 completed
Learning objectives
Run AI judgment and the human workflow in parallel under shadow mode, comparing the two without affecting real case outcomes.
Set explicit thresholds for moving from shadow mode to small-scale, human-confirmed use.
Keep a consistent pilot log, and respond correctly to anomalies using the mechanisms designed in Module 4.
From shadow mode to small-scale confirmed use
90 min
Shadow mode: observe before acting
In shadow mode, the AI produces a judgment on real cases without affecting the actual case outcome; the human process runs as usual. The goal of this stage is to observe, at zero risk, how closely AI judgment matches what humans actually do.
What to record in shadow mode
Item
Description
Agreement rate
The proportion of cases where the AI judgment matches the human's final outcome, tallied separately across SYN-CS-001's six case categories
Divergent cases
Cases where AI and human disagree, with the reason for divergence logged case by case (not just the word "disagreed")
High-risk case performance
Whether high-risk categories such as restricted_access or no_answer show especially unstable AI performance
System stability
Whether tool failures or delays occurred during shadow mode, as a technical-risk indicator before advancing to the next stage
Warning
Common mistake: after a few days of shadow mode showing a high agreement rate, rushing to skip small-scale confirmed use and go straight to full rollout. Shadow mode only proves "AI judgment usually matches a human"; it does not prove "the human-confirmation workflow design also works": these are two different things.
Key takeaway: Shadow mode validates AI judgment quality, not the full human-AI collaboration workflow; both must be validated before full rollout.
90 min
Thresholds for moving to small-scale confirmed use
Moving from shadow mode to "AI produces a suggestion, a human confirms it before it takes effect" needs an explicit transition threshold, not a gut feeling that "it seems good enough now."
Example thresholds for shadow mode -> small-scale confirmed use
Condition
Description
Agreement rate meets the target
E.g. normal and outdated_source case types reach the threshold set at kickoff
Zero major divergence on high-risk cases
No out-of-scope suggestions occurred for restricted_access cases during shadow mode
Error-handling mechanism proven to work
The Module 4 human-handoff trigger has actually fired at least once, with response time matching the design
Sponsor and process owner jointly confirm
Both parties confirm in writing that the pilot can advance, not just a unilateral technical-team decision
Info
Small-scale, human-confirmed use still covers only a portion of case volume or staff, letting the AI suggestion genuinely influence the case outcome while keeping final human sign-off; any further expansion should be re-checked against the expansion threshold first.
Key takeaway: The transition threshold must be defined at kickoff or before Module 5 begins: it cannot be improvised as "good enough" once shadow mode finishes.
90 min
Pilot-log discipline
The pilot log is the raw evidence source for Module 6's evaluation and Module 9's executive briefing; incomplete or inconsistent logging undermines the credibility of every analysis that follows.
Required pilot-log fields
Field
Description
Time and version
Execution time and the corresponding version number (whether the architecture or config changed during the pilot)
Task and usage scope
Case type, and whether it is shadow mode or small-scale confirmed use
Result
AI suggestion, the human's final decision, and whether the two agree
Exception and handoff
Whether human handoff was triggered, why, and the response time
Impact and handling
If an error occurred, its actual impact and how it was subsequently handled
1Exercise: fill in five pilot-log entries (using SYN-CS-001) Simulate five case records for the small-scale confirmed-use stage.
2Cases 1-2: agreeing cases Log two normal-category cases where the AI suggestion and human confirmation agree, complete with full fields.
3Case 3: a divergent case Log one conflicting_sources case where AI and human disagree, explaining the reason for divergence.
4Case 4: a triggered handoff Log one missing_context case that triggered handoff, with the response time attached.
5Case 5: an anomaly event Log one system-delay event, explaining its actual impact on case handling and the subsequent response.
Key takeaway: The quality of the pilot log determines the credibility of the evaluation and briefing that follow; anything not logged effectively never happened.
90 min
Anomaly-response drills during the pilot
The controlled pilot should include at least one drill of how the team actually responds when a Module 1 stop trigger genuinely fires, rather than attempting it for the first time when it really happens.
Example anomaly-response drill scenarios (reusing SYN-CS-001)
Scenario
Response focus
A signal that an AI suggestion may be leaking another customer's information
Immediately assess against the Module 1 stop triggers whether to call a stop, and record the assessment process
Knowledge-base sources are found to be widely outdated and un-updated
Pause auto-replies, switch to full human review until sources are updated
Small-scale trial users report suggestion quality has dropped collectively
Trigger the Module 4 error taxonomy for a full re-check, and pause any further expansion of the trial
Info
The drill record itself is part of the evidence Module 7's governance review needs: stop authority is not a clause on paper, it is a mechanism confirmed workable through at least one real or simulated drill.
Check yourself
After two weeks of shadow mode showing a high agreement rate between AI judgment and human outcomes, the team decides to skip small-scale, human-confirmed use and go straight to full rollout. What is the problem with this approach?
Key takeaway: Stop authority that has never been drilled tends to cause hesitation when a real risk event occurs; drill records are important evidence for governance review.
