PrerequisiteCompleted Module 5 (Controlled Pilot), with a complete pilot log
0 / 3 completed
Learning objectives
Compare actual pilot-period results against the baseline re-documented in Module 2, not a stale baseline from earlier stages.
Identify and disclose confounders that could bias interpretation: workload changes, shifts in case-difficulty mix, and staffing turnover.
Read an evaluation report example with a deliberate error, and find how ignoring confounders can mislead the conclusion.
Compare against baseline and disclose confounders
60 min
Compare against baseline, reporting quality, time, and cost separately
The evaluation's comparison baseline is the current-process record re-verified in Module 2 before the pilot began, not the earlier, possibly stale, baseline log from IL 101 or IL 201. Quality, time, and cost must be reported as three separate metrics: a single blended score cannot be used to mask any one of them getting worse.
Three independent dimensions of the evaluation report
Dimension
How it is measured
Common mistake
Quality
Accuracy and review-edit magnitude, tallied separately across SYN-CS-001's six case categories
Reporting only the overall average accuracy, masking poor performance in a high-risk category like restricted_access
Time
Full end-to-end time, including input preparation, AI processing, human review and correction, and rework
Counting only the time the AI takes to produce a suggestion, excluding subsequent review and correction time
Cost
Including tool fees, review labor, operations, and exception-handling cost
Counting only tool licensing fees, ignoring the added review and operations labor cost
Warning
Common mistake: if the "AI-assisted full-workflow time" only counts the seconds it takes the AI to produce a suggestion, without the subsequent waiting, review, correction, and rework, it will seriously underestimate the actual time invested.
Key takeaway: The evaluation report must be able to answer three independent questions: did quality improve, did time improve, did cost improve: never blur them into a single composite score.
60 min
Disclose confounders
During a pilot, AI assistance is often not "the only variable that changed": workload, the mix of case difficulty, and staffing can all shift within the same period. Without disclosing these confounders, an observed improvement or decline may come entirely from other causes yet get mistakenly credited to the AI.
Three common categories of confounders
Confounder
Question to check
Workload change
Did case volume during the pilot rise or fall notably compared to the baseline period (e.g. a peak driven by a promotion)?
Case-difficulty mix
Was the pilot-period case-type mix skewed more toward simple cases than the baseline period?
Staffing
Did a senior staff member leave, a new hire join, or temporary staff get reassigned during the pilot?
Info
Disclosing confounders is not meant to deny the pilot's value: it lets the sponsor and governance reviewer judge how much of an observed improvement can genuinely be attributed to AI assistance. This is exactly why IL 301's evaluation must be more rigorous than IL 201's: IL 201 compares within a controlled experimental setting, while IL 301 evaluates within a real, constantly changing operational environment.
Key takeaway: An evaluation report that never discloses confounders: however nice the numbers look: gives no way to judge how much is genuinely the AI's contribution.
A full example: find the error from ignoring confounders
60 min
A full example (SYN-CS-001): find the deliberately preserved error
Below is an excerpt of a draft evaluation report for "Demo Service Company A"'s customer-service knowledge assistant (SYN-CS-001), which deliberately preserves one common error. Try to find the problem yourself before reading the correction notes below.
Draft evaluation report excerpt (with a deliberately preserved error)
Metric
Pilot-period result
Baseline-period result
Reported conclusion
Average full-workflow processing time
4.2 min/case
7.0 min/case
Conclusion: AI assistance cut processing time by 40%, a major efficiency gain
Overall accuracy
92%
88%
Conclusion: quality improved in lockstep, no tradeoff
Case volume
1,200/week during the pilot
900/week during baseline
(Not mentioned in the report)
Staffing
2 senior staff on long leave during the pilot, covered by 3 new hires
Stable staffing during baseline
(Not mentioned in the report)
Warning
Found the problem? The report never discloses that pilot-period case volume rose 33% and that senior staff were on leave, covered by new hires: two major confounders: before attributing the drop in processing time directly to AI assistance. Higher case volume can pressure staff into working faster and skipping some quality-check steps; new-hire coverage can also change how cases get assigned. Either could partly explain the faster processing time, and neither was disclosed at all.
Erroneous conclusion
Conclusion: AI assistance cut processing time by 40%, a major efficiency gain, with quality improving in lockstep and no tradeoff.
Revised conclusion
Disclose confounders
Pilot-period case volume was 33% higher than baseline, and 2 senior staff were on leave, covered by new hires: both factors could affect processing time and quality simultaneously, and their influence cannot be fully ruled out.
Re-attribute
After controlling for the difference in case volume and staffing (e.g. comparing only a subset of matched weeks and staffing), AI assistance's actual contribution to processing time needs a more cautious estimate; a preliminary estimate suggests at least part of the time reduction comes from process compression under higher case volume, not entirely from AI.
The quality-time tradeoff
Accuracy rose from 88% to 92% while processing time also fell: this needs further confirmation on whether any review step was skipped, rather than directly claiming "quality and time improved together with no tradeoff."
Recommended next step
Extend the pilot period, or run an additional comparison on a matched subsample with similar staffing and case volume, to reach a more credible attribution.
Info
This correction is exactly the course's required "one fully-worked example with at least one error and correction notes." When practicing, check your own project's evaluation report for the same blind spot: ignoring changes in workload, case-difficulty mix, or staffing.
The example evaluation report claims "AI assistance cut processing time by 40%, a major efficiency gain." What is the main problem with this?
Key takeaway: The most easily overlooked evaluation error often hides inside a report whose numbers look great; an undisclosed confounder is more dangerous than a disclosed one that cannot be fully ruled out.
