# IL 301: Pilot Report：填寫範例（SYN-CS-001）/ Filled Example (SYN-CS-001)

版本 / Version: v1.0（2026-09-26）

> **合成教學資料聲明 / Synthetic teaching data disclaimer**：本範例使用 IL 系列共用合成案例
> `SYN-CS-001`（示範服務公司 A 客服知識助理），所有組織、人物、對話與數字均為合成教學資料，
> 不代表任何真實企業或真實成效。/ This example uses the shared IL-series synthetic case
> `SYN-CS-001` (Demo Service Company A's customer-service knowledge assistant). All
> organizations, people, conversations, and figures are synthetic teaching material and do not
> represent any real company or real outcome.
>
> 本範例的第 6 節（評估報告）刻意保留一處常見錯誤（忽略干擾因素），並在其後附上修訂說明，
> 對應課程模組 6 的完整範例。/ Section 6 (Evaluation Report) below deliberately preserves one
> common error (ignoring confounders), followed by correction notes, matching the worked example
> in course Module 6.

## 中文

### 1. Kickoff 摘要

| 欄位 | 內容 |
|---|---|
| 核准版本 | IL 202 Readiness Review：Proceed with Conditions（2026-02-20），IL 201 Evidence Review：proceed_to_readiness（2026-02-10） |
| RACI | Sponsor：客服處處長（合成角色）；流程負責人：客服營運經理；技術負責人：AI 平台工程師；治理審查者：資安暨隱私辦公室；使用者代表：資深客服代表 |
| 停止權與觸發條件 | 客服處處長持有停止權；觸發條件：（1）連續 3 件個資外洩風險事件（2）系統延遲超過 5 分鐘且無法在 1 小時內恢復 |
| Pilot 期間 | 2026-03-01 至 2026-04-25（8 週）；檢查點：影子模式結束（3/14）、小範圍試行中期（4/04）、Pilot 結束評估（4/25） |
| 資料／環境等級 | Level B（去識別化資料），部分案例含 Level C 受控客戶帳號資料 |

### 2. 流程與使用者訪查

| 欄位 | 內容 |
|---|---|
| 現況流程重新驗證發現 | 相較 IL 101 記錄，客服系統已於 2026-01 升級，新增「風險案件自動標記」步驟，處理時間略增 0.5 分鐘／件 |
| 工作難度差異切分維度 | 案件類型（六類）、客服資歷（<3 個月 vs. >1 年）、尖峰／離峰 |
| 不願使用 AI 的原因 | 新進客服：技能型（不確定如何核對 AI 引用來源）；資深客服：流程型（覺得多一道核對步驟）；曾被客訴同仁：信任型（擔心建議出錯自己要負責） |

### 3. 技術與資料審查

| 欄位 | 內容 |
|---|---|
| 有效權限重新驗證結果 | 5 個帳號重新測試，1 個離職帳號已停用但尚未從群組移除，已補正 |
| 資料範圍再核對結果 | 發現技術團隊為除錯另存一份含客戶姓名的試算表，已刪除並改用去識別化資料重現問題 |
| IL 202 附條件事項結案驗證 | API key 撤權延遲：重新演練確認 40 秒內生效，已結案；隱私準備度個資標註：已補做標註，資料流向圖已更新 |

### 4. 架構與資料／治理控管

| 欄位 | 內容 |
|---|---|
| 架構草稿 | 輸入 → 知識庫查詢（KD-001~KD-022）→ AI 判斷 → 信心分流（>=0.8 自動回覆草稿／<0.8 人工接手）→ 人工核可 → 監測記錄 |
| 監測記錄的最小資訊集 | 執行版本、時間、案件類別、信心分數、是否觸發接手、人工修改幅度（不記錄完整對話明文） |
| 錯誤處理分類與策略 | 依 SYN-CS-001 六類案件對應：missing_context／conflicting_sources 一律轉人工；restricted_access 直接拒絕並記錄 |
| 人工接手觸發條件與介面 | 信心 <0.8、案件屬 conflicting_sources 或 restricted_access 時觸發；介面顯示 AI 判斷理由、引用文件、信心分數 |

### 5. Pilot Log 摘要

| 欄位 | 內容 |
|---|---|
| 影子模式一致率 | normal 96%、outdated_source 90%、missing_context 78%、conflicting_sources 70%、no_answer 85%、restricted_access 100%（零重大分歧） |
| 轉換門檻與確認人 | 高風險類別零重大分歧、接手機制實測有效；由客服處處長與客服營運經理於 3/16 書面確認進入小範圍試行 |
| 代表性案例 | （1）normal 案件一致；（2）conflicting_sources 案件觸發人工判斷，AI 正確標示矛盾；（3）missing_context 案件觸發接手，客服代表 12 分鐘內回應 |
| 異常應變演練 | 3/20 模擬「知識庫來源大量過期」情境，暫停自動回覆並全面轉人工，耗時 8 分鐘完成切換 |

### 6. 評估報告（含刻意保留的錯誤）

**草稿版本（含錯誤）：**

| 指標 | Pilot 期間 | 基準期間 | 結論 |
|---|---|---|---|
| 平均全流程處理時間 | 4.2 分鐘／件 | 7.0 分鐘／件 | AI 輔助讓處理時間縮短 40%，效率大幅提升，品質同步提升，沒有取捨問題 |
| 整體正確率 | 92% | 88% | 品質提升 |
| 案件量 | 1,200 件／週 | 900 件／週 | （草稿未提及） |
| 人員配置 | 2 位資深客服請假，3 位新進支援 | 穩定 | （草稿未提及） |

> **⚠️ 找出問題**：這份草稿完全沒有揭露案件量增加 33%、且有資深人力請假由新進人員支援這兩項重大干擾因素，就直接把處理時間縮短歸因於 AI。

**修訂版本（修正說明）：**

- 揭露干擾因素：Pilot 期間案件量比基準期間高 33%，且有 2 位資深客服請假由新進人員支援。
- 重新歸因：控制案件量與人員配置差異後（比較相同週間、相同人員配置的子集），AI 輔助對處理時間縮短的實際貢獻需要更謹慎估計，初步估計部分縮短來自案件量增加下的流程壓縮。
- 品質與時間取捨：正確率從 88% 提升到 92%，但需進一步確認是否有覆核步驟被省略，而非直接宣稱「同步提升，沒有取捨」。
- 建議下一步：延長 Pilot 期間或補做人員配置相近的子樣本比較，取得更可信的歸因結論。

### 7. 失敗分析

| 欄位 | 內容 |
|---|---|
| 主要錯誤類型與案例 | conflicting_sources 案件中，有 2 件 AI 誤選較舊文件版本（KD-014 而非 KD-027） |
| 根因分析 | 知識庫檢索未優先採用 `contradicts_doc_id` 標註的最新版本 |
| 修正措施與再測結果 | 修正檢索邏輯優先採用未過期且無 `contradicts_doc_id` 指向的文件，再測 10 件同類案件全數正確 |

### 8. 人機與治理審查

| 欄位 | 內容 |
|---|---|
| 停用／復原／撤權演練結果 | 停用：47 秒生效；復原：2 分鐘，確認無資料遺失；撤權：立即生效，測試一個離職帳號確認無法再存取 |
| 例外處理紀錄 | 尖峰時段暫時放寬信心門檻至 0.75，範圍限當日 14:00-16:00，由治理審查者核准，期限當日到期，事後複核未發現超出範圍 |
| 治理結論 | 三項演練通過，例外處理有紀錄可循，判定具備進入 Adoption Gate 討論的治理基礎 |

### 9. 導入重設計

| 欄位 | 內容 |
|---|---|
| 交接與支援套件 | 作業指引、支援模式、復原手冊、教育訓練材料已備齊；已知限制清單已列出 restricted_access 與高度矛盾案件仍需人工優先處理 |
| 殘餘風險登錄 | （1）知識庫持續產生新矛盾，仍需人工定期複核；（2）新進客服長期依賴 AI 建議的技能流失風險，需持續觀察 |

### 10. Sponsor 評估與導入建議

| 段落 | 內容 |
|---|---|
| Adoption Gate 建議 | LIMITED ADOPTION（先維持目前小範圍試行規模，暫不全面擴大） |
| 建議理由 | （1）conflicting_sources 類案件一致率仍偏低（70%）需持續優化；（2）評估報告干擾因素尚未完全排除，需要更謹慎的歸因；（3）治理演練已通過，具備繼續小範圍運作的基礎 |
| 成本與後續責任 | 客服營運經理持續負責日常維運預算；治理審查者每季複核殘餘風險；下一次治理審查訂於 2026-07 |
| Sponsor 意見 | 客服處處長同意 LIMITED ADOPTION 建議，要求下次審查前補做人員配置控制後的評估子樣本 |
| 已知限制 | 本次 Pilot 僅涵蓋去識別化與部分受控帳號資料（Level B/C），未涵蓋完整正式環境資料 |
| 合成聲明 | 本範例為合成練習，不等同正式專案，僅供教學示範 |

## English

> **Synthetic teaching data disclaimer**: this example uses the shared IL-series synthetic case
> `SYN-CS-001` (Demo Service Company A's customer-service knowledge assistant). All organizations,
> people, conversations, and figures are synthetic teaching material and do not represent any real
> company or real outcome. Section 6 (Evaluation Report) below deliberately preserves one common
> error (ignoring confounders), followed by correction notes, matching the worked example in
> course Module 6.

### 1. Kickoff summary

| Field | Content |
|---|---|
| Approved version | IL 202 Readiness Review: Proceed with Conditions (2026-02-20); IL 201 Evidence Review: proceed_to_readiness (2026-02-10) |
| RACI | Sponsor: Head of Customer Service (synthetic role); Process owner: Customer Service Operations Manager; Technical owner: AI Platform Engineer; Governance reviewer: Security & Privacy Office; User representative: Senior Customer Service Rep |
| Stop authority and triggers | The Head of Customer Service holds stop authority; triggers: (1) 3 consecutive incidents with personal-data leakage risk (2) system delay over 5 minutes that cannot be resolved within 1 hour |
| Pilot period | 2026-03-01 to 2026-04-25 (8 weeks); checkpoints: end of shadow mode (3/14), mid-point of small-scale trial (4/04), pilot close-out evaluation (4/25) |
| Data/environment level | Level B (de-identified data), with some cases involving Level C controlled customer-account data |

### 2. Process & user immersion

| Field | Content |
|---|---|
| Current-process re-verification findings | Compared to the IL 101 record, the customer-service system was upgraded in 2026-01, adding a "risk-case auto-flag" step, slightly increasing processing time by 0.5 min/case |
| Work-difficulty variance dimensions | Case type (six categories), staff tenure (<3 months vs. >1 year), peak vs. off-peak |
| Reasons for reluctance to use AI | New hires: skill type (unsure how to cross-check AI-cited sources); veterans: process type (feel it adds an extra check step); a previously complained-about staffer: trust type (worried about being blamed if the suggestion is wrong) |

### 3. Technical & data review

| Field | Content |
|---|---|
| Effective-permission re-verification result | 5 accounts re-tested; 1 departed-employee account was disabled but not yet removed from the group: corrected |
| Data-scope re-check result | Found a debugging spreadsheet with customer names copied outside approved scope; deleted and replaced with de-identified data for reproduction |
| IL 202 conditional-item closure verification | API key revocation delay: re-drilled, confirmed to take effect within 40 seconds, closed; privacy-readiness personal-data tagging: tagging completed, data flow map updated |

### 4. Architecture and data/governance controls

| Field | Content |
|---|---|
| Architecture draft | Input -> knowledge-base lookup (KD-001~KD-022) -> AI judgment -> confidence routing (>=0.8 auto-reply draft / <0.8 human handoff) -> human sign-off -> monitoring log |
| Minimum monitoring information set | Execution version, time, case category, confidence score, whether handoff was triggered, magnitude of human edits (no full plaintext conversation logged) |
| Error-handling taxonomy and strategy | Mapped to SYN-CS-001's six case categories: missing_context / conflicting_sources always hand off to a human; restricted_access is refused outright and logged |
| Human-handoff triggers and interface | Triggered when confidence <0.8, or the case is conflicting_sources or restricted_access; the interface shows the AI's reasoning, cited documents, and confidence score |

### 5. Pilot log summary

| Field | Content |
|---|---|
| Shadow-mode agreement rate | normal 96%, outdated_source 90%, missing_context 78%, conflicting_sources 70%, no_answer 85%, restricted_access 100% (zero major divergence) |
| Transition threshold and approver | Zero major divergence on high-risk categories and a proven handoff mechanism; the Head of Customer Service and the Operations Manager jointly confirmed in writing on 3/16 to enter small-scale trial |
| Representative log entries | (1) an agreeing normal case; (2) a conflicting_sources case that triggered human judgment, with the AI correctly flagging the conflict; (3) a missing_context case that triggered handoff, with the rep responding within 12 minutes |
| Anomaly-response drill | On 3/20, simulated "knowledge-base sources widely outdated," pausing auto-replies and switching fully to human review; the switch took 8 minutes |

### 6. Evaluation report (with a deliberately preserved error)

**Draft version (with the error):**

| Metric | Pilot period | Baseline period | Conclusion |
|---|---|---|---|
| Average full-workflow processing time | 4.2 min/case | 7.0 min/case | AI assistance cut processing time by 40%, a major efficiency gain, with quality improving in lockstep and no tradeoff |
| Overall accuracy | 92% | 88% | Quality improved |
| Case volume | 1,200/week | 900/week | (Not mentioned in the draft) |
| Staffing | 2 senior staff on leave, covered by 3 new hires | Stable | (Not mentioned in the draft) |

> **Find the problem**: this draft never discloses that case volume rose 33% and that senior staff on leave were covered by new hires: two major confounders: before attributing the drop in processing time directly to AI.

**Revised version (correction notes):**

- Disclose confounders: pilot-period case volume was 33% higher than baseline, and 2 senior staff were on leave, covered by new hires.
- Re-attribute: after controlling for case-volume and staffing differences (comparing a matched subset of weeks and staffing), AI assistance's actual contribution to the time reduction needs a more cautious estimate; a preliminary estimate suggests part of the reduction comes from process compression under higher case volume.
- Quality-time tradeoff: accuracy rose from 88% to 92%, but whether any review step was skipped needs further confirmation, rather than directly claiming "improved together with no tradeoff."
- Recommended next step: extend the pilot period, or run an additional comparison on a matched subsample with similar staffing, for a more credible attribution.

### 7. Failure analysis

| Field | Content |
|---|---|
| Main error types and cases | In 2 conflicting_sources cases, the AI mistakenly selected an older document version (KD-014 instead of KD-027) |
| Root-cause analysis | The knowledge-base retrieval did not prioritize the newer version flagged via `contradicts_doc_id` |
| Corrective actions and retest results | Fixed retrieval logic to prioritize non-expired documents with no `contradicts_doc_id` pointing away from them; retested 10 similar cases, all correct |

### 8. Human-AI governance review

| Field | Content |
|---|---|
| Disable/recovery/revocation drill results | Disable: took effect in 47 seconds; recovery: 2 minutes, confirmed no data loss; revocation: took effect immediately, tested on a departed employee's account, confirmed no further access |
| Exception-handling record | Confidence threshold temporarily loosened to 0.75 during a peak period, scoped to 14:00-16:00 that day, approved by the governance reviewer, expiring the same day; after-the-fact review found no scope violation |
| Governance conclusion | All three drills passed, and the exception was properly recorded: judged to have the governance foundation needed to enter the Adoption Gate discussion |

### 9. Adoption redesign

| Field | Content |
|---|---|
| Handoff and support package | Operating instructions, support model, recovery runbook, and training material are complete; the known-limitation list flags that restricted_access and highly conflicting cases still need human-first handling |
| Residual-risk register | (1) the knowledge base keeps generating new conflicts, still requiring periodic human review; (2) long-term skill-erosion risk from new hires over-relying on AI suggestions, requiring continued observation |

### 10. Sponsor assessment and adoption recommendation

| Section | Content |
|---|---|
| Adoption Gate recommendation | LIMITED ADOPTION (maintain the current small-scale trial for now, do not expand fully yet) |
| Rationale | (1) the agreement rate for conflicting_sources cases is still relatively low (70%) and needs continued improvement; (2) confounders in the evaluation report have not been fully ruled out, requiring a more cautious attribution; (3) governance drills have passed, providing a foundation to continue small-scale operation |
| Cost and ongoing responsibility | The Operations Manager continues to own the day-to-day operating budget; the governance reviewer reviews residual risk quarterly; the next governance review is set for 2026-07 |
| Sponsor comments | The Head of Customer Service agrees with the LIMITED ADOPTION recommendation and requests a staffing-controlled evaluation subsample before the next review |
| Known limitations | This pilot covered only de-identified and some controlled account data (Level B/C), not full production data |
| Synthetic disclaimer | This example is a synthetic exercise, not equivalent to a formal project, for teaching demonstration only |