# IL 201 — Evaluation Decision Brief：填寫範例（SYN-CS-001）/ Filled Example (SYN-CS-001)

> 本範例使用合成案例 SYN-CS-001（示例公司「示範服務公司 A」），所有人物、產品與數據均為合成，不代表真實
> 企業、成功案例或實際收益。/ This example uses the synthetic case SYN-CS-001 (the example company "Demo
> Service Company A"); all people, products, and figures are synthetic and do not represent a real
> company, a real success story, or real financial results.

版本 / Version: v1.0（2026-09-26）

## 中文

| 段落 | 內容 |
|---|---|
| 解決方案假設 | 情境：示範服務公司 A 客服知識問答案件，聚焦一般與缺漏資訊兩類。AI 做的事：AI 依知識庫草擬回覆初稿附引用來源，人工核對後送出。預期結果：驗收集上平均全流程時間比 baseline 減少至少 25%，重大錯誤率不高於現況。推翻條件：若重大錯誤率高於現況或處理時間未減少，假設不成立。 |
| 初步使用範圍核准 | 核准人：專案 Sponsor（合成角色）與流程負責人。日期：2026-02-01（合成日期）。範圍：一般、缺漏資訊兩類，不含來源矛盾與權限限制類別。 |
| 評估設計摘要 | 三方比較：現行人工搜尋 vs. 知識庫整理（非 AI 改善）vs. AI 輔助草擬。開發集 24 筆（DEV-001~024），驗收集 16 筆（ACC-001~016），驗收集在原型版本凍結後才使用。 |
| 量測指標 | (1) 逐類別正確率；(2) 全流程分鐘數（含核對與修正）；(3) 人工修改比例；(4) 重大錯誤率。 |
| 停止條件 | 若任一類別重大錯誤率超過 10%，立即停止並列為 Hold。 |
| 原型版本與 run log 摘要 | 凍結版本 v0.3（run_id=RUN-014），模型 demo-llm-v2，設定 prompt_v3/top_k=5，輸入版本 docs_2026-02。 |
| 測試結果 | 一般案例（12 筆）正確率 92%；缺漏資訊案例（4 筆）正確率 75%（樣本數過少，僅供參考）。 |
| 失敗分析 | 1 筆缺漏資訊案例被誤判為可回答並給出臆測答案（重大錯誤，1/4=25%，但樣本僅 4 筆，需擴大樣本後重測）。 |
| 成本 | 工具：估計 NT$4,000／月；人工覆核：估計每月增加 20 小時 × NT$450/小時 = NT$9,000／月；資料準備（一次性）：估計 NT$36,000。 |
| 淨價值估計 | 容量價值估計 NT$63,000／月（估計值）− 人工覆核 NT$9,000／月 − 工具 NT$4,000／月 = NT$50,000／月（估計值），另需攤提一次性資料準備成本 NT$36,000。 |
| 限制與風險 | 缺漏資訊案例樣本僅 4 筆，重大錯誤率估計不穩定；本輪未測試來源矛盾與權限限制類別。 |
| 建議決定 | Iterate（修訂再測）：一般案例表現符合假設，但缺漏資訊案例樣本不足且出現 1 筆重大錯誤，建議擴大該類別樣本至至少 15 筆後重新測試，暫不建議直接 Proceed to Readiness。 |
| 合成練習聲明 | 本範例為合成練習，不代表真實企業專案或實際核准結果。 |

## English

| Section | Content |
|---|---|
| Solution hypothesis | Scenario: Demo Service Company A's customer-service Q&A cases, focused on normal and missing-information categories. AI role: drafts a reply with cited sources; an agent verifies before sending. Expected outcome: end-to-end time on the acceptance set drops at least 25% versus baseline, with a critical-error rate no higher than the current process. Falsification condition: the hypothesis fails if the critical-error rate is higher or handling time does not drop. |
| Initial usage-scope approval | Approver: the project sponsor (synthetic role) and process owner. Date: 2026-02-01 (synthetic date). Scope: normal and missing-information categories only; excludes conflicting-sources and permission-restricted categories. |
| Evaluation design summary | Three-way comparison: current manual search vs. a knowledge-base cleanup (non-AI improvement) vs. AI-assisted drafting. Dev set: 24 cases (DEV-001-024). Acceptance set: 16 cases (ACC-001-016), used only after the prototype version was frozen. |
| Measurement metrics | (1) accuracy per category; (2) end-to-end minutes (including verification and correction); (3) manual-edit ratio; (4) critical-error rate. |
| Stop criteria | If any category's critical-error rate exceeds 10%, stop immediately and record a Hold. |
| Prototype version and run-log summary | Frozen version v0.3 (run_id=RUN-014), model demo-llm-v2, config prompt_v3/top_k=5, input version docs_2026-02. |
| Test results | Normal cases (12): 92% accuracy. Missing-information cases (4): 75% accuracy (small sample, informational only). |
| Failure analysis | 1 missing-information case was wrongly treated as answerable and given a guessed answer (critical error, 1/4 = 25%, but the sample is only 4 cases and needs a larger retest). |
| Cost | Tooling: estimated NT$4,000/month; human review: estimated extra 20 hours/month x NT$450/hour = NT$9,000/month; data preparation (one-time): estimated NT$36,000. |
| Net-value estimate | Capacity-value estimate NT$63,000/month (estimate) - human review NT$9,000/month - tooling NT$4,000/month = NT$50,000/month (estimate), with a one-time NT$36,000 data-preparation cost to amortize separately. |
| Limitations and risk | The missing-information category has only 4 samples, so its critical-error-rate estimate is unstable; the conflicting-sources and permission-restricted categories were not tested this round. |
| Recommended decision | Iterate: normal cases met the hypothesis, but the missing-information category is under-sampled and produced 1 critical error. Recommend expanding that category to at least 15 samples and retesting before recommending Proceed to Readiness. |
| Synthetic-exercise statement | This example is a synthetic exercise and does not represent a real company project or an actual approval outcome. |