# IL 201 — Evaluation Decision Brief Rubric：行為錨點 / Behavior Anchors

> 本文件只補充現行八個評分面向的「行為錨點」說明，權重沿用課程頁「評分項目與權重」既有設定，不更動
> 任何計分邏輯或權重。/ This document only adds behavior anchors for the eight existing scoring
> dimensions; the weights are unchanged from the course page's existing "Scoring Criteria and Weights,"
> and no scoring logic or weight is modified here.

共同四級量尺 / Common 4-level scale: 0 = 無法判定 / cannot be judged,
1 = 待改善 / needs improvement, 2 = 合格 / adequate, 3 = 優良 / strong.

## 中文

| 面向（權重） | 0 | 1 | 2 | 3 |
|---|---|---|---|---|
| 問題一致性（10%） | 假設與 IL 101 Charter 的問題陳述明顯脫節 | 大致相關，但情境或使用者已悄悄改變 | 假設與 Charter 一致，並清楚交代任何範圍調整 | 同左，並說明調整如何取得 Sponsor 重新確認 |
| 解決方案設計（15%） | 沒有可測試的假設 | 假設存在但無法被推翻（見 Module 1） | 假設具體到情境、AI 角色、預期結果、推翻條件四項齊全 | 同左，並能說明為何這個設計是目前最小可行版本 |
| Prototype 品質（15%） | 沒有可執行的原型或無法重現 | 原型可執行，但缺乏版本或設定紀錄 | 有完整 run log（版本／時間／模型工具／設定／輸入版本／人工判定者，見 Module 3） | 同左，並有明確的人工接手路徑且經實際觸發測試 |
| 評估嚴謹度（20%） | 沒有三方比較或開發／驗收集切分 | 有切分但驗收集曾被用於調整原型 | 三方比較完整，開發／驗收集獨立且有停止條件（見 Module 2） | 同左，並對 LLM-as-judge（若使用）有人工校準說明 |
| 商業價值證據（15%） | 沒有成本或價值估計 | 只計算工具成本，未含人工覆核／維運／資料準備 | 五類成本齊全，一次性與經常性分開列示並標記估計／實測（見 Module 5） | 同左，並揭露重複計算風險與淨價值的敏感度 |
| 負責任 AI／風險（10%） | 未提及重大錯誤或風險 | 有提及但整體平均掩蓋重大錯誤 | 逐類別報告品質，重大錯誤單獨列出（見 Module 4） | 同左，並說明樣本不足類別的後續補測計畫 |
| 人類監督（5%） | 沒有人工接手或最終判定機制 | 有提及但未落實在原型或流程中 | 人工接手路徑具體且已測試（見 Module 3） | 同左，並定期檢視人工接手是否被恰當觸發 |
| Sponsor 審查（10%） | 沒有模擬或實際 Sponsor 審查紀錄 | 有審查但只有籠統回饋 | 審查涵蓋假設、失敗案例與成本的具體追問與回答（見 Module 6） | 同左，並將審查意見轉化為 decision brief 的具體修訂 |

## English

| Dimension (weight) | 0 | 1 | 2 | 3 |
|---|---|---|---|---|
| Problem fidelity (10%) | The hypothesis clearly diverges from the IL 101 charter's problem statement | Broadly related, but the scenario or users have quietly changed | The hypothesis is consistent with the charter, with any scope change clearly explained | As left, and explains how the change was re-confirmed with the sponsor |
| Solution design (15%) | No testable hypothesis | A hypothesis exists but is not falsifiable (see Module 1) | The hypothesis is specific: scenario, AI role, expected outcome, and falsification condition are all present | As left, and explains why this is the current minimal-viable design |
| Prototype quality (15%) | No runnable prototype, or it cannot be reproduced | The prototype runs, but lacks version or configuration records | A complete run log exists (version/timestamp/model-tool/config/input version/human rater, see Module 3) | As left, with a clear human-handoff path that was actually triggered and tested |
| Evaluation rigor (20%) | No three-way comparison or dev/acceptance split | A split exists, but the acceptance set was used to tune the prototype | The three-way comparison is complete, the dev/acceptance sets are independent, and stop criteria exist (see Module 2) | As left, with a human-calibration note for LLM-as-judge (if used) |
| Business-value evidence (15%) | No cost or value estimate | Only tooling cost is calculated, missing human review/operations/data preparation | All five cost categories are present, one-time vs. recurring are listed separately and tagged estimate/verified (see Module 5) | As left, disclosing double-counting risk and the net value's sensitivity |
| Responsible AI/risk (10%) | No mention of critical errors or risk | Mentioned, but critical errors are hidden inside an overall average | Quality is reported per category, with critical errors listed separately (see Module 4) | As left, with a follow-up retest plan for under-sampled categories |
| Human oversight (5%) | No human-handoff or final-judgment mechanism | Mentioned but not implemented in the prototype or process | The human-handoff path is concrete and has been tested (see Module 3) | As left, with periodic review of whether the handoff triggers appropriately |
| Sponsor review (10%) | No mock or real sponsor-review record | A review happened but only produced vague feedback | The review covers concrete follow-up questions and answers about the hypothesis, failure cases, and cost (see Module 6) | As left, with review feedback turned into specific revisions in the decision brief |