PrerequisiteComplete Module 1 (Problem and Hypothesis)
0 / 3 completed
Learning objectives
Design a three-way comparison of the current process, a feasible non-AI improvement, and the AI-assisted approach.
Split a development set and an acceptance set with stable ids, and define exclusion rules.
Choose case categories appropriate to the project, and define measurement and stop criteria.
Three pillars of evaluation design
60 min
Three-way comparison: current process, non-AI improvement, AI-assisted
The evaluation method must include a three-way comparison of the current process, a feasible non-AI improvement (for example refreshing the knowledge base, adding staff, or simplifying a form), and the AI-assisted approach. If one arm does not apply, the evaluation plan must say why explicitly; it cannot simply be left out.
Example three-way comparison design (reusing SYN-CS-001)
Arm
Description
Expected metric
Current process (baseline)
The agent manually searches the knowledge base and drafts a reply
End-to-end minutes from baseline_log.csv
Non-AI improvement
Clean up and refresh expired knowledge-base documents, add keyword indexing
Re-test the same case batch; expect reduced search time
AI-assisted
AI drafts a reply with cited sources; the agent verifies before sending
End-to-end minutes and critical-error rate on the same case batch
Warning
Common mistake: comparing only "current process" against "AI-assisted" and skipping the non-AI improvement arm. If a simple knowledge-base cleanup captures most of the benefit, the sponsor needs to know that; it must not be hidden behind the AI arm's headline numbers.
Key takeaway: Without a non-AI improvement baseline, you cannot answer the question every sponsor will ask: "did this benefit actually require AI?"
60 min
Dev set vs. acceptance set: stable ids and exclusion rules
The development set is used to tune the prototype; the acceptance set is used for the final judgment. The acceptance set must never be used to tune the prototype, or the evaluation result will be systematically overstated. Both sets need stable ids, a clear purpose, and exclusion rules.
Dev/acceptance set fields (reusing SYN-CS-001's schema.md)
Field
Description
case_id
A stable id (e.g. DEV-001, ACC-001); never reuse an id for a different case
case_category
Normal, missing information, conflicting sources, expired sources, no answer, permission-restricted (adjust to the project)
split
dev or acceptance; once assigned, never changed after the fact to fit a result
exclusion_reason
If a case is excluded (e.g. corrupted data), record why: never quietly drop it
Warning
Common mistake: "borrowing" a few acceptance-set cases to debug the prototype during development, then using the same acceptance set to compute the final score. That is training and grading on the same data, and the score will be inflated; once acceptance cases have been looked at, they must be replaced or explicitly relabeled as dev cases.
Info
SYN-CS-001's cases_practice.csv provides 40 synthetic cases across six categories, ready to split into a practice dev/acceptance set (a synthetic exercise does not represent a real project's required acceptance sample size).
Key takeaway: The acceptance set exists to make the final call; once it has been used to tune the prototype, it has lost its purpose.
60 min
Measurement and stop criteria
Evaluations must report task quality, end-to-end time, manual-edit volume, and cost separately, never as a single blended score. Critical errors (for example, confidently stating wrong information) must never be hidden behind an overall average, and insufficient sample sizes must be disclosed explicitly.
Task quality: report accuracy/quality per case category, not averaged across categories in a way that hides a weak spot.
End-to-end time: includes AI generation, human verification, correction, and rework: not just the AI's response seconds.
Manual-edit volume: the proportion and extent of edits a human needs to make, reflecting real usability.
Critical errors: list high-risk error categories and cases separately, even if few in number: never dilute them into an average.
Info
If LLM-as-judge scoring is used, it can only be a supporting signal: it needs a human-calibration sample, and the plan must record how disagreements between human and LLM scores are resolved (for example, human judgment prevails).
Warning
Stop criteria must be defined before testing begins, for example "if the critical-error rate exceeds 10%, stop immediately and record a Hold": not decided on the fly once disappointing results appear.
1Exercise For your own project, write: the three-way comparison design, dev/acceptance split rules, at least three measurement metrics, and one stop criterion: assemble these into a draft evaluation plan.
Check yourself
While debugging the prototype, an engineer used a few acceptance-set cases to tune the prompt, then used the same acceptance set to compute the final score. What is the problem?
Key takeaway: Stop criteria must be defined before you see any results, or they will be too easily shaped by the results themselves.
