PrerequisiteComplete Module 2 (Evaluation Design), with a draft evaluation plan in hand
0 / 3 completed
Learning objectives
Distill the hypothesis card and evaluation plan into a minimal-scope prototype definition.
Keep a reproducible run log: version, timestamp, model/tool, configuration, input version, and human rater.
Design a human-handoff path so the AI can safely defer to a person when uncertain or out of scope.
From hypothesis to a reproducible prototype
100 min
Define the minimal scope
The prototype exists to produce evaluable evidence, not to be the most feature-complete system possible. A minimal-scope prototype only needs to cover the scenario and case categories named in the Module 1 hypothesis card; everything else waits for a later stage.
Only the categories named in the hypothesis card (e.g. normal, missing information); other categories are marked out of scope
Input source
Use a fixed version of SYN-CS-001's documents/ knowledge files; do not mix in undocumented document versions on the fly
Output format
A draft reply plus cited sources for human verification, not a direct send to the customer
Explicit exclusions
List what this prototype deliberately does not do, to prevent scope creep
Warning
Common mistake: scope creep: gradually adding case types or features the hypothesis card never covered, just to make the demo look better. The evaluation then no longer maps to the hypothesis, and the Evidence Review cannot say clearly what the prototype actually tested.
Key takeaway: The prototype's scope must map one-to-one to the Module 1 hypothesis card; anything extra is not a bonus: it just unfocuses the evaluation.
100 min
A reproducible run log
The Evidence Review needs to answer "under what version and configuration was this result produced?" A result with no run record cannot be double-checked, and cannot be reproduced at the next stage.
Minimum run-log fields
Field
Description
run_id
A unique id for each run
prototype_version
The prototype version (e.g. v0.3), tied to a code or prompt version
run_timestamp
When the run happened
model_or_tool
The model/tool name and version used
config
Key settings (e.g. prompt version, temperature, max retrieved documents)
input_version
The version of the input data (the documents/ knowledge base)
Common mistake: tweaking the prompt or settings several times but only keeping the last run's result, so no one can explain "why did the score go up this time?" Every meaningful change should get its own run_id.
Key takeaway: An evaluation result with no recorded version, configuration, or input cannot survive being questioned in the Evidence Review, and cannot be trusted.
100 min
The human-handoff path
The prototype must clearly define what happens when the AI is uncertain or out of scope, instead of letting it force a plausible-sounding but unsupported answer. The human-handoff path is the Human-AI-Human principle made concrete at the prototype level.
Example human-handoff path (customer-service knowledge Q&A)
The AI searches the knowledge base for relevant documents
No sufficiently supported answer is found -> mark "cannot answer" and attach the list of documents searched but insufficient
Hand off to a customer-service agent, along with the AI's search record so far (to avoid rework)
The agent decides and replies to the customer, and logs this case under the "no answer" category for later analysis
Info
A human handoff is not a failure: it is one of the normal paths through a well-designed prototype. The acceptance set deliberately includes "no answer" and "permission-restricted" cases precisely to verify that this path actually triggers correctly, rather than being papered over by a forced AI answer.
1Exercise Draw the human-handoff path for your prototype, covering at least "insufficient evidence found" and "outside the defined case categories."
2Self-check Confirm you have a complete run log for at least 3-5 runs, and can point to which things the prototype "deliberately does not do" against the hypothesis card.
Check yourself
When the AI prototype cannot find enough evidence to support an answer, it chooses to "force a plausible-sounding reply" instead of handing off to a human. What principle does this mainly violate?
Key takeaway: A prototype that never triggers a human handoff usually means it is forcing an answer even when uncertain: that is a warning sign for the Evidence Review, not a strength.
