Reliability, Failure Handling, and AI System Evaluation

12 min

Judge pass/fail with evidence, not impression

Hallucination refers to AI producing content that sounds plausible but is actually incorrect or ungrounded. Grounding means an AI output is backed by real, checkable data or sources; Verification is confirming (manually or systematically) that an output is correct. Before a government AI system goes live, it cannot rely on "it looked fine in the demo", it needs a structured test model.

Key takeaway: The core test model turns "is this system reliable" from a subjective impression into a series of concrete, checkable steps.

12 min

Error taxonomy and test coverage

• Error Taxonomy: breaking Unexpected Behavior down further, e.g. data error, tool-call failure, misunderstanding, permission error, to help target improvement.
• Evaluation dataset: a collection of cases used for systematic testing, it should cover a range of situations, not just cases that are easy to pass.
• Edge cases: hard-to-judge situations such as ambiguous content, content that fits multiple categories, or incomplete information.
• Adversarial / misuse cases (conceptual level): deliberate attempts to get the system to do something it should not, e.g. asking it to leak another person's data or bypass review.

Key takeaway: A good evaluation dataset must deliberately include edge cases and adversarial scenarios, not just the typical cases the system already handles well.

12 min

Failure handling and observability

• Retry: automatically retry once or a few times on a transient error.
• Fallback: when AI cannot reliably complete a task, fall back to a conservative behavior or hand off to a human, instead of forcing an output.
• Failure Classification: categorize errors (e.g. data error, tool-call failure, misunderstanding) to track and improve them.
• Recovery: the concrete way a system returns from a failed state to normal operation, e.g. clearing a bad intermediate result and re-running.
• Observability: keep enough logs to reconstruct what happened after the fact.
• Regression Testing: after an update, re-run existing test cases to make sure previously correct cases have not broken.

Key takeaway: Retry, Fallback, Logging, Observability, and Regression Testing together form the full answer to "what happens after the system fails"; missing any one leaves a blind spot.

12 min

Three test cases: a government complaint-classification example

• Every case must state both Expected Behavior and Forbidden Behavior: "what it should do" matters just as much as "what it must never do."
• The failure/restricted case maps directly to the Prompt Injection risk from Module 5, showing that governance design and test design must echo each other.

Key takeaway: The three test-case types each check whether the system has a clear, safe expected behavior under "normal," "uncertain," and "maliciously prompted" conditions.

12 min

Practice: Test the AI, Not the Demo

Activity 6 asks you to write at least three test cases for your own use case and fill in the Government AI Test Case record, one of the key sources of evidence for the Module 8 acceptance decision.

1. 1Pick a use case
   Reuse a scenario from an earlier module.
2. 2Write at least three test cases
   At least one each for normal, edge/ambiguous, and failure/restricted.
3. 3Define Expected Behavior and Forbidden Behavior
   State clearly what it should do and what it must never do.
4. 4Record evidence and pass/fail
   Every case needs checkable evidence, not just an impression.

• Government AI Test Case minimum schema: Test ID, Requirement ID, Scenario, Input, Expected Behavior, Forbidden Behavior, Expected Source / Evidence, Permission Expectation, Human Gate, Actual Result, Evidence, Pass / Fail, Failure Type, Remediation, Regression Test?.

• Go to the practice page: Activity 6 Test the AI, Not the Demo

Key takeaway: This test-case record becomes core evidence for the Module 8 acceptance decision, not just a Module 6 homework exercise.