Monthly analytics package (clean)
A monthly analytics dataset: run_id, source_system, batch_id, event_time, ingested_at, entity_id, metric, value, schema_version. Used for the Guided Lab.
DS 301 Practice Center
Use the synthetic monthly analytics package to practice refactoring a notebook into a pipeline, handling a data contract failure, designing an idempotent rerun, checking observability, drilling a rollback, and writing a runbook.
Info
DS 301 practice data is synthetic teaching data. Every source system, batch, and value is illustrative and does not represent any real organization or individual.
Warning
Only use the synthetic files provided by the course for practice. Unless authorized, never paste confidential organizational data, personal data, or sensitive information into an AI tool or practice environment; check your organization’s policy before using real operational data.
Download the practice files
Monthly analytics package (practice, with challenges)
Same field structure, with deliberately designed challenges: late-arriving events, one schema change, a duplicated batch_id, and at least one row that violates the contract. Used for the Practice Center and the Independent Lab.
Data dictionary (Traditional Chinese)
Field types, units, grain, allowed values, and provenance for the analytics package (Traditional Chinese).
Data dictionary (English)
English data dictionary; fields and definitions match the Chinese version.
Guided Lab starter notebook (notebook → pipeline refactor)
A starter notebook with guidance and tasks to complete, practicing splitting exploratory code into ingest/validate/transform/analyze/publish stages.
Independent Lab starter notebook / template
Starter notebook and checklist template for the Analytics Workflow Canvas + Executable Workflow Prototype + Runbook.
Practice tasks
A: Notebook Refactor
Refactor a piece of exploratory notebook code into functions and stages with explicit inputs/outputs.
Files used: Monthly analytics package (practice, with challenges), Data dictionary (Traditional Chinese)
Steps
- Find the natural breakpoints where loading, cleaning, filtering, and aggregating occur in the current code.
- Write one function per breakpoint, with an explicit input type and output type.
- Confirm no function depends on a variable left over in memory from an earlier cell.
- Write one simple test for at least one error-prone function, such as cleaning or aggregation.
What you produce
- A list of the resulting functions and their inputs/outputs
- At least one test case
- A demonstration of rerunning one stage without affecting the others
B: Data Contract Failure
Define a data contract for the practice data, and demonstrate how it makes the pipeline fail early when the data violates it.
Files used: Monthly analytics package (practice, with challenges), Data dictionary (Traditional Chinese)
Steps
- List the required columns, the allowed metric values, and the acceptable null ratio.
- Write a validate function that distinguishes blocking errors from warnings.
- Run the validation against the practice data and find at least one row that violates the contract.
- Explain how this bad row would have affected downstream results without this validation layer.
What you produce
- A complete data contract definition
- A validation function and its decision logic
- At least one caught blocking error and an explanation of its impact
C: Idempotent Rerun
Check whether a piece of write logic is idempotent, and fix it into a safely rerunnable version.
Files used: Monthly analytics package (practice, with challenges), Guided Lab starter notebook (notebook → pipeline refactor)
Steps
- Find the current output-writing logic and determine whether it appends or overwrites/replaces.
- Rerun the same input batch twice and observe whether the output table’s row count or total changes.
- Rewrite the write logic to be idempotent: delete the old result for the same range first, then write the new one.
- Rerun twice again and confirm the output no longer depends on the number of reruns.
What you produce
- A before/after comparison of the write logic
- A comparison of output row counts/totals across two reruns
- An explanation of how you confirmed the write logic is now idempotent
D: Observability Checklist
Add the necessary observability logging to a pipeline, and check whether the logs are enough to diagnose a problem.
Files used: Monthly analytics package (practice, with challenges), Independent Lab starter notebook / template
Steps
- Add a run_id to the pipeline and confirm every log line from the same run can be traced together.
- Log row counts before and after each stage.
- Log validation results and the data’s freshness (what time period it covers).
- Simulate an abnormal data volume (for example, deliberately filtering out most of the data) and check whether your logging would let someone catch it.
What you produce
- A sample log including run_id, row counts, validation results, and freshness
- A record of detecting one simulated anomaly
- A note on what observability items you think are still missing
E: Rollback Scenario
Design a versioned output mechanism and drill what happens when a newly published result turns out to be wrong.
Files used: Monthly analytics package (practice, with challenges), Guided Lab starter notebook (notebook → pipeline refactor)
Steps
- Design a mechanism that preserves the prior version of the output (for example, a versioned file or table snapshot).
- Publish a new version of an aggregated result.
- Assume the new version is found to have a serious error, and drill the concrete steps to switch back to the prior version.
- Record how long the rollback drill took and which steps needed human confirmation.
What you produce
- A description of the versioned-output design
- The concrete steps of the rollback drill
- Any process gap discovered during the drill
F: Runbook Drill
Write a runbook for at least two common pipeline failure scenarios, and test it with a simulated scenario.
Files used: Monthly analytics package (practice, with challenges), Independent Lab starter notebook / template
Steps
- List at least two possible failure scenarios (for example, a temporary upstream API timeout, a permanent data-format error).
- Write down how to diagnose, respond to, and escalate each scenario.
- Have a classmate or colleague handle one scenario using only your written runbook, with no extra explanation.
- Record any confusion or gaps they hit while following the runbook, and revise it.
What you produce
- A runbook for at least two failure scenarios
- A record of one simulated test
- A revised runbook based on the test results
The done mark is stored in this browser only (and synced once you join a cohort). Keep your work in your own files or notebook.