Data Products & Analytics Workflow Engineering
Turn one-off notebooks into repeatable, observable analytics workflows and data products with validation, recovery, and human approval points: start by defining users, inputs/outputs, and ownership, refactor exploratory code into explicit functions and stages, separate configuration and secrets from code, use data contracts to fail a pipeline early, refactor a notebook into a pipeline in a Guided Lab, design safe scheduling and incremental runs, add logging, metrics, lineage, and observability, place human approval and rollback before publishing, add tests and failure recovery with a runbook, and deliver a reliable Analytics Workflow Canvas, an executable prototype, and a runbook in an Independent Lab.
The course storyline: from a one-off analysis to a reliable data product
The whole course follows one learning loop: Input Contract, Transform, Validate, Publish, Approve, Observe, Recover. AI may help write pipeline code; a person stays responsible for defining the contract, reviewing the publish, and owning the result.
- 1. Input Contract
What is this pipeline’s input data contract and what do its users expect?
- 2. Transform
Has the exploratory code been split into stages with explicit inputs/outputs?
- 3. Validate
Does the pipeline fail early when the data violates the contract?
- 4. Publish
Is the write logic idempotent, and does publishing keep a rollback-able prior version?
- 5. Approve
Is there a human approval point before a high-impact publish?
- 6. Observe
Are run IDs, row counts, freshness, and lineage being logged?
- 7. Recover
When a failure happens, can the runbook let someone safely diagnose and recover?
Your next step
DS 301 · Module 1: From Notebook to Repeatable Analytics Workflow
Curriculum
- Module 1From Notebook to Repeatable Analytics WorkflowStart by defining a data product’s users, inputs/outputs, and owner, refactor exploratory code into explicit functions and stages, separate configuration and secrets from code, use data contracts to fail a pipeline early, then refactor a notebook into a pipeline in a Guided Lab.
- Module 2Reliable Data Products: Validation, Delivery, Observability, and RecoveryLearn safe scheduling and incremental runs, full logging, metrics, lineage, and observability, human approval and rollback before publishing, and testing, failure recovery, and runbooks, then deliver a reliable analytics workflow in an Independent Lab.
Progress is stored in this browser and does not identify you. It is uploaded only if you join progress sync below.
Sync my progressJoin a session to keep progress across devices and share it with your instructor
Practice and assessment
- Practice A: Notebook RefactorRefactor exploratory code into functions and stages with explicit inputs/outputs.Not done yet
- Practice B: Data Contract FailureDefine a data contract and demonstrate how it fails a pipeline early.Not done yet
- Practice C: Idempotent RerunCheck and fix write logic so reruns are safe and never double-count.Not done yet
- Practice D: Observability ChecklistAdd run IDs, row counts, and validation logging to a pipeline.Not done yet
- Practice E: Rollback ScenarioDesign a versioned output and drill a rollback after a bad publish.Not done yet
- Practice F: Runbook DrillWrite and test a runbook for common failure scenarios.Not done yet
- DS 301 pre-test and post-testPre- and post-tests check how your judgment changes on data products, functions, secrets, contracts, idempotency, incremental runs, logging, approval, rollback, testing, retries, and runbooks.Not done yet
- Analytics Workflow Canvas + Executable Workflow Prototype + RunbookTurn what you learned into your own Analytics Workflow Canvas + Executable Workflow Prototype + Runbook, stating the problem, data, method, verification, and governance judgment.Not done yet
About this course
Outcomes
- Define users, inputs, outputs, SLA/freshness, quality expectations, ownership
- Refactor exploratory code into idempotent stages with explicit inputs/outputs
- Separate config from code; never hardcode secrets; define dev/test/prod concepts
- Fail early on schema/quality violations; distinguish warning from blocking errors
- Build ingest→validate→transform→analyze→publish stages with checkpoints
- Complete the final artifact: Analytics Workflow Canvas + Executable Workflow Prototype + Runbook
Prerequisites
- DS 202 recommended; AI 104 also recommended
- DS 101–202 level Python/pandas skills expected
Who it is for
- NTPU students, faculty and administrative staff who work with structured data
- Learners who completed DS 202 and want to turn one-off analysis into a repeatable, reliable workflow
- Analysts and data practitioners who need validation, delivery, observability, and recovery skills
Certificate
Earns a course completion certificate (L1); a capability certificate (L2) is also available through additional artifact review.
Requirements to earn it
- Complete all 2 course modules
- Pass the post-test assessment at 80% or higher
- Submit a project artifact and pass review
Course feedback
It takes 2 to 3 minutes and is used to improve the course.