From Notebook to Repeatable Analytics Workflow

40 min

What Is a Data Product / Analytics Workflow?

Learning objective: define users, inputs, outputs, SLA/freshness, quality expectations, ownership.

Every month you manually run a notebook that produces a report your manager reads. Last month you were out, and a colleague had to take over with no idea which source file to use, where the output belongs, how often it should refresh, or who to call when something breaks. This analysis was never actually turned into a product, it only lived in your head and on your laptop. The first step toward a data product is not writing code; it is writing down the tacit knowledge: who uses this output, how fresh they expect it to be, what quality counts as trustworthy, and who is accountable for its correctness.

import pandas as pd

# clean: monthly analytics package, one row is a metric observation for a source system, batch, and entity
df = pd.read_csv("DS301_Analytics_Package_Clean.csv", parse_dates=["event_time", "ingested_at"])
df.shape
df.dtypes
df[["run_id", "source_system", "batch_id", "metric", "value", "schema_version"]].head()

# Ask first: who owns this data? If schema_version changes, who needs to be told?
df["schema_version"].unique()

• Only you know where to pull the raw data from, and the whole process lives in your head
• Nobody knows how often the output should refresh, and nobody checks whether it actually does
• "Is this number right?" can only be answered by the original author's word, with no verification record
• Running it on a different machine or by a different person fails outright because of paths or credentials

Key takeaway: Completing the steps is not enough; the result must be verifiable, reproducible, and explainable.

45 min

Notebook → Functions → Stages

Learning objective: refactor exploratory code into idempotent stages with explicit inputs/outputs.

An exploratory notebook is often one long chain of cells executed top to bottom: loading, cleaning, filtering, aggregating, and plotting are all tangled together, with variables depending on whatever earlier cells left in memory. Change cell 3 and you have to rerun everything after it. That style is fine for exploration, but it does not survive becoming a reliable workflow: nobody can rerun just the "cleaning" step without touching everything else, and no single step can be unit-tested on its own. Turning exploratory code into functions and stages is the difference between "one long spell" and "a set of blocks you can inspect and rerun individually."

# Exploration: a long chain of interdependent cells (hard to rerun, hard to test)
df = pd.read_csv("DS301_Analytics_Package_Clean.csv")
df = df[df["schema_version"] == "v2"]
df["value"] = df["value"].fillna(0)
summary = df.groupby(["source_system", "metric"])["value"].sum()

# Refactored into stages: each function has a clear input/output and can be tested or rerun on its own
def load_raw(path: str) -> pd.DataFrame:
    return pd.read_csv(path, parse_dates=["event_time", "ingested_at"])

def filter_current_schema(df: pd.DataFrame, version: str) -> pd.DataFrame:
    return df[df["schema_version"] == version].copy()

def fill_missing_values(df: pd.DataFrame) -> pd.DataFrame:
    df["value"] = df["value"].fillna(0)
    return df

def summarize_by_metric(df: pd.DataFrame) -> pd.DataFrame:
    return df.groupby(["source_system", "metric"], as_index=False)["value"].sum()

1. 1Find the natural breakpoints
   Locate where loading, cleaning, filtering, and aggregating naturally divide in the exploratory notebook.
2. 2Write one function per stage
   Give each stage an explicit input type and output type, and avoid relying on global variables.
3. 3Remove hidden dependencies
   Confirm no function depends on a variable left over in memory from an earlier cell.
4. 4Add tests for critical functions
   Write at least one test for an error-prone stage, such as cleaning or aggregation.

Key takeaway: Completing the steps is not enough; the result must be verifiable, reproducible, and explainable.

40 min

Configuration, Secrets, and Environments

Learning objective: separate config from code; never hardcode secrets; define dev/test/prod concepts.

A colleague hardcodes a database password into the first cell of a notebook, for convenience while developing. Three months later that notebook is shared with another team for a training demo, and the password leaks along with it. An even more common problem: the same code runs fine on your laptop but fails outright on another machine or against a different database, because paths, credentials, and table names are all baked into the code. Pulling configuration and secrets out of the code is the precondition for the same code running safely across dev, test, and prod.

# Wrong: credentials and environment settings hardcoded into the code
DB_PASSWORD = "sup3r-secret-2026"  # never do this
INPUT_PATH = "/Users/alice/Desktop/data.csv"

# Right: read from environment variables / a secret manager; the code itself carries no secrets
import os

DB_PASSWORD = os.environ["DS301_DB_PASSWORD"]  # injected by the secret manager, never committed to version control
ENVIRONMENT = os.environ.get("DS301_ENV", "dev")  # dev / test / prod
INPUT_PATH = os.environ["DS301_INPUT_PATH"]  # points at a different data source per environment

1. 1Find every hardcoded value
   Check the code for passwords, paths, and table names written directly into it.
2. 2Move them into environment variables or config
   Read these values from the environment or a config file instead, so the code itself carries no secrets.
3. 3Separate dev/test/prod settings
   Make sure the same code runs correctly across environments through different environment configurations.
4. 4Confirm secrets are never committed
   Check version control settings to ensure secret files can never be accidentally committed.

Key takeaway: Completing the steps is not enough; the result must be verifiable, reproducible, and explainable.

45 min

Data Contracts and Pipeline Validation

Learning objective: fail early on schema/quality violations; distinguish warning from blocking errors.

An upstream system silently misspells one of the values in the metric column, from "conversion_rate" to "converion_rate," without telling anyone. Your pipeline never checks this and feeds the data straight into the aggregation logic; that metric quietly gets bucketed into "other," the monthly report undercounts an entire product line’s conversion rate, and nothing ever raises an error. A data contract writes down what the data is supposed to look like, and fails loudly and early when it does not, instead of letting bad data silently flow downstream.

ALLOWED_METRICS = {"conversion_rate", "active_users", "avg_response_ms"}
REQUIRED_COLUMNS = {"run_id", "source_system", "batch_id", "event_time", "ingested_at", "entity_id", "metric", "value", "schema_version"}

def validate_contract(df: pd.DataFrame) -> list[str]:
    errors = []
    missing = REQUIRED_COLUMNS - set(df.columns)
    if missing:
        errors.append(f"missing required columns (blocking): {sorted(missing)}")

    unknown_metrics = set(df["metric"].unique()) - ALLOWED_METRICS
    if unknown_metrics:
        errors.append(f"unknown metric values found (blocking): {sorted(unknown_metrics)}")

    null_ratio = df["value"].isna().mean()
    if null_ratio > 0.2:
        errors.append(f"value null ratio too high (warning, needs human review): {null_ratio:.1%}")

    return errors

errors = validate_contract(df)
blocking = [e for e in errors if "blocking" in e]
if blocking:
    raise ValueError(f"Pipeline failed early on contract violation: {blocking}")

• The list of required columns and each column’s expected data type
• The list of allowed categorical values (for example, which values metric may take)
• Acceptable ranges and the maximum tolerable null ratio
• Whether a given violation should be treated as a warning or a blocking error
• A notification process and versioning rule for when the schema changes

Key takeaway: Completing the steps is not enough; the result must be verifiable, reproducible, and explainable.

70 min

Guided Lab: Refactor a Notebook into a Pipeline

Learning objective: with instructor guidance, build ingest→validate→transform→analyze→publish stages with checkpoints.

• Go to the DS 301 Practice page
• PreviewDownload the clean analytics package (CSV, synthetic data)
• Download the Guided Lab starter notebook (notebook → pipeline refactor)
• PreviewDownload the data dictionary (Traditional Chinese)
• PreviewDownload the data dictionary (English)

1. 1Ingest
   Isolate the logic that reads raw data into its own stage that only loads data, with no cleaning or computation.
2. 2Validate
   Apply the data contract check before data enters later processing, failing early on blocking violations.
3. 3Transform
   Isolate cleaning, filtering, and type conversion into their own stage, keeping each function’s input/output explicit.
4. 4Analyze
   Isolate aggregation and computation into their own stage, and add a checkpoint (such as a row count or a total) to confirm the result is reasonable.
5. 5Publish
   Isolate writing the output to its final location into its own stage, and log the config and environment used for this run.

• Copying notebook cells directly into functions, but forgetting to remove their dependency on global variables
• Adding a validate stage that exists only in name, without actually applying any contract checks
• Hardcoding config or secrets into the pipeline code instead of reading them from the environment
• Adding no checkpoints at all, and judging success only by whether the pipeline errored, not whether the intermediate results are reasonable

Key takeaway: Completing the steps is not enough; the result must be verifiable, reproducible, and explainable.