Frame a Prediction Problem and Build a First Model

35 min

Prediction Problems: Target, Features, Prediction Time

Learning goal: define prediction unit, target, horizon, decision, and information available at prediction time.

Your manager says: "Let's predict next week's case volume for each service site." That sounds like a clear task, but a lot is still undefined: what does one "prediction" correspond to: one site, one day, or one shift? Is the target the raw case count, or a binary flag for exceeding capacity? When does "next week" actually start? Most importantly, at the moment you make the prediction, which fields do you really already know, and which only appear afterward?

Skipping this step and feeding data straight into a model has a familiar consequence: a team uses a column that is only produced after a case closes, the model looks great on historical data, but the moment it goes live the numbers do not match at all, because that column simply does not exist yet at prediction time. Writing down the prediction unit, the target, and the prediction time up front avoids this kind of after-the-fact, costly surprise.

import pandas as pd

# clean: a service-demand prediction dataset, one row per prediction case
df = pd.read_csv("DS201_Service_Demand_Clean.csv", parse_dates=["created_at", "split_time"])
df.shape
df.dtypes
df[["case_id", "split_time", "target_numeric", "target_binary"]].head()

# Ask first: which features_* columns are actually known before split_time?
[c for c in df.columns if c.startswith("features_")]

• Treating 'prediction' as the same thing as 'analysis,' without first defining a clear prediction time
• Judging whether a column can be used purely from its name, instead of actually checking when it is produced
• Using the target's own historical values directly as a feature without checking whether it leaks the answer at prediction time
• Different people defining 'one prediction' inconsistently, which makes later evaluation results impossible to compare

Key takeaway: Completing the steps is not enough; the result must be verifiable, reproducible, and explainable.

35 min

Baselines: What Must the Model Beat?

Learning goal: use naive/majority/mean or rule-based baselines before ML.

A colleague announces excitedly: "Our model is 82% accurate!" But if simply guessing "every site's case volume equals the overall average" already reaches 78%, the model's real gain is only 4 points, not the 82% it sounds like. Without a baseline, no score can be judged as worth the cost of building and maintaining a model.

Which baseline to pick depends on the data's own characteristics: data with a clear time pattern suits a naive baseline; data with an uneven class or value distribution often makes a majority or mean baseline already hard to beat. Spending ten minutes computing a baseline up front is far cheaper than discovering later that a model added no real value at all.

import numpy as np

# Regression baseline: use the training mean case volume as every prediction
mean_baseline = df["target_numeric"].mean()
baseline_mae = (df["target_numeric"] - mean_baseline).abs().mean()

# Classification baseline: majority class (always predict "never exceeds threshold")
majority_class = df["target_binary"].mode()[0]
baseline_accuracy = (df["target_binary"] == majority_class).mean()

print(round(baseline_mae, 2), round(baseline_accuracy, 3))

• Reporting only the model's score without also reporting the matching baseline score
• Computing baseline statistics from the training set, then comparing them against a model score measured on the test set
• Looking only at accuracy for a classification problem without first checking the majority class share
• Forgetting to recompute the baseline after a data refresh, leaving the comparison point stale

Key takeaway: Completing the steps is not enough; the result must be verifiable, reproducible, and explainable.

45 min

Train / Validation / Test and Leakage

Learning goal: separate fitting/tuning/final evaluation; prevent future/target leakage.

A colleague keeps tweaking model parameters on the same dataset, checking the test-set score every time and keeping whichever setting scores highest. The problem: the test set has now been used to make decisions, so it is no longer "unseen data." The final reported score will be systematically inflated, and real-world performance after deployment usually disappoints.

Leakage is not always a coding mistake; often the problem is in how a column itself is defined: for example, a column computed using a statistic over the whole dataset (like a global mean) that already bakes in information from the test set. The basic rule for avoiding this kind of hidden leakage is to split the data first, and only compute any statistic afterward.

df_sorted = df.sort_values("split_time")
n = len(df_sorted)
train = df_sorted.iloc[: int(n * 0.6)]
valid = df_sorted.iloc[int(n * 0.6): int(n * 0.8)]
test = df_sorted.iloc[int(n * 0.8):]

# Check: is train's latest time earlier than test's earliest time?
train["split_time"].max() < test["split_time"].min()

1. 1Sort by time before splitting
   Service-demand data has a time order: train must be entirely earlier than valid, and valid entirely earlier than test: never shuffled randomly.
2. 2Check for leakage sources
   Confirm no column is known only after the fact (for example, a field produced after a case closes), and that no single site's rows straddle both train and test.
3. 3Check test only once, at the end
   The test-set score is checked once, after all tuning is finished; going back to adjust the model afterward means starting a new round of the whole process.

• Any statistic (mean, standard deviation, group aggregate) should be computed from train only, then applied to validation / test
• Confirm the same service site or case does not appear across more than one split
• Confirm a column's production time is before split_time, rather than judging it only by whether its name sounds like a feature
• Check the test-set score only once; if the model needs further adjustment afterward, restart the whole process

Key takeaway: Completing the steps is not enough; the result must be verifiable, reproducible, and explainable.

40 min

Regression and Regression Metrics: MAE, RMSE

Learning goal: fit a simple regression model; interpret MAE/RMSE in domain units.

After training, a colleague reports a single number: "RMSE is 12.4." That number is meaningless on its own unless you know its unit, how it compares to the scale of the target, and what the gap between it and MAE reveals. Regression evaluation is not about computing a metric: it is about translating that metric back into business language like "case-volume error."

When translating MAE or RMSE into business language, always report it alongside the scale of the target. "MAE is 3.2" means roughly a 16% error when the average case volume is 20, but the same 3.2 means only a 1.6% error when the average case volume is 200. Without that scale for comparison, a single error number is nearly impossible to judge.

from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_absolute_error, mean_squared_error

feature_cols = [c for c in df.columns if c.startswith("features_")]
model = LinearRegression().fit(train[feature_cols], train["target_numeric"])
pred = model.predict(valid[feature_cols])

mae = mean_absolute_error(valid["target_numeric"], pred)
rmse = mean_squared_error(valid["target_numeric"], pred, squared=False)
print(round(mae, 2), round(rmse, 2))
# Compare against baseline_mae from Lesson 2 to judge whether the model is worth using

• Report MAE and RMSE together, and explain what the gap between them reveals
• Convert the error into a percentage of the target's mean or median instead of giving only the raw number
• State which split (validation or test) the error was computed on
• Report the gap against the baseline alongside the error, rather than presenting it in isolation

Key takeaway: Completing the steps is not enough; the result must be verifiable, reproducible, and explainable.

55 min

Guided Lab: First Verifiable Predictive Model

Learning goal: with instructor guidance, build a baseline and a model with a fixed split, and compare held-out performance.

• Go to the DS 201 practice page
• PreviewDownload the clean service-demand dataset (CSV, synthetic)
• Download the Guided Lab starter notebook
• PreviewDownload the data dictionary (Traditional Chinese)
• PreviewDownload the data dictionary (English)

1. 1Confirm the prediction problem
   Write down this lab's prediction unit, target_numeric, decision, and the features_* available before split_time.
2. 2Split by time
   Sort by split_time and cut train/validation/test, checking that the three time ranges do not overlap.
3. 3Build a baseline
   Use the training mean as the mean baseline, and compute its MAE on validation.
4. 4Train and compare
   Train a simple regression model, compute its MAE and RMSE on validation, and compare it to the baseline.
5. 5Final evaluation
   Once a model is chosen, check the test-set score exactly once and do not adjust afterward.
6. 6Note what still needs human judgment
   List what this result cannot yet answer or still needs further checking or human judgment (for example, seasonality or extreme events).

• Confirm every reported output can actually be re-run, and that no number was hand-edited
• Clearly state which columns were excluded because of leakage, rather than because they simply hurt performance
• Any "the model beats the baseline" conclusion should include the actual size of the gap, not just "it improved"

Key takeaway: Completing the steps is not enough; the result must be verifiable, reproducible, and explainable.