From Samples to Judgment: Association, Uncertainty, and Limits

35 min

Population, Sample, and Sampling Bias

Learning objective: define target population; assess who is represented and who is missing.

The survey data in front of you was collected from respondents who chose to answer, and your manager wants to generalize the conclusion to "every student at the university." Before generalizing, the first question you must answer is: who does this sample actually represent, and who is missing from it? If the target population (all students) systematically differs from the people who ended up in the sample, the conclusion cannot simply be applied to the target population.

sample_share = df["sampling_stratum"].value_counts(normalize=True).round(3)
print(sample_share)

completion_by_stratum = df.groupby("sampling_stratum")["response_complete"].mean().round(3)
print(completion_by_stratum)
# A stratum whose sample share differs a lot from its known population share
# may be over- or under-represented in this sample

• Before generalizing to a target population, write down how that population is defined and how the sample was recruited or collected
• Compare each stratum or group's share in the sample against its known or estimated share in the target population
• Assuming a sample is representative just because it is large enough is one of the most common and most easily overlooked mistakes
• Collecting more data through the same recruitment channel usually just makes the bias more stable, not smaller

Key takeaway: Completing the steps is not enough; the result must be verifiable, reproducible, and explainable.

35 min

Correlation, Confounding, and Causal Limits

Learning objective: interpret association without causal overreach; identify plausible confounders.

You find a clear positive correlation between exposure and outcome, and a colleague immediately says, "So increasing exposure raises outcome." Correlation only describes whether two variables move together; it cannot by itself prove that one causes the other. Before making a causal claim, you need to ask: could a third variable be driving both, making the observed association partly or entirely an illusion?

overall_corr = df[["exposure", "outcome"]].corr().iloc[0, 1]
print(round(overall_corr, 3))

by_age = df.groupby("age_band").apply(lambda g: g["exposure"].corr(g["outcome"]))
print(by_age.round(3))
# If the overall correlation is strong but weakens or disappears within every age_band,
# age_band is likely a confounder rather than exposure truly causing outcome

• When you see a strong correlation, first ask "is there a variable driving both of these," rather than jumping to a causal conclusion
• Use known stratifying variables (such as age_band or group) to check whether the correlation still holds within subgroups
• If the correlation disappears or weakens sharply within strata, the original overall correlation was likely influenced by a confounder
• Causal claims require a more rigorous design than observational correlation; EDA is responsible for description and hypothesis generation, not causal proof

1. 1Compute the overall correlation
   Calculate the overall correlation between exposure and outcome, and note its value and direction.
2. 2List plausible confounders
   Think through which fields (such as age_band or group) could plausibly affect both variables at once.
3. 3Recompute within strata
   Group by the plausible confounder and recompute the correlation within each group, comparing it to the overall figure.
4. 4Adjust the wording to match
   If the correlation weakens or disappears within strata, replace causal language with "the overall association may be driven by another variable."

Key takeaway: Completing the steps is not enough; the result must be verifiable, reproducible, and explainable.

35 min

Estimation, Uncertainty, and Confidence-Interval Intuition

Learning objective: understand interval estimates conceptually; distinguish uncertainty from data quality.

You compute a sample mean outcome of 58.4 and report it as "the answer." But if you drew a fresh sample, that number would almost certainly come out a little different. A sample mean is an estimate of the true value in the target population, not an exact answer - and the point of an interval estimate is to honestly express how uncertain that estimate is.

import numpy as np

rng = np.random.default_rng(seed=104)
values = df["outcome"].dropna().to_numpy()

boot_means = []
for _ in range(1000):
    resample = rng.choice(values, size=len(values), replace=True)
    boot_means.append(resample.mean())

boot_means = np.array(boot_means)
lower, upper = np.percentile(boot_means, [2.5, 97.5])
print(round(lower, 2), round(upper, 2))
# Roughly: if this sampling-and-estimation process were repeated many times,
# about 95% of the resulting mean estimates would fall in this range

• A narrow interval does not mean the estimate is correct; it only means resampling would produce consistent results
• Before growing the sample size, confirm that the measurement method and data collection process have no systematic problem of their own
• When different strata collect data in different ways, check their interval widths separately rather than relying on the pooled overall interval
• When reporting an interval estimate, state clearly that it addresses sampling uncertainty only, not any other data quality issue

Key takeaway: Completing the steps is not enough; the result must be verifiable, reproducible, and explainable.

30 min

Statistical vs Practical Significance

Learning objective: separate evidence of difference from magnitude and decision relevance.

Group A and Group B have mean outcomes of 60.2 and 60.5, and their 95% interval estimates do not overlap - meaning there is evidence the two groups genuinely differ. But the difference is only 0.3, and if the team's decision threshold is "a difference must reach 5 points before it is worth changing policy," this evidence-backed difference may not be worth acting on at all. "There is a difference" and "the difference is large enough to matter" are two separate questions.

group_means = df.groupby("group")["outcome"].mean().round(2)
diff = round(group_means.loc["B"] - group_means.loc["A"], 2)
print("Mean difference:", diff)

decision_threshold = 5.0  # Defined in advance by the team: how large a difference is worth changing policy
print("Meets practical significance threshold:", abs(diff) >= decision_threshold)

• Define the "how large a difference is worth acting on" threshold with stakeholders before the analysis starts, not after the fact
• Report "is there evidence" and "is the difference large enough" as two separate statements, not one blended conclusion
• With a very large sample, even a tiny difference can show up as "evidence exists," which makes the threshold comparison even more important
• If the interval is too wide to judge whether a difference exists, honestly report "insufficient evidence" rather than treating it as "no difference"

Key takeaway: Completing the steps is not enough; the result must be verifiable, reproducible, and explainable.

45 min

Independent Lab: EDA Report

Learning objective: independently deliver a bounded exploratory report with findings, uncertainty, alternative explanations, and next questions.

• Go to the DS 104 Practice page
• PreviewDownload the practice data (with exploration challenges)
• Download the Independent Lab starter notebook
• PreviewDownload the data dictionary (Traditional Chinese)
• PreviewDownload the data dictionary (English)

1. 1State a question
   Write down one concrete question this report will answer, and who would use the result.
2. 2Explore the data
   Check distribution, spread, and missingness, and use the IQR rule to flag and investigate any suspicious observations.
3. 3Quantify uncertainty
   Use resampling or a percentile method to build an interval estimate for your main estimate.
4. 4Compare groups
   Compare at least two groups or strata, and check whether the comparison holds consistently across subgroups.
5. 5Challenge your own explanation
   Identify at least one plausible confounder or sampling bias, and explain how it could change your conclusion.
6. 6Assess practical significance
   Compare the size of the difference you observed against a decision threshold you define and justify yourself.
7. 7Report limitations
   Write down what this analysis cannot yet answer, and what would need more data or a more rigorous design to confirm.

• A clearly stated question and its intended audience
• An exploration summary covering distribution, spread, missingness, and suspicious observations
• An interval estimate for at least one main estimate, with its method explained
• At least one challenge to an association or causal interpretation (confounder or sampling bias)
• Evidence of a difference and practical relevance of that difference, stated as two separate points
• At least one limitation and one follow-up question
• A record of any AI assistance used and what you verified yourself

• Reporting only the effect size and whether evidence exists, with no mention of plausible confounders or sampling bias
• Hiding the multiple groupings tried during exploration and showing only the one that looks most supportive of the conclusion
• Omitting what the analysis cannot answer, which makes the report sound more certain than the evidence actually supports
• Not stating which steps AI helped produce and which parts were verified independently

Key takeaway: Completing the steps is not enough; the result must be verifiable, reproducible, and explainable.