Explore Data: Distributions, Variation, and Outliers

25 min

Purpose of EDA: Find Patterns and Counterexamples

Learning objective: use EDA to generate/check questions, not to prove a preselected story.

Your manager looks at a survey summary and says, "Group B clearly did worse - let's report that to the committee." Before you nod along, this is exactly where exploratory data analysis (EDA) belongs: EDA does not exist to prove a story your manager already believes. It exists to generate questions, look for evidence that supports a pattern, and just as actively look for counterexamples that would overturn it.

import pandas as pd

# clean: a survey + outcomes dataset, one row per observation
df = pd.read_csv("DS104_Survey_Outcomes_Clean.csv")
df.shape
df.head()
df["group"].value_counts()

# The manager's claim: Group B has a lower outcome. Look at the whole picture, not one chart
df.groupby("group")["outcome"].mean()

• Write down the question you want to answer before you start charting or computing statistics, so the question does not quietly shift mid-analysis
• For any "A is better/worse than B" claim, recheck it with at least one different cut (stratum, age band, completion status)
• Actively look for a subgroup that runs opposite to the overall pattern, and note what share of the data it represents
• Keep "what the data shows" separate from "whether the data quality supports this claim"

Key takeaway: Completing the steps is not enough; the result must be verifiable, reproducible, and explainable.

35 min

Center and Distribution: Mean, Median, Percentiles

Learning objective: compare robust and non-robust summaries; connect metrics to distribution shape.

A colleague reports, "The mean outcome is 62 - that looks fine." But if a handful of observations sit far above or below everyone else, the mean gets pulled toward those extreme values, and "that looks fine" can be wrong. The median and percentiles offer a view of the data that is not dominated by a few extreme points.

summary = df["outcome"].describe(percentiles=[0.25, 0.5, 0.75])
print(summary)

mean_val = df["outcome"].mean()
median_val = df["outcome"].median()
gap = round(mean_val - median_val, 2)
# A larger gap suggests a skewed distribution, or a few extreme values pulling the mean

p10, p90 = df["outcome"].quantile([0.1, 0.9])
# 10th and 90th percentiles: 90% of observations fall within this range

• Whenever you report a "center" number for outcome, compute both the mean and median first, then decide which to lead with
• A mean-median gap larger than roughly 5-10% of the scale is usually worth checking with a histogram
• When communicating results, describe where most observations fall and how the few outliers deviate, rather than reporting one number alone
• Percentiles (such as p10 and p90) show the overall shape of the data better than a single mean or median can

Key takeaway: Completing the steps is not enough; the result must be verifiable, reproducible, and explainable.

35 min

Variation: Range, IQR, Standard Deviation

Learning objective: describe spread; avoid interpreting a mean without variability.

Two strata both have a mean outcome of 55, which sounds like the same situation. But if Stratum A's observations cluster tightly between 50 and 60 while Stratum B's observations spread from 10 to 100, the two datasets represent very different risks - and reporting only the mean hides that difference entirely.

by_stratum = df.groupby("sampling_stratum")["outcome"].agg(
    mean="mean",
    std="std",
    q25=lambda s: s.quantile(0.25),
    q75=lambda s: s.quantile(0.75)
)
by_stratum["iqr"] = by_stratum["q75"] - by_stratum["q25"]
by_stratum["range"] = df.groupby("sampling_stratum")["outcome"].agg(lambda s: s.max() - s.min())
print(by_stratum)
# Means are close, but std and iqr differ a lot: the two strata spread very differently

• Pair any reported mean with at least one measure of spread (IQR or standard deviation)
• Prefer IQR over standard deviation when the distribution may have extreme values or is clearly skewed
• Range is extremely sensitive to a single extreme value; use it for a quick check only, never as a conclusion by itself
• When comparing two groups, compare their spread before deciding whether a mean difference is meaningful

1. 1Plot the distribution first
   Glance at a histogram or box plot to judge whether it is clearly skewed or has extreme values.
2. 2Pick a metric to match the shape
   Use standard deviation when the distribution is roughly symmetric with no extreme values; prefer IQR when it is skewed or has extreme values.
3. 3Compute both as a cross-check
   Even if you only plan to report one, compute the other too and confirm the two tell a consistent story.

Key takeaway: Completing the steps is not enough; the result must be verifiable, reproducible, and explainable.

35 min

Outliers: Error, Exception, or Signal?

Learning objective: investigate outliers before removal; document rationale and sensitivity.

While checking the exposure column, you find one observation ten times larger than everyone else's. The instinct is often "that must be wrong, delete it" - but an outlier can be one of three very different things: a measurement or entry error, a rare but real exception, or a signal pointing at something important. You need to investigate which one it is before deciding what to do.

q1, q3 = df["exposure"].quantile([0.25, 0.75])
iqr = q3 - q1
lower, upper = q1 - 1.5 * iqr, q3 + 1.5 * iqr

flagged = df[(df["exposure"] < lower) | (df["exposure"] > upper)]
flagged[["observation_id", "exposure", "group", "response_complete", "sampling_stratum"]]
# Look at the other fields for these rows before deciding: error, exception, or signal

1. 1Flag candidates with a rule
   Use the IQR rule (below Q1 - 1.5*IQR or above Q3 + 1.5*IQR) to list candidate outliers without deleting anything yet.
2. 2Check the context
   Look at each candidate's other fields (response_complete, weight, age_band) to judge whether the value is logically consistent.
3. 3Run a sensitivity check
   Compute the mean, median, and IQR both with and without these observations, and see whether the conclusion changes.

• A single extreme value versus a cluster of similar extreme values: the former is more likely an isolated case, the latter more likely a signal
• Whether an extreme value logically contradicts other fields (weight, response_complete); a contradiction usually points to an error
• Before excluding any observation, check whether the conclusion actually changes with and without it; if it does not, the exclusion matters less
• Record both the investigation process and the conclusion in the report, not just the cleaned result after exclusion

Key takeaway: Completing the steps is not enough; the result must be verifiable, reproducible, and explainable.

50 min

Guided Lab: Build an EDA Profile

Learning objective: with instructor guidance, profile distributions, missingness, group differences, and suspicious observations.

• Go to the DS 104 Practice page
• PreviewDownload the clean survey + outcomes file (CSV, synthetic data)
• Download the Guided Lab starter notebook
• PreviewDownload the data dictionary (Traditional Chinese)
• PreviewDownload the data dictionary (English)

1. 1Load and inspect the schema
   Load the clean data and check row count, column types, and the allowed values for group, age_band, and sampling_stratum.
2. 2Check missingness and incomplete responses
   Compute the missing rate for each field and the share of observations where response_complete is false, and see whether it clusters in a specific stratum.
3. 3Build a distribution summary
   Compute mean, median, IQR, and standard deviation for outcome and exposure, and plot a histogram to check the shape.
4. 4Compare group differences
   Compute the mean and spread of outcome by group and age_band, and note any subgroup that looks inconsistent.
5. 5Flag suspicious observations
   Use the IQR rule to flag candidate outliers, investigate one of them in context, and record your judgment.
6. 6State what still needs human judgment
   List the questions this profile cannot yet answer and still need further confirmation or human judgment.

• Showing only the final clean chart without saying what was excluded along the way and on what basis
• Writing "the means differ" as "the two groups are different" without mentioning their respective spread
• Checking only the overall data without confirming the same pattern separately by group, age_band, or sampling_stratum
• Leaving out questions this lab still cannot answer, which makes the report look more certain than it actually is

Key takeaway: Completing the steps is not enough; the result must be verifiable, reproducible, and explainable.