AI-Assisted Analysis: From Question to Executable Code

35 min

What AI Can and Cannot Do in Data Science

Learning goal: map AI roles across framing, code, debugging, interpretation, documentation; retain human responsibility.

You hand an AI assistant the "Analysis Package" dataset and ask: "Tell me which group most needs attention." It quickly returns a block of code, a chart, and a conclusion: "Group unit_b has the worst numbers; recommend reinforcing it first." That answer looks complete, but it hides several unchecked assumptions: does the AI realize the unit column determines which metric value actually represents this time (case volume, response hours, cost per case...)? Did it exclude rows still marked draft or superseded that should not feed a live analysis? Without a human check, this fluent-sounding conclusion may just be a well-written mistake.

import pandas as pd

# clean: the Analysis Package dataset, one row per capacity record for a service group in a given period
df = pd.read_csv("DS202_Analysis_Package_Clean.csv", parse_dates=["period"])
df.shape
df.dtypes
df[["record_id", "period", "group", "value", "unit", "status"]].head()

# Ask first: what unit does value use this time? How many status values exist?
df["unit"].unique()
df["status"].unique()

• Confirm which unit (metric) value represents this time, before judging whether a number is high or low
• Confirm whether rows with status draft or superseded have been excluded
• Confirm whether rows with sensitive_flag true, which should not flow into a general analysis, have been excluded
• Confirm the group and period scope being aggregated actually matches the question you want answered

Key takeaway: Completing the steps is not enough; the result must be verifiable, reproducible, and explainable.

40 min

Data Context: Schema, Definitions, Constraints

Learning goal: provide schema, grain, units, allowed operations, and forbidden assumptions.

A colleague hands AI a file path and says only "compute the average value per group," without mentioning that the unit column doubles as both a measurement unit and a metric identifier (unit "count" means case volume; unit "hours" means average response time), and without mentioning that the status column has three values (validated, draft, superseded) and not every row is ready to analyze. To complete the task, the AI has to guess a set of assumptions on its own: for example, that the whole value column represents one single metric, or that every row is already approved. Those assumptions are rarely spoken aloud, but they are enough to make the entire analysis wrong from the start.

import pandas as pd

df = pd.read_csv("DS202_Analysis_Package_Clean.csv", parse_dates=["period"])

# Grain check: is group + period + unit unique?
df.groupby(["group", "period", "unit"]).size().max()

# What unit values exist? Each one determines which metric this row is.
df["unit"].unique()

# Status distribution: are there rows not yet ready to analyze (draft, superseded)?
df["status"].value_counts()

1. 1Write out the schema and types
   List every column's name, type, and allowed values; never assume AI already knows them.
2. 2Define the grain
   State clearly what one row represents, and whether duplicates or version relationships exist.
3. 3Flag units and conversion rules
   If one column carries different units across rows, say so explicitly and state the conversion or grouping rule.
4. 4List forbidden assumptions
   Write out explicit "do not assume..." statements, such as never assuming every row's status is already validated.

• Have every column's name, type, and allowed values been listed out?
• Has it been stated what one row represents, and whether duplicates or version relationships exist?
• If one column carries different units, has the conversion or grouping rule been made explicit?
• Is the "do not assume..." forbidden list more explicit than the list of what is allowed?

Key takeaway: Completing the steps is not enough; the result must be verifiable, reproducible, and explainable.

40 min

From Analysis Need to Verifiable Prompt

Learning goal: write task/context/requirements/output/verification prompts for data analysis.

"Please analyze this data for me" is a task, but not a verifiable prompt. A verifiable prompt is one where you can later check whether AI actually did what you asked. Breaking a prompt into five parts: task (what to do), context (what the data is), requirements (constraints and rules), output (the output format), verification (what to attach so a person can check it) sharply reduces the room AI has to fill in its own assumptions.

mask = (df["unit"] == "count") & (df["status"] == "validated") & (~df["sensitive_flag"])
filtered = df[mask]

print("Rows before filter:", len(df), "Rows after filter:", len(filtered))

monthly = filtered.groupby(["group", filtered["period"].dt.to_period("M")])["value"].mean()
monthly.sort_values(ascending=False).head()

• Does Task state clearly which column and which metric to use?
• Does Context cover the schema, grain, and version relationships?
• Do Requirements spell out the filter conditions and forbidden operations?
• Does Verification require the code and the before/after row counts to be attached?

Key takeaway: Completing the steps is not enough; the result must be verifiable, reproducible, and explainable.

40 min

AI-Generated Python: Read Before Run

Learning goal: review imports, columns, filters, joins, formulas, side effects, and expected outputs before execution.

A colleague receives a block of AI-generated code that looks professional and runs it right away. It finishes with no error message, so the output goes straight into a report. Three days later, someone discovers the join used the wrong key, silently double-counting some groups' data. Read before run means: before pressing execute, work through a checklist of what the code is actually going to do, rather than waiting for the result and tracing the problem backward.

import pandas as pd

df = pd.read_csv("DS202_Analysis_Package_Clean.csv")
volume = df[df["unit"] == "count"][["group", "period", "value"]].rename(columns={"value": "volume"})
cost = df[df["unit"] == "usd"][["group", "period", "value"]].rename(columns={"value": "cost"})

# This join uses only "group" as the key and forgets to also match on "period",
# so every row for a group gets paired with every other month for that group
merged = volume.merge(cost, on="group")

merged.to_csv("output_summary.csv")  # side effect: overwrites an existing file
print(len(volume), len(cost), len(merged))  # merged has far more rows than volume or cost alone

• Do the column names and types this code uses actually exist and match expectations?
• Do the filter conditions truly correspond to the business rule you need?
• Is the join key unique in the source data? Could it duplicate rows?
• Does the code write to or overwrite an existing file? Is that what you expected?

Key takeaway: Completing the steps is not enough; the result must be verifiable, reproducible, and explainable.

55 min

Guided Lab: AI Pair Analyst

Learning goal: with instructor guidance, use AI to propose and implement an analysis, then independently verify each critical step.

• Go to the DS 202 practice page
• PreviewDownload the clean analysis package dataset (CSV, synthetic)
• Download the Guided Lab starter notebook
• PreviewDownload the data dictionary (Traditional Chinese)
• PreviewDownload the data dictionary (English)

1. 1Supply the data context
   Give AI the schema, grain, unit rules, and forbidden assumptions as the basis for its proposal.
2. 2Ask AI to propose and generate code
   Write a task/context/requirements/output/verification prompt and ask AI to propose a method and implement it.
3. 3Read before run
   Before executing, check imports, columns, filters, joins, formulas, and side effects one by one.
4. 4Independently verify a critical step
   Recompute at least one key number yourself (for example, the filtered row count or one group's average) and confirm it matches AI's output.
5. 5Record what still needs human judgment
   List which decisions in this analysis (for example, whether to include draft rows) still need business confirmation rather than being something the code can decide on its own.

• Judging only whether the conclusion looks reasonable, without recomputing any key number yourself
• Forgetting to exclude rows where sensitive_flag is true from the analysis
• Never requiring the prompt to attach before/after row counts, so there is nothing to check afterward
• Running AI-generated code directly without first reviewing the join and filter logic item by item

Key takeaway: Completing the steps is not enough; the result must be verifiable, reproducible, and explainable.