Company master file
Basic data for 5 synthetic companies: company_id, company_name, sector, hq_region.
DS 102 Practice Center
Use synthetic company and site data to practice type cleaning, missing/duplicate judgment, join cardinality, unit normalization and validation rules, ending in a trustworthy cleaning process.
Info
DS 102 practice data is synthetic teaching data. All company names and values are illustrative and do not represent any real organization.
Warning
Use only the synthetic files provided for practice. Unless authorized, never paste confidential institutional or company data, personal data, or sensitive data into an AI tool; confirm your institution's rules before using real internal operations data.
Download the practice files
Site monthly data (wide format)
Electricity data for 2 companies, 2 sites each, 12 month columns; used for the reshape and join-cardinality practice.
Company-year clean data
5 companies, 2023 to 2025, 15 rows in total. Used for the Guided Lab.
Company-year practice data
Practice data with intentional data-quality problems. Used for the Data Quality Challenge and the Independent Lab.
資料字典（繁體中文）
Column type, unit, required status and description for all three files, in Traditional Chinese.
Data dictionary (English)
English data dictionary; columns and units match the Chinese version.
Guided Lab starter notebook
A starter notebook with guidance and TODOs that runs top to bottom from a fresh kernel.
Independent Lab starter notebook
A starter notebook with a cleaning-decision-log template.
Practice tasks
Practice A: Type & Date Repair
The practice data has a revenue column stored as text with a thousands comma, and one date that cannot be parsed. Inspect first, then handle each problem, keeping a trace of anything that fails to parse.
Files used: Company-year practice data, 資料字典（繁體中文）
Steps
- Load DS102_Company_Year_Practice.csv and check each column's type with df.dtypes.
- Find the numeric column stored as text, confirm the problem character (e.g. a thousands comma), then convert it.
- Parse reported_at with pd.to_datetime(..., errors="coerce") and flag the rows that fail to parse.
- Confirm the handled types match the data dictionary.
What you produce
- A fixed numeric column (a plain numeric type)
- A flag column marking date-parsing failures
- No date that was guessed and filled in
Practice B: Missing/Duplicate Diagnosis
The practice data has missing values and a duplicate row. First judge what the missing value means and whether the duplicate is a true duplicate, then decide how to handle each one.
Files used: Company-year practice data, 資料字典（繁體中文）
Steps
- Find missing values with df.isna().sum() and judge whether each one means not disclosed, not collected, or something else.
- Check for exactly duplicated rows with df.duplicated().sum().
- Build a company_id + year candidate key and confirm whether the key repeats.
- Write down the handling decision and reason for each finding.
What you produce
- A judgment and handling for the missing value
- A judgment and handling for the duplicate row
- The result of checking candidate-key uniqueness
Info
This page does not say how many problems there are in total, and no answer key is published. Bring what you found and your reasoning to the class discussion.
Practice C: Join Cardinality Challenge
site_id in the monthly site data repeats across different companies. Find the trap first, then build the correct join key.
Files used: Site monthly data (wide format), Company master file, 資料字典（繁體中文）
Steps
- Check whether site_id repeats across companies with groupby("site_id")["company_id"].nunique().
- Try joining with the company master using site_id alone, and observe whether the row count matches expectations.
- Build the correct join key from company_id + site_id, and redo the join.
- Compare the row counts of the two joins and explain why they differ.
What you produce
- The result of checking whether site_id repeats
- A comparison of the row counts from both join approaches
- An explanation of the many-to-many risk
Practice D: Unit Normalization Review
Assume another source records the same metric in a different unit. Normalize the unit only when a documented conversion rule exists, and keep the original unit.
Files used: Company-year clean data, 資料字典（繁體中文）
Steps
- Confirm the unit of waste_tonnes from the data dictionary.
- Assume a source recorded in kilograms, and write the formula to convert it to tonnes.
- Keep a column recording the original unit in the converted data.
- Check whether a same-named column has a consistent definition across sources (for example, whether employees always counts full-time only).
What you produce
- A statement of the conversion rule used
- A column preserving the original unit
- The result of checking whether the definition is consistent
Practice E: Data Contract Validation
Turn the data's expected rules into executable checks, and confirm the rules genuinely fail when the data has a problem.
Files used: Company-year clean data, Company master file, Site monthly data (wide format), 資料字典（繁體中文）
Steps
- Write a uniqueness check: company_id + year must never repeat.
- Write a range check: employees must not be negative.
- Write an allowed-values check: sector must be in the allowed list.
- Write a referential-integrity check: site_monthly's company_id must exist in the company master.
- Rerun the checks on the practice data (with quality problems) and confirm at least one rule fails.
What you produce
- At least four executable validation checks
- Proof that a rule genuinely fails on the practice data
- The rule type matching each check (range, allowed-values, uniqueness, referential-integrity)
The done mark is stored in this browser only (and synced once you join a cohort). Keep your work in your own notebook.