Cleaning Data with pandas: missing values and duplicates

Why clean data before analysing it?

In the Python + AI/ML Beginner roadmap (/roadmap/python-ai-ml-beginner/), Weeks 10–13 are "Data with Python": NumPy and pandas, cleaning and exploring datasets, charts that explain data. Cleaning sits there for a reason — it is the gate between "I have data" and "I can trust what the data says".

Real datasets arrive messy. A shop export has a customer name missing here, a duplicated order row there, a blank cell where the clerk forgot to type. If you average the "age" column without noticing a blank, the blank is silently ignored and your average is wrong. If you count orders without removing a duplicate, you report five orders when there were four. Machine learning models are even less forgiving: most of them refuse to train on data with missing values at all, or silently learn the wrong pattern from them.

So cleaning is not housekeeping you rush through. It is the step where you decide, column by column, what each gap means — and document the decision in code.

The messy dataset we will work with

Everything below runs in any Python environment with pandas installed. Check first:

import pandas as pd
print(pd.__version__)   # expected: 2.1.4 (the version this tutorial was tested on)

No pandas? pip install pandas in your terminal fixes that in one line.

Our dataset is a tiny shop export — five rows, deliberately broken:

data = {
    "name":   ["Ali", "Sara", "Ali", "Bilal", None],
    "age":    [25, 30, 25, None, 28],
    "city":   ["Lahore", "Karachi", "Lahore", "Islamabad", "Karachi"],
    "orders": [3, 5, 3, 2, None],
}
df = pd.DataFrame(data)

Read it like a spreadsheet: rows are records (people), columns are facts about them. Two defects are hidden in here — a duplicated row and three missing values. The point of the next two sections is that you *discover* them with code instead of squinting.

How do I find the missing values?

pandas represents a missing value as NaN ("not a number") — even in a text column. isna() asks every cell "are you missing?" and returns a table of True/False; .sum() adds those up per column:

print(df.isna().sum())          # find the gaps

name      1
age       1
city      0
orders    1
dtype: int64

Three columns have one gap each; city is complete. This one-liner is the report you run at the start of every cleaning session — it tells you *where* the problems are before you touch anything.

Notice what the output teaches you about the column type: there is one gap in age, so pandas stores the whole column as float (25.0, 30.0 …), because NaN is a float value and a column must have one type. That is why Bilal's age prints as 28.0 later, not 28. Not a bug — just how mixed missing data works.

How do I find the duplicate rows?

duplicated() compares every row against every earlier row and flags exact repeats:

print(df.duplicated())          # find the duplicates

0    False
1    False
2     True
3    False
4    False
dtype: bool

Only row 2 is flagged — it is an exact copy of row 0 (Ali, 25, Lahore, 3). Row 1 (Sara) and row 3 (Bilal) are one-offs; row 4 looks similar to row 1 but the missing name and missing order count make it a different row, so it is kept. duplicated() keeps the *first* occurrence and flags later copies, which is the behaviour you want: the first row is the original, everything after is the accident.

How do I fix them — drop, or fill?

The rule of thumb: drop rows that add no information, fill values where the row is otherwise useful.

• Duplicates → drop. Row 2 is an exact copy; keeping it would double-count Ali. drop_duplicates() returns a new DataFrame without it. Assign that to a new variable so the raw data survives — and add .copy() after it, which lets you edit the new frame without pandas warning you that you might be editing a slice of the old one.
• Bilal's age (numeric, one gap) → fill with the median. The median of the remaining ages (25, 30, 28) is 28.0 — it is not pulled toward extremes the way an average can be, and it keeps the row usable instead of deleting a real customer.
• The missing name (text) → fill with "Unknown". Honest: it says "we do not know", not a name you invented.
• The missing order count → fill with 0, as an integer. A customer with no recorded orders is treated as zero orders; .astype(int) restores the column to whole numbers after the fill.

Each decision is one line of code — and each line is the documentation of what you chose.

Worked example: the full cleaning run, start to finish

Here is the complete script, run exactly as written (pandas 2.1.4, Python 3.12.3, November 2026). The outputs after it are the real ones:

import pandas as pd

data = {
    "name":   ["Ali", "Sara", "Ali", "Bilal", None],
    "age":    [25, 30, 25, None, 28],
    "city":   ["Lahore", "Karachi", "Lahore", "Islamabad", "Karachi"],
    "orders": [3, 5, 3, 2, None],
}
df = pd.DataFrame(data)

print(df.isna().sum())          # find the gaps
print(df.duplicated())          # find the duplicates

clean = df.drop_duplicates().copy()
clean["age"] = clean["age"].fillna(clean["age"].median())
clean["name"] = clean["name"].fillna("Unknown")
clean["orders"] = clean["orders"].fillna(0).astype(int)
print(clean)
print(clean.isna().sum())       # confirm: nothing missing

name      1
age       1
city      0
orders    1
dtype: int64

0    False
1    False
2     True
3    False
4    False
dtype: bool

      name   age       city  orders
0      Ali  25.0     Lahore       3
1     Sara  30.0    Karachi       5
3    Bilal  28.0  Islamabad       2
4  Unknown  28.0    Karachi       0

name      0
age       0
city      0
orders    0
dtype: int64

Read the last two blocks carefully — they are the proof:

• The cleaned table has four rows, not five: row 2 is gone. Note the index jumps from 1 to 3; pandas keeps the original row labels, which is fine for analysis.
• Bilal's age is now 28.0 — the median of 25, 30 and 28.
• The nameless row 4 is "Unknown", with 0 orders (as an integer, not 0.0).
• The final isna().sum() shows zero in every column. That last line is the habit to keep: after every cleaning session, re-run the gap report and confirm it reads all zeros.

Common mistakes

• Editing a slice and getting a SettingWithCopyWarning. After drop_duplicates(), always add .copy() before assigning into the frame. The warning means pandas is not sure which DataFrame you are editing — the copy removes the doubt.
• Filling numbers with 0 by default. Zero orders is honest; zero age would be nonsense. Match the fill value to what the column means, not what is convenient.
• Dropping rows with any missing value too eagerly. df.dropna() on the raw data would delete rows 2, 3 and 4 — three of five rows gone, including real customers. Drop only what you can defend; fill the rest.
• Forgetting to re-check after cleaning. The final isna().sum() is not decoration — run it every time. A fill you typo'd ("Unknown" vs a column name) leaves the gap in place, and only the re-check catches it.
• Cleaning the original instead of a copy. If a fill strategy turns out wrong (the median was the wrong choice), you want the raw data untouched to start over. clean = df.drop_duplicates().copy() keeps df pristine.

Frequently asked questions