Beginner 16 min

Cleaning Data

Missing values, duplicates, types and text cleanup.

What you will learn

Handle NaN
Fix dtypes
Remove duplicates

Real data is messy: blanks, duplicates, wrong types, inconsistent spelling. Analysts commonly spend most of their time cleaning, and pandas has tools for each problem. The golden rule: inspect first, change deliberately, and keep the raw file untouched.

import pandas as pd
import numpy as np

df = pd.DataFrame({
    "name": ["  Ada ", "linus", "Grace", "Grace", None],
    "age": ["36", "28", "abc", "45", "30"],
    "salary": [120.0, np.nan, 150.0, 150.0, np.nan],
    "joined": ["2021-03-01", "2022-07-15", "not a date", "2019-11-30", "2020-01-01"],
})

df.isna().sum()                        # count missing per column
df.dropna(subset=["name"])             # drop rows missing a name
df["salary"].fillna(df["salary"].median())   # fill with the median
df["salary"].fillna(0)
df.ffill()                             # carry the previous value forward

Decide what missing means before you fill. A blank salary is unknown; filling it with 0 would badly distort averages. Sometimes keeping NaN, and letting statistics skip it, is right.

df["age"] = pd.to_numeric(df["age"], errors="coerce")   # bad values become NaN
df["joined"] = pd.to_datetime(df["joined"], errors="coerce")
df["country"] = df.get("country", "UK").astype("category")   # saves memory
df.dtypes

errors="coerce" turns unparseable values into NaN/NaT so you can find and inspect them rather than crashing.

df.duplicated().sum()
df.duplicated(subset=["name"], keep="first")
df = df.drop_duplicates(subset=["name"], keep="first")

df["name"] = (
    df["name"]
    .str.strip()          # remove surrounding spaces
    .str.title()          # Consistent capitalization
)
df["email"] = df["email"].str.lower().str.replace(r"\s+", "", regex=True)
df[df["name"].str.contains("ada", case=False, na=False)]

df = df.rename(columns={"joined": "join_date"})
df["country"] = df["country"].replace({"United Kingdom": "UK", "U.K.": "UK"})

# flag outliers with the IQR rule
q1, q3 = df["salary"].quantile([0.25, 0.75])
iqr = q3 - q1
outliers = df[(df["salary"] < q1 - 1.5 * iqr) | (df["salary"] > q3 + 1.5 * iqr)]

Method chaining

Chaining steps reads like a recipe and avoids scattered temporary variables.

clean = (
    df
    .assign(name=lambda d: d["name"].str.strip().str.title())
    .dropna(subset=["name"])
    .drop_duplicates(subset=["name"])
    .assign(age=lambda d: pd.to_numeric(d["age"], errors="coerce"))
)

SettingWithCopyWarning

This warning means you may be modifying a temporary copy. Use df.loc[mask, "col"] = value to assign, or call .copy() after slicing.

Try it yourself

Given a column of prices as strings like "$1,200", "850" and "n/a", convert it to numbers (NaN for bad values) and fill NaN with the median.

Show solution

s = pd.Series(["$1,200", "850", "n/a"])
num = pd.to_numeric(s.str.replace(r"[$,]", "", regex=True), errors="coerce")
print(num.fillna(num.median()))