Text and Categorical Data

Work with text in pandas using .str methods, regex extract and split, then save memory and set sort order with the category dtype and ordered categories.

• Intermediate
• 16 min read
• 4 objectives

Before this lessonLesson 5: Transforming Columns: vectorization, map and apply

The cleaning lesson used a few .str methods to trim and lowercase messy values. This lesson goes further. Text columns often hide structured information (an email domain, a product code, a version number) that you want as separate columns. And many text columns are not really free text at all: they hold a small set of repeated labels like country, plan or status. pandas has a special category type for those, which saves memory and lets you define a meaningful order.

The .str accessor

Any string method you know from Python has a vectorized twin under .str: lower, upper, strip, startswith, len and more. They work on the whole column and skip missing values instead of crashing on them.

import pandas as pd

emails = pd.Series(["Ada@Stackcone.io", "grace@example.com", None, "linus@stackcone.io"])

print(emails.str.lower())
print(emails.str.len().tolist())
print(emails.str.endswith("stackcone.io").tolist())

0      ada@stackcone.io
1     grace@example.com
2                   NaN
3    linus@stackcone.io
dtype: str
[16.0, 17.0, nan, 18.0]
[False, False, False, True]

In pandas 3, text columns use a dedicated string dtype (shown as str) by default instead of the generic object type. Missing values in such columns show up as NaN, and boolean results from methods like endswith return False for missing entries. Also notice that Ada@Stackcone.io did not match: string methods are case-sensitive, so lowercase first when case should not matter.

Searching and replacing with regex

str.contains returns a boolean mask you can filter with. By default the pattern is a regular expression; pass regex=False to search for a literal string (important when your text contains characters like . or +). case=False makes the match case-insensitive.

import pandas as pd

tickets = pd.DataFrame({
    "id": [101, 102, 103, 104],
    "subject": ["Login fails on Safari", "Refund request", "LOGIN loop after reset", "Invoice #88 wrong"],
})

print(tickets[tickets["subject"].str.contains("login", case=False)])

tickets["clean"] = tickets["subject"].str.replace(r"#\d+", "#<n>", regex=True)
print(tickets["clean"].tolist())

    id                 subject
0  101   Login fails on Safari
2  103  LOGIN loop after reset
['Login fails on Safari', 'Refund request', 'LOGIN loop after reset', 'Invoice #<n> wrong']

Splitting and extracting

str.split(sep, expand=True) splits each value and spreads the pieces into columns. str.extract uses a regex with capture groups; each group becomes a column, and named groups ((?P<name>...)) become column names. Rows that do not match get missing values, which makes bad data easy to find.

import pandas as pd

df = pd.DataFrame({
    "email": ["ada@stackcone.io", "grace@example.com", "linus@kernel.org"],
    "sku": ["BK-2031-L", "GM-0042-M", "bad-code"],
})

df[["user", "domain"]] = df["email"].str.split("@", expand=True)

parts = df["sku"].str.extract(r"(?P<category>[A-Z]{2})-(?P<number>\d{4})-(?P<size>[SML])")
df = df.join(parts)
print(df)

               email        sku   user        domain category number size
0   ada@stackcone.io  BK-2031-L    ada  stackcone.io       BK   2031    L
1  grace@example.com  GM-0042-M  grace   example.com       GM   0042    M
2   linus@kernel.org   bad-code  linus    kernel.org      NaN    NaN  NaN

The category dtype

A column with a million rows but only five distinct values stores the same strings over and over. Converting it to category stores each distinct value once and keeps a small integer code per row. That typically cuts memory by 90% or more and speeds up grouping and sorting.

import pandas as pd

plans = pd.Series(["free", "pro", "team", "free", "pro"] * 20_000)
as_cat = plans.astype("category")

print(as_cat.cat.categories.tolist())
print(as_cat.cat.codes.head(5).tolist())
print("smaller:", as_cat.memory_usage(deep=True) < plans.memory_usage(deep=True))

['free', 'pro', 'team']
[0, 1, 2, 0, 1]
smaller: True

Categories behave like normal text in filters and groupby. The main thing to know is that you cannot assign a value that is not one of the categories without adding it first with .cat.add_categories([...]).

Ordered categories

Some labels have a natural order that alphabetical sorting gets wrong: sizes (S, M, L), priorities (low, medium, high), or ratings. An ordered categorical fixes that. Sorting follows your order, and comparisons like >= "medium" just work.

import pandas as pd

tasks = pd.DataFrame({
    "task": ["deploy", "docs", "hotfix", "refactor"],
    "priority": ["medium", "low", "high", "medium"],
})

print(tasks.sort_values("priority")["priority"].tolist())   # alphabetical, wrong

level = pd.CategoricalDtype(["low", "medium", "high"], ordered=True)
tasks["priority"] = tasks["priority"].astype(level)

print(tasks.sort_values("priority")["priority"].tolist())   # logical order
print(tasks[tasks["priority"] >= "medium"]["task"].tolist())

['high', 'low', 'medium', 'medium']
['low', 'medium', 'medium', 'high']
['deploy', 'hotfix', 'refactor']

When you group by a categorical column, pandas uses observed=True by default, so only categories that actually appear in the data show up in the result. Pass observed=False if you want empty categories listed with zero counts, which is useful for reports that must always show every level.

Recap

• .str gives you vectorized string methods that skip missing values.
• str.contains and str.replace use regex by default; use raw strings and regex=False for literals.
• str.split(expand=True) and str.extract with named groups turn text into columns.
• Convert low-cardinality text to category for big memory savings.
• Ordered categoricals sort and compare in the order you define.

# Write your solution here