Beginner 14 min

DataFrames and Series

Load data, inspect it and select rows and columns.

What you will learn

Create and load DataFrames
Inspect with head/info/describe
Select with loc and iloc

pandas is Python's standard library for working with tables of data: spreadsheets, CSV files, database results. Its two core types are the Series (one column of labeled values) and the DataFrame (a table of Series sharing the same row index). If you can picture a spreadsheet, you can picture a DataFrame.

pip install pandas

import pandas as pd

df = pd.DataFrame({
    "name": ["Ada", "Linus", "Grace", "Alan"],
    "country": ["UK", "FI", "US", "UK"],
    "age": [36, 28, 45, 41],
    "salary": [120, 95, 150, 110],
})
print(df)

Output

    name country  age  salary
0    Ada      UK   36     120
1  Linus      FI   28      95
2  Grace      US   45     150
3   Alan      UK   41     110

df = pd.read_csv("employees.csv")
df = pd.read_excel("report.xlsx")       # needs openpyxl
df = pd.read_json("data.json")
# from a database: pd.read_sql("SELECT * FROM users", connection)

df.head()          # first 5 rows
df.shape           # (rows, columns)
df.info()          # column names, types, non-null counts
df.describe()      # count, mean, std, min, quartiles, max for numbers
df["country"].value_counts()   # frequency of each value

Always do this before anything else. Surprises such as unexpected types, empty columns or odd ranges show up immediately.

df["name"]                      # one column -> Series
df[["name", "salary"]]            # several -> DataFrame
df.loc[0]                         # a row by label
df.loc[0:1, "name":"age"]         # label slices are inclusive
df.iloc[0:2, 0:2]                 # by integer position, end exclusive

uk = df[df["country"] == "UK"]
rich_uk = df[(df["country"] == "UK") & (df["salary"] > 115)]   # & | ~, with parentheses!
names = df[df["age"] > 30]["name"]
df.query("age > 30 and country == 'UK'")

Common error

Use & and |, not and and or, and wrap each comparison in parentheses. Otherwise pandas raises "truth value of a Series is ambiguous".

New columns, vectorized

Operate on whole columns at once instead of looping; it is far faster and shorter.

df["salary_k"] = df["salary"] * 1000
df["senior"] = df["age"] >= 40
df["name_upper"] = df["name"].str.upper()
df = df.sort_values("salary", ascending=False)

Try it yourself

Create a DataFrame of five products with price and quantity, add a total column, and show only rows where the total is above 100, sorted by total descending.

Show solution

items = pd.DataFrame({"product": list("ABCDE"), "price": [10, 25, 5, 40, 8], "qty": [3, 5, 20, 2, 10]})
items["total"] = items["price"] * items["qty"]
print(items[items["total"] > 100].sort_values("total", ascending=False))