DataFrame

DataFrame#

pandas.DataFrame is a 2-D labelled tabular container: rows indexed by a row-index, columns indexed by a column-index, each column a Series of one dtype. Third-party (pip install pandas), not in the standard library. The operator’s default container for any analysis that fits in memory; for larger-than-memory workloads, polars or duckdb cover the same structure.

Construct from a dict of column-name → list.

import pandas as pd

df = pd.DataFrame({
    "host":   ["a", "b", "c"],
    "port":   [80, 443, 22],
    "open":   [True, True, False],
})

Construct from a list of records.

df = pd.DataFrame([
    {"host": "a", "port": 80,  "open": True},
    {"host": "b", "port": 443, "open": True},
    {"host": "c", "port": 22,  "open": False},
])

Read from CSV.

df = pd.read_csv("hosts.csv")

Read from JSON Lines.

df = pd.read_json("events.ndjson", lines=True)

Select a column as a Series.

df["port"]

Select multiple columns as a sub-DataFrame.

df[["host", "port"]]

Label-based row selection with .loc.

df.loc[df["open"], ["host", "port"]]

Positional row selection with .iloc.

df.iloc[0:5]

Filter with a boolean mask.

df[df["port"] >= 443]

Add a computed column.

df["is_https"] = df["port"] == 443

Group and aggregate.

df.groupby("open")["port"].mean()

Merge two DataFrames on a key.

merged = pd.merge(df, scans, on="host", how="left")

Pivot long → wide.

wide = events.pivot_table(index="host", columns="event", values="count", aggfunc="sum")

Method catalog#

The everyday surface, grouped by purpose.

Shape and inspection#

Method

Effect

shape

(n_rows, n_cols) attribute.

columns

The column index.

dtypes

Per-column dtype.

info()

Memory + dtype summary.

describe()

Summary statistics on numeric columns.

head(n=5)

First n rows.

tail(n=5)

Last n rows.

sample(n)

Random sample.

Filter and transform#

Method

Effect

query(expr)

Filter rows by a string expression.

assign(**kw)

Return a copy with new / replaced columns.

rename(columns=mapping)

Rename columns or index.

drop(columns=...)

Drop columns (or rows with index=).

drop_duplicates()

Deduplicate rows.

sort_values(by=...)

Sort by one or more columns.

apply(f, axis=0|1)

Apply f over rows (axis=1) or columns.

map(f) / applymap(f)

Element-wise transform.

Group, join, reshape#

Method

Effect

groupby(key).agg(...)

Split-apply-combine; agg, transform, filter.

merge(other, on=...)

SQL-style join.

concat([a, b])

Stack along an axis.

pivot_table(...)

Long → wide aggregation.

melt(...)

Wide → long.

stack() / unstack()

Move a level between index and columns.

Missing data#

Method

Effect

isna() / notna()

Boolean masks.

dropna(how="any"|"all")

Drop rows / columns with missing values.

fillna(v)

Replace missing values.

interpolate()

Fill via interpolation along an axis.

I/O#

Method

Effect

to_csv(path) / read_csv(path)

CSV round-trip.

to_json(path) / read_json(path)

JSON round-trip.

to_parquet(path) / read_parquet(path)

Columnar binary; preferred for large data.

to_sql(table, conn) / read_sql(sql, conn)

Round-trip against any DBAPI connection.

to_numpy()

Underlying numpy 2-D array.

to_dict(orient=...)

Convert to a dict (records, list, index, etc.).

References#

Contents