Data Structures

Data Structures#

Rust’s standard library covers the most common containers, all generic and allocation-aware.

Arrays#

Fixed length, stack-allocated.

let xs: [i32; 5] = [1, 2, 3, 4, 5];
let zeros = [0u8; 1024];

Slices#

A view into contiguous memory. &[T] is read-only, &mut [T] is writable.

let xs = vec![1, 2, 3, 4];
let sub: &[i32] = &xs[1..3];

Vec#

A growable, heap-allocated array.

let mut v: Vec<i32> = Vec::new();
v.push(1);
v.extend([2, 3, 4]);
let n = v.len();

Strings#

String owns; &str borrows. Both are guaranteed UTF-8.

let owned: String = String::from("hello");
let borrowed: &str = &owned;

HashMap / Set#

use std::collections::{HashMap, HashSet};

let mut counts: HashMap<&str, i32> = HashMap::new();
*counts.entry("a").or_insert(0) += 1;

let s: HashSet<i32> = HashSet::from([1, 2, 3]);

Structs#

struct Person {
    name: String,
    age:  u32,
}

impl Person {
    fn new(name: &str, age: u32) -> Self {
        Self { name: name.into(), age }
    }
}

Enums#

Tagged unions; the foundation of Option and Result.

enum Shape {
    Circle(f64),
    Rect { w: f64, h: f64 },
    Square(f64),
}

Option and Result#

fn first_word(s: &str) -> Option<&str> {
    s.split_whitespace().next()
}

fn parse(s: &str) -> Result<i32, std::num::ParseIntError> {
    s.parse::<i32>()
}

Tabular (third-party)#

polars is the operator’s default for tabular workloads in Rust. Lazy or eager columnar DataFrames built on Apache Arrow; faster than pandas on most operations and at home in both single-machine and streaming pipelines. Third-party (cargo add polars), not in the standard library.

Series#

A Series is a 1-D labelled column of one dtype. Construct from a name and a slice.

use polars::prelude::*;
let s = Series::new("latency_ms".into(), &[10_i64, 20, 30]);

Type-cast to a typed iterator.

let xs: Vec<i64> = s.i64()?.into_no_null_iter().collect();

Aggregations are method calls.

let mean = s.mean();      // Option<f64>
let max  = s.max::<i64>()?;

DataFrame#

A DataFrame is a 2-D labelled table; each column is a Series. Construct from columns with the df! macro.

use polars::prelude::*;
let df = df!(
    "host" => &["a", "b", "c"],
    "port" => &[80_i64, 443, 22],
    "open" => &[true, true, false],
)?;

Read from CSV (eager).

let df = CsvReadOptions::default()
    .with_has_header(true)
    .try_into_reader_with_file_path(Some("hosts.csv".into()))?
    .finish()?;

Lazy scan + filter + collect; the query planner pushes the filter to the scanner.

let df = LazyCsvReader::new("hosts.csv")
    .with_has_header(true)
    .finish()?
    .filter(col("open").eq(true))
    .select([col("host"), col("port")])
    .collect()?;

Filter rows on a boolean mask.

let opened = df.clone().lazy().filter(col("open").eq(true)).collect()?;

Add a computed column.

let with_https = df.clone().lazy()
    .with_column(col("port").eq(443).alias("is_https"))
    .collect()?;

Group and aggregate.

let by_open = df.clone().lazy()
    .group_by([col("open")])
    .agg([col("port").mean().alias("avg_port")])
    .collect()?;

Join two DataFrames on a key.

let merged = df.clone().lazy()
    .join(scans.lazy(), [col("host")], [col("host")], JoinArgs::new(JoinType::Left))
    .collect()?;

Pivot long to wide.

use polars::prelude::*;
let wide = pivot::pivot(&events, ["event"], Some(["host"]), Some(["count"]), false, None, None)?;

Method surface (everyday)#

Call

Effect

df.shape()

(n_rows, n_cols).

df.columns()

Column names.

df.dtypes()

Per-column dtypes.

df.head(Some(5))

First N rows.

df.tail(Some(5))

Last N rows.

df.describe(None)?

Summary statistics.

df.sort(["col"], false, false)

Sort by column.

df.unique(...)?

Deduplicate rows.

df.drop("col")?

Drop a column.

df.rename(old, new.into())?

Rename a column.

df.filter(&mask)?

Boolean-mask row filter.

df.lazy().with_column(...)

Add or replace a column (lazy).

df.lazy().group_by(...).agg(...)

Split-apply-combine.

df.lazy().join(other, ...)

SQL-style join.

I/O#

Reader / writer

Format

CsvReadOptions / CsvWriter

CSV.

ParquetReader / ParquetWriter

Apache Parquet (preferred for large data).

JsonReader / JsonWriter

Newline-delimited JSON.

IpcReader / IpcWriter

Apache Arrow IPC.

LazyCsvReader / LazyJsonLineReader / scan_parquet

Lazy variants that compose with the query planner.

The wider Rust tabular / columnar ecosystem. Polars is the operator’s default; the rest are the right tool when the workload is SQL, on-cluster, or interop-first.

Crate

When to reach for it

polars

DataFrames built on Arrow; eager and lazy. The default.

arrow / arrow-rs

The native Rust implementation of the Apache Arrow columnar format. The interchange layer everything else sits on; reach for it directly when building a custom reader / writer.

datafusion

In-process SQL + DataFrame query engine over Arrow. The right tool when the question reads as SQL or when the operator wants a query planner.

connector-x

Fast database → Arrow / pandas / polars loader; the fastest path from Postgres / MySQL / SQLite / Oracle into an in-memory DataFrame.

duckdb

In-process DuckDB bindings; embed an analytical SQL engine inside a Rust binary with zero-copy Arrow interchange.

parquet

Standalone Apache Parquet reader / writer; sits beneath polars / datafusion but is the right dependency on its own when the operator only needs columnar I/O.

References#

Contents