HDF5

HDF5#

HDF5 (Hierarchical Data Format version 5) is a binary container format for large, heterogeneous numerical datasets. A single .h5 (or .hdf5) file holds a directory-like hierarchy of named datasets, each with its own shape, dtype, compression, and metadata.

Built and maintained by the HDF Group; the de-facto format for scientific computing, climate / earth-science data, NASA missions, many machine-learning checkpoints, and large numerical archives.

The Mental Model#

An HDF5 file looks like a filesystem inside a single file.

experiment.h5
├── /metadata
│     attrs: {date: 2026-04-27, author: "operator"}
├── /run/00
│     ├── temperature   (float64, shape (3600, 4))
│     ├── pressure      (float32, shape (3600,))
│     └── attrs: {sensor: "S1", units: "K, hPa"}
├── /run/01
│     ├── temperature   (float64, shape (3600, 4))
│     └── pressure      (float32, shape (3600,))
└── /labels             (variable-length string, shape (1000,))

Two main object kinds.

Groups, like directories. Contain other groups and datasets.
Datasets, like files. N-dimensional typed arrays with optional compression.

Both can carry attributes, small key/value metadata.

What HDF5 Solves#

The kind of problem HDF5 was designed for: multi-gigabyte arrays with metadata, mixed in one file, read with random- access slices, archived for decades. Each capability below maps to a real pain in scientific computing the format set out to remove.

Large arrays, multi-GB datasets in a single file, with partial / chunked reads.
Heterogeneous bundles, many arrays of different shapes / dtypes in one container.
Self-describing, schema, dtypes, units, history embedded in the file.
Random access, read a slice from a 1 TB dataset without loading it all.
Cross-platform binary, IEEE-754 floats, well-defined integer byte order; machine-independent.
Compression per dataset, gzip, LZF, SZIP, Zstd (via plugin).
Long-term archival, documented format with stable readers going back 25+ years.

File Anatomy (At a Glance)#

The on-disk format is a self-describing tree pointed to from a superblock at the head of the file. B-trees index the groups and dataset chunks; heaps hold names and small metadata. The operator never authors these structures directly; libhdf5 manages them.

Superblock, file-level metadata; version info; root group pointer.
B-tree indexes, locate groups and chunks within datasets.
Heaps, store object names and small metadata.
Chunks, datasets are stored as fixed-size N-dimensional tiles; compression / I/O / cache work per chunk.

You don’t write this layer by hand; the library (libhdf5) handles it.

Chunking and Compression#

The two performance levers.

Chunk shape, pick chunks that match the access pattern.

Reading whole rows? Make the row axis the small axis.
Reading time slices of multi-dimensional data? Make the time axis a single chunk.
Bad chunking can produce 100× I/O slowdowns on otherwise-good datasets.

Compression, per dataset.

gzip, universal, modest ratio, slow.
lzf, fast, lower ratio.
szip, niche; license-restricted.
zstd, via filter plugin; modern default for new datasets.
blosc, multi-codec; fast for numerical data.

Plus optional shuffle filter, byte-permute floats before compression; often improves the ratio significantly on numeric data.

Tooling#

The HDF5 Group ships a CLI and GUI suite for inspecting, diffing, and rewriting files; on top of that, every scientific computing language has a binding. The list below is the set of tools an operator reaches for when an .h5 file lands on disk.

HDFView, GUI for exploring files.
h5dump, text dump of any file.
h5ls, list groups / datasets.
h5repack, rewrite with different chunking / compression.
h5diff, compare two files.
h5stat, summary statistics.
pyhdf5, h5py, PyTables, Python.
hdf5r, R.
JLD2.jl, Julia (HDF5-shaped).
Native bindings in Java, C, C++, Fortran, MATLAB.

Per-Language Examples#

Three concrete entry points, one Python via h5py, one Python via PyTables (a pandas-friendly layer with query-on-disk), and one shell session for inspecting and repacking files. Together they cover the read/write/inspect cycle that an operator works through daily.

Python (h5py):

import h5py, numpy as np

# Write
with h5py.File("experiment.h5", "w") as f:
    g = f.create_group("run/00")
    g.attrs["sensor"] = "S1"
    g.create_dataset("temperature",
                     data=np.random.randn(3600, 4),
                     compression="gzip", compression_opts=4,
                     chunks=(360, 4), shuffle=True)

# Read a slice without loading the whole file
with h5py.File("experiment.h5", "r") as f:
    last_hour = f["run/00/temperature"][-3600:]

Python (pandas + PyTables):

import pandas as pd
df.to_hdf("store.h5", key="orders", mode="w", complib="blosc",
          complevel=5, format="table")
df = pd.read_hdf("store.h5", key="orders",
                 where="amount > 100")     # query subset

Command line:

$ h5dump -H experiment.h5
$ h5ls -r experiment.h5
$ h5dump -d /run/00/temperature experiment.h5

$ h5repack -f GZIP=9 -l CHUNK=360x4 in.h5 out.h5

Where HDF5 Wins#

The kind of workload where HDF5 is the right tool, not a fallback. Scientific multi-array bundles, large local archives, mixed-dtype containers, and long-lived data formats all play to HDF5’s strengths, and the format has 25+ years of stable readers backing the choice.

Scientific datasets, climate, earth, astronomy, particle physics. Standard in HDF5 and its sibling NetCDF-4 (which uses HDF5 as its underlying format).
Machine learning checkpoints, Keras / TensorFlow saved models historically; PyTorch can use it via torch.save to HDF5 custom paths.
Large numerical archives, TBs of typed arrays in one file.
Mixed dtypes in one container, floats, ints, strings, complex, enums, compound types.
Long-term data, the format is stable; old files keep working.

Where HDF5 Loses#

The mirror image: the workloads where HDF5 was not designed to shine. Cloud-native partial reads, multi-writer concurrency, distributed processing, and partial-write recovery are all weaknesses inherited from the single-file tree-on-disk model. For these, Zarr or Parquet wins.

Concurrent writes, HDF5 is single-writer. SWMR (Single Writer, Multiple Reader) mode helps but with constraints.
Cloud storage, “one big file in S3” works for read-only whole-file but partial reads are awkward; modern tooling prefers Parquet / Zarr for cloud-native arrays.
Concurrent partial writes, not designed for many writers.
Distributed processing, Parquet / Zarr partition naturally across files; HDF5 doesn’t.
Schema evolution, you can add datasets but renaming / reorganizing existing ones is awkward.
File corruption, a partial write can render the whole file unreadable. Always have backups; prefer write-then-rename.
Variable-length strings, supported but slow and easy to use inefficiently.
License gotchas, the format is open; some legacy tools (SZIP) carried license complications.

HDF5 vs. Parquet#

The most common comparison an operator picking a binary format faces. The two solve different kinds of problem; HDF5 a hierarchy of N-D arrays, Parquet a single tabular dataset. The cloud-native gap is what most often drives a project toward Parquet today.

Aspect	HDF5 Parquet
Structure	Multi-dataset hierarchy Single tabular dataset (typically)
Layout	Chunked N-D arrays Columnar row groups
Best for	Scientific multi-array data Tabular analytics
Cloud-native	Awkward First-class
Schema	Embedded; hierarchical Embedded; flat-with-nesting
Concurrency	Single writer (mostly) Many small files concurrent
Compression	gzip / lzf / szip / blosc / zstd snappy / gzip / zstd / lz4 / brotli
Tools	h5py / PyTables / HDFView pyarrow / DuckDB / Spark / Polars

For new analytics data: Parquet. For scientific multi-array bundles or HDF5-shaped legacy: HDF5. Zarr is the cloud-native spiritual successor to HDF5 for N-D arrays.

Zarr, the Modern Cousin#

Zarr takes HDF5’s chunked-N-D-array idea and restructures it as many small files in a hierarchy (filesystem, S3, GCS) plus JSON metadata. Each chunk is a single object. Result. parallel reads, easy cloud storage, multi-writer (with care).

In 2026, Zarr is the recommended choice for new cloud-native scientific datasets; HDF5 remains for local files, legacy compatibility, and tools that already integrate.

NetCDF-4#

NetCDF is a scientific-data format from Unidata. NetCDF-4 uses HDF5 as its container with conventions on top: dimensions, coordinate variables, CF-Conventions metadata.

If you work in climate / oceanography / atmospheric science, you’ll read NetCDF-4 files; underneath, it’s HDF5.

Pitfalls#

The traps that catch teams new to HDF5. Chunk shape and single-writer semantics are the two that bite first; the others surface as datasets grow and the ecosystem of files matures. Most are recoverable with h5repack and care during writes.

Bad chunk shapes, read patterns that cross chunks pull whole chunks for a few values. Profile and h5repack if needed.
String columns, variable-length strings are slow; fixed-length is much faster.
Many tiny datasets, HDF5 has per-object overhead; avoid thousands of tiny groups when one big array would do.
Single writer, two processes opening the same file for write produces corruption. Coordinate, or use the SWMR mode carefully.
File handle leaks, libhdf5 caches; close files explicitly or use context managers.
Endianness, HDF5 stores files in their native byte order; cross-platform readers handle the swap, but extreme-performance paths assume one direction. Specify the type with explicit byte order if portability matters.

When to Use HDF5#

The kinds of project where HDF5 is still the right pick in 2026. Local single-machine numerical work, scientific ecosystems with HDF5-native tooling, and long-term archives all fit. The common thread: data that lives on a filesystem, not on object storage.

Scientific data with multiple typed arrays in one container.
Long-term local archives of numerical data.
Existing ecosystems, climate / NetCDF, ML weights using HDF5, HDF-based legacy pipelines.
Single-machine numerical workflows with random-access slices into multi-GB arrays.

When to Skip#

The mirror list: the kinds of project that should reach for something other than HDF5. Cloud-native, multi-writer, streaming, and tabular-analytics workloads all have better homes in 2026, in Zarr, Parquet, or row-oriented formats depending on which axis matters most.

Cloud storage, prefer Zarr or Parquet.
Multi-writer / distributed, prefer Zarr or Parquet.
Streaming / append-only, prefer line-based formats or Parquet files per partition.
Tabular analytics, Parquet.
Configuration / human readability, YAML or JSON.

Workflow#

Extract, parse, filter, save. HDF5 is hierarchical: datasets sit at paths inside a file. h5dump for inspection on the command line; h5py or pandas for processing.

Extract and inspect.

$ h5dump -n events.h5                     # list datasets and groups
$ h5dump -d /events/severity -H events.h5 # header of one dataset
$ h5ls -r events.h5                       # tree view

Parse and filter (Python).

import h5py

with h5py.File("events.h5", "r") as f:
    severity = f["/events/severity"][:]
    mask     = severity == b"high"
    user_id  = f["/events/user_id"][mask]
    action   = f["/events/action"][mask]
    ts       = f["/events/ts"][mask]

Save the filtered subset.

with h5py.File("high.h5", "w") as out:
    g = out.create_group("/events")
    g.create_dataset("user_id", data=user_id, compression="gzip")
    g.create_dataset("action",  data=action,  compression="gzip")
    g.create_dataset("ts",      data=ts,      compression="gzip")

Tabular data via pandas (HDF5 as a pandas-compatible store).

import pandas as pd
df = pd.read_hdf("events.h5", "events")
df[df["severity"] == "high"].to_hdf("high.h5", "events", complib="zlib")