Parquet

Parquet#

Apache Parquet is a columnar binary file format optimized for analytical workloads. Originated at Twitter and Cloudera, donated to the Apache Foundation in 2013; the storage format the operator finds under nearly every data lake, lakehouse, and analytics warehouse in 2026.

Where JSON / Protobuf encode a single message, Parquet stores many rows of structured data per file, organized by column for fast aggregation.

Row-Oriented vs. Column-Oriented#

The single most important structural distinction in tabular storage, and the one that drives every other Parquet design decision. Row layouts pack a record together; column layouts pack a field together across records. The trade-off mirrors the workload.

Given a table.

id

age

name

1

30

Ada

2

28

Alan

3

35

Grace

Row-oriented (CSV, JSON Lines, Avro) writes 1, 30, Ada together, then 2, 28, Alan, and so on. Reading one row is one seek. Aggregating one column means reading every byte.

Column-oriented (Parquet, ORC, Arrow) writes [1, 2, 3] together, then [30, 28, 35], then ["Ada", "Alan", "Grace"]. Aggregating one column reads only that column’s bytes; reading one row touches every column.

Analytics is column-aggregating; OLTP is row-fetching. Parquet’s layout mirrors the workload it targets.

File Anatomy#

A Parquet file is a stack of row groups, each row group a stack of column chunks, each column chunk a stack of pages, with a Thrift-encoded metadata footer at the end. Readers parse the footer first, then range-fetch only the column chunks they need.

        flowchart TB
  M1["Magic ("PAR1")"]
  subgraph RG1 [Row Group 1]
    direction TB
    CC1["Column Chunk: id<br/>Page 1, Page 2, ..."]
    CC2["Column Chunk: age"]
    CC3["Column Chunk: name"]
    CC1 --- CC2 --- CC3
  end
  RG2["Row Group 2<br/>..."]
  META["File Metadata (Thrift-encoded)<br/>• Schema<br/>• Row group locations<br/>• Column statistics (min/max/null)<br/>• Encoding / compression info"]
  FT["Footer length (4 bytes)<br/>Magic (&quot;PAR1&quot;)"]
  M1 --> RG1 --> RG2 --> META --> FT

  • Row group, a horizontal slice of N rows (typically 128 MB to 1 GB of data per group). The unit of parallelism.

  • Column chunk, one column inside one row group. The unit of read.

  • Page, the smallest IO unit (default 1 MB). Compressed individually.

  • Footer, read first; gives offsets, schema, and statistics so readers can skip irrelevant data.

Why It’s Fast#

The speed comes from four compounding wins, all enabled by the columnar layout. Each one alone would justify the format; in combination they’re why a 2026 analytics stack runs queries over terabytes on object storage and answers in seconds rather than minutes.

  • Predicate pushdown, a query like WHERE age > 50 reads column statistics in the footer; row groups whose max(age) <= 50 are skipped entirely.

  • Column pruning, SELECT id, name reads only those columns’ bytes.

  • Compression-friendly, columns of the same type compress much better than rows. RLE on low-cardinality, DELTA on monotonically increasing, DICTIONARY on string columns with repeats.

  • Vectorized scan, modern engines (DuckDB, ClickHouse, Arrow-backed pandas) read pages straight into SIMD-friendly buffers.

A Parquet file is often 5-10× smaller than the equivalent CSV and 10-100× faster to query column-wise.

Encodings#

Each column picks an encoding independently, and the writer chooses based on data profile; low-cardinality strings get dictionary, sorted integers get delta, floats get bit-stream splitting. The encoded bytes then go through a compression codec layered on top.

Per-column choice; the encoder picks based on data profile.

Encoding

Best for

PLAIN

Default fallback; raw values

RLE / Bit-packed

Booleans, low-cardinality columns

DICTIONARY

Strings / categoricals that repeat

DELTA_BINARY_PACKED

Sorted / nearly-sorted integers

DELTA_BYTE_ARRAY

Strings with shared prefixes

BYTE_STREAM_SPLIT

Floats (FP separation for compressors)

Plus a compression codec on top of the encoded bytes.

  • SNAPPY, default; fast, modest ratio.

  • ZSTD, modern; better ratio at similar speed.

  • GZIP, common, slow.

  • LZ4, fast, low ratio.

  • BROTLI, best ratio, slowest.

  • UNCOMPRESSED, when CPU is more expensive than IO.

ZSTD is the 2026 default for new datasets.

Schema and Types#

Parquet’s type system is two layers. Physical types describe the bytes on disk; logical types annotate them with semantic meaning; a BYTE_ARRAY becomes a STRING, an INT64 becomes a TIMESTAMP. Schemas live in the footer, so files are self-describing.

Parquet has its own type system, with logical types layered on top.

Physical type

Use

BOOLEAN

booleans

INT32 / INT64

integers

INT96

legacy timestamp (avoid)

FLOAT / DOUBLE

IEEE-754

BYTE_ARRAY

variable-length bytes / strings

FIXED_LEN_BYTE_ARRAY

decimals, UUIDs, fixed-width binary

Logical types extend these.

  • STRING (UTF-8 BYTE_ARRAY).

  • DECIMAL(p, s).

  • DATE, TIME, TIMESTAMP (with isAdjustedToUTC flag).

  • UUID.

  • ENUM, JSON, BSON.

  • LIST, MAP, STRUCT, nested types via Dremel encoding.

Schemas live in the footer; readers don’t need an external file.

Nested Data: Dremel#

Parquet handles nested / repeated fields with definition and repetition levels (the Dremel paper, Google 2010). Every value gets two small integers describing its position in the nested structure. The encoder can flatten arbitrary JSON-shaped data into columnar storage without losing the hierarchy.

Most users never see this; Arrow / DuckDB / Spark take care of it.

Tooling#

The Parquet ecosystem clusters around Apache Arrow as the in-memory representation, with query engines and CLIs reading and writing through Arrow’s columnar buffers. The list below covers the tools an operator reaches for when inspecting, querying, or producing Parquet on the command line.

  • Apache Arrow, the in-memory columnar layer. pyarrow, arrow-cpp, arrow-rs, and bindings for many languages all read/write Parquet.

  • parquet-tools / parquet-cli, CLI for inspecting files.

  • DuckDB, SQL on Parquet at command line:

    SELECT user_id, SUM(amount)
FROM read_parquet('s3://bucket/orders/*.parquet')
GROUP BY user_id;

  • ClickHouse, DataFusion, Trino, Spark, Polars, query engines.

  • pandas.read_parquet, the everyday Python entry point.

Per-Language Bindings#

Every mainstream language has a Parquet implementation, and most route through Arrow under the hood. The table below is the fast lookup; pick the standard binding for the runtime, drop in the dependency, and reading or writing Parquet is one function call.

Language

Library

Python

pyarrow, fastparquet, pandas, polars

Rust

parquet (Apache Arrow Rust), arrow

Go

apache/arrow-go, segmentio/parquet-go, parsyl/parquet-go (xitongsys archived)

Java

parquet-mr (the original); now also Arrow Java

JavaScript / TypeScript

parquetjs, parquet-wasm, DuckDB-Wasm

C++

arrow-cpp parquet module

Spark / Scala

native; spark.read.parquet(...)

Julia

Parquet.jl

R

arrow, nanoparquet

Where Parquet Lives#

Parquet sits at the storage layer of nearly every modern analytics stack. Data lakes hold the raw files; open table formats add metadata; warehouses query them through external tables; ETL pipelines emit them; ML datasets distribute as them. The list below is a tour of the placements.

  • Object storage data lakes, S3, GCS, Azure Blob, R2, MinIO. Cheap, infinite, columnar files queried by everything.

  • Open table formats, Infrastructure as Code adjacent; Iceberg, Delta Lake, Hudi all store data as Parquet files plus metadata.

  • Data warehouses, BigQuery (via external tables), Snowflake (Iceberg integration), Redshift Spectrum; all read Parquet directly from object storage.

  • ETL output, Spark / Flink / dbt commonly emit Parquet.

  • ML datasets, Hugging Face Datasets defaults to Parquet.

Parquet vs. Other Formats#

A side-by-side with the formats Parquet is most often weighed against. The dividing lines are row vs. column, text vs. binary, single-message vs. table, and storage vs. transport. Most of these formats are not competitors so much as tools for different jobs in the same pipeline.

Format

Notes

CSV

Row-oriented text; universal but slow and verbose.

JSON Lines

Row-oriented text; readable; bigger and slower.

Avro

Row-oriented binary; great for streaming / Kafka.

ORC

Column-oriented; similar capabilities; Hive-native.

Arrow IPC

In-memory columnar; not a long-term storage format.

Feather

Arrow IPC on disk; convenient short-lived format.

ProtoBuf

Single-message binary; not a table format.

For analytical batch storage in 2026, Parquet is the default. Arrow IPC is its in-memory cousin; the two interoperate seamlessly via Arrow.

When to Pick Parquet#

The kinds of workload where Parquet is the obvious answer. Each one has the same root cause: data that is wider than it is point-accessed, and queries that scan a few columns over many rows. If the workload looks like one of the bullets, the default choice is Parquet.

  • Storing tabular data on object storage / data lakes.

  • Datasets that will be scanned column-by-column for analytics.

  • Need for predicate pushdown and column pruning.

  • Long-term archival of analytical history.

  • ML training datasets.

When to Skip#

The mirror image; the cases where Parquet is the wrong tool. Each comes back to one of three frictions: row-shaped access, streaming arrival, or files small enough that the metadata overhead dominates. Reach for Avro, JSON Lines, or a row store instead.

  • Row-oriented workloads (point lookups, transactional updates).

  • Streaming event-by-event data, use Avro / JSON Lines, then consolidate into Parquet on a schedule.

  • Tiny datasets, the file overhead dominates.

  • Human readability, it’s a binary format.

Pitfalls#

The traps that catch teams adopting Parquet for the first time. Most of them surface only at scale; many-small-files and schema-evolution headaches in particular don’t bite a 10-file demo but become first-rank operational problems on a real data lake.

  • Many small files, Parquet’s overhead is per-file. Compact small writes into ~256 MB-1 GB row groups (Iceberg / Delta do this).

  • Schema evolution, adding columns is fine; renaming or reordering breaks readers that match by name. Prefer Iceberg / Delta for managed evolution.

  • Timestamps, INT96 is legacy; new files should use TIMESTAMP(MILLIS) or TIMESTAMP(MICROS). Some old engines read only INT96.

  • Encryption, Parquet Modular Encryption exists; not all readers support it. Plan if compliance requires it.

  • Compression choice, ZSTD almost always wins; if old readers are in the picture, fall back to SNAPPY.

  • Streaming reads, the footer is at the end. Range-getting the footer first then the relevant row groups is the pattern; naïve full-download is wasteful on object storage.

A Few Practical Snippets#

Three concrete entry points, one Python via Arrow, one SQL via DuckDB, one DataFrame via Polars. The same files are readable by all three engines, which is the point of an open columnar format.

Python (pyarrow):

import pyarrow as pa
import pyarrow.parquet as pq

# Write
table = pa.table({"id": [1, 2, 3], "name": ["a", "b", "c"]})
pq.write_table(table, "out.parquet", compression="zstd")

# Read
table = pq.read_table("out.parquet", columns=["id"])
df = table.to_pandas()

DuckDB:

COPY (SELECT * FROM events) TO 'events.parquet' (FORMAT PARQUET);

SELECT user_id, COUNT(*)
FROM read_parquet('events/*.parquet')
WHERE date >= '2026-01-01'
GROUP BY 1;

Polars:

import polars as pl
df = pl.read_parquet("events.parquet")
df.write_parquet("out.parquet", compression="zstd")

Workflow#

Extract, parse, filter, save. duckdb (one binary, SQL over files) is the operator’s daily driver; Python pyarrow / pandas for pipelines.

Extract and inspect.

$ duckdb -c "describe select * from 'events.parquet'"
$ duckdb -c "select count(*) from 'events.parquet'"
$ parquet-tools meta events.parquet                  # if installed

Parse and filter.

$ duckdb -c "select user_id, action, ts
             from 'events.parquet'
             where severity = 'high'
               and ts > timestamp '2026-01-01'"

Save the filtered subset back to Parquet.

$ duckdb -c "copy (
               select * from 'events.parquet' where severity = 'high'
             ) to 'high.parquet' (format parquet, compression zstd)"

Python with pyarrow (predicate pushdown reads only needed pages).

import pyarrow.parquet as pq

tbl = pq.read_table(
    "events.parquet",
    columns=["user_id", "action", "ts", "severity"],
    filters=[("severity", "==", "high")],
)
pq.write_table(tbl, "high.parquet", compression="zstd")

See Also#

  • ProtoBuf, the message-shaped binary format.

  • Analytics, engines that consume Parquet.

  • Storage, where Parquet sits in the storage stack.

Contents