Archives

Archives#

Compress to save space, decompress to read. Some tools compress a single stream (gzip, bzip2, xz, zstd); others bundle a tree first and then compress it (tar with -z, -j, -J, or --zstd). Pick the algorithm by speed and ratio for the workload.

Single-file compressors#

Tool

Suffix

Speed / Ratio

Common use

gzip / gunzip

.gz

fast / modest

ubiquitous default

bzip2 / bunzip2

.bz2

slow / better

legacy archives

xz / unxz

.xz

slow / best

source tarballs

zstd / unzstd

.zst

fast / good

modern default

lz4

.lz4

very fast / lower

high-throughput

brotli

.br

slow / text-best

web assets

Warning

xz (CVE-2024-3094). Upstream xz / liblzma 5.6.0 and 5.6.1 shipped a backdoor that hooked OpenSSH authentication on distros that linked sshd against liblzma. Confirm xz --version is 5.4.x or >= 5.6.2 before trusting any build of xz on a target.

Warning

gzip (CVE-2022-1271). zgrep and xzgrep on maliciously named .gz / .xz files could write to arbitrary paths. Patched in gzip 1.12 and xz-utils 5.2.6; older distro images are still in the wild.

Warning

bzip2 (CVE-2019-12900). Out-of-bounds write in the decoder when handling crafted .bz2 data. Fixed in 1.0.8.

$ gzip file
$ gzip -k file
$ gunzip file.gz
$ gzip -9 file

$ zstd file
$ zstd -d file.zst
$ zstd --long=27 -19 huge.bin

$ xz file
$ xz -d file.xz
$ xz -T0 file

$ bzip2 file
$ bunzip2 file.bz2

Read directly through a compressor#

Format

cat

less

grep

.gz

zcat

zless

zgrep PATTERN file.gz

.bz2

bzcat

bzless

bzgrep

.xz

xzcat

xzless

xzgrep

.zst

zstdcat

zstdgrep

Archives (multi-file containers)#

Tool

Purpose

tar

the universal Unix archive; combine with a compressor (see below)

zip / unzip

cross-platform, native Windows compatibility

7z

multi-format with strong compression

cpio

legacy archive format; survives in initramfs and RPM

Warning

unzip (Zip Slip / path traversal). Multiple CVEs over the years (CVE-2018-1000035, CVE-2022-0529, CVE-2022-0530) plus the recurring Zip Slip class where archive entries with ../ escape the extraction directory. Always extract untrusted archives into a sandbox and verify entry paths before extraction (unzip -l, zipinfo).

Warning

p7zip is largely unmaintained. The Linux p7zip fork has not seen a meaningful upstream release since 2016; CVE-2022-29072 went unpatched there. Prefer the actively maintained 7zip package on modern distros, or extract 7z archives via bsdtar / libarchive.

Warning

cpio is largely abandoned. GNU cpio gets only sporadic security fixes; it survives mainly as the initramfs and RPM payload format. Treat untrusted .cpio archives the same way you treat untrusted tarballs (sandbox, list before extract).

tar is the workhorse. It bundles a directory tree, then pipes the stream through a compressor.

$ tar -czf out.tgz dir/
$ tar -cjf out.tbz dir/
$ tar -cJf out.txz dir/
$ tar --zstd -cf out.tar.zst dir/

$ tar -xzf out.tgz
$ tar -xf out.tar.zst
$ tar -tf out.tgz
$ tar -czf - dir/ | ssh host 'tar -xzf - -C /target'

Filesystem-level compression#

Filesystem

How to enable

Btrfs

mount -o compress=zstd:3 (or compress=zlib / compress=lzo); per-file via chattr +c FILE

ZFS

zfs set compression=zstd POOL/DATASET (default in modern OpenZFS)

F2FS

mount -o compress_algorithm=zstd; per-directory via chattr +c

Pipelines#

$ tar -czf - dir/ | ssh host 'tar -xzf - -C /target'

$ tar -cf - dir/ | pigz -p 8 > dir.tgz
$ tar -cf - dir/ | xz -T0     > dir.txz
$ tar --use-compress-program='zstd -T0' -cf dir.tar.zst dir/

$ zgrep ERROR /var/log/syslog.*.gz

Example:

# Compare ratios on the same input
$ cp /var/log/syslog sample
$ for c in gzip bzip2 xz zstd; do cp sample sample.$c; $c -k sample.$c; done
$ ls -lS sample.* | awk '{print $5, $9}'
1832104 sample.gz
1457392 sample.xz
1798416 sample.zst
1620008 sample.bz2

# Bundle a directory and stream it to a remote host (no temp file)
$ tar -cf - /var/log | zstd -T0 -19 | ssh host 'cat > logs.tar.zst'

# Search compressed logs without decompressing first
$ zgrep -h 'OOM' /var/log/syslog.*.gz | wc -l

# Per-file Btrfs compression on an existing tree
$ sudo chattr +c -R /mnt/data/archive
$ sudo btrfs filesystem defragment -r -czstd /mnt/data/archive

# Receive a mission package, edit it, repack it
$ ls -l mission.tar.gz
$ tar -xzf mission.tar.gz                        # unpack into ./mission/
$ rm mission/secrets.txt                         # drop the file we don't want shipped
$ tar -czf mission.tar.gz mission/               # repack (overwrites the original)
$ tar -tzf mission.tar.gz | head                 # verify contents

References#

  • man 1 gzip, man 1 bzip2, man 1 xz, man 1 zstd, man 1 lz4, man 1 brotli (single-stream compressors).

  • man 1 zcat, man 1 bzcat, man 1 xzcat, man 1 zstdcat, man 1 zgrep, man 1 xzgrep, man 1 zstdgrep (read through a compressor).

  • man 1 tar, man 1 zip, man 1 unzip, man 1 7z, man 1 cpio (multi-file archives).

  • man 5 btrfs, man 8 mount.zfs, man 5 f2fs (filesystem-level compression).

  • man 1 chattr, man 8 btrfs-filesystem (per-file compression controls).

  • Encryption for the encrypt-then-compress order and at-rest encryption.

  • Disks for backup, snapshot, and image workflows.

  • Files for sparse-file and atomic-write patterns that interact with archive tooling.

  • xz utils

  • zstd

  • 7-Zip

Contents