Hashing

Hashing#

A hash is a fixed-size fingerprint of arbitrary-length data. For the operator, hashing is the first integrity check on contact. The operator hashes a downloaded payload to confirm it matches the vendor’s published digest, fingerprints every binary on the target to compare against an indicator-of-compromise list, baselines a directory tree before an engagement so any tampering shows up later, and de-duplicates a collection of pulled artifacts by their content rather than their names.

Hashes are one-way and collision-resistant by design. Same input, same digest, every time. Different input, almost certainly a different digest. That property is what makes a hash a trustworthy fingerprint on contact, and what makes a broken hash family (MD5, SHA-1) dangerous for adversarial work even though it is still fine for casual checksumming.

The sections below walk the anatomy, the GNU coreutils family the operator reaches for first, the broader catalog openssl dgst exposes, the patterns for hashing whole trees, the constraints that bite, and the muscle-memory tasks.

A hash function eats bytes and emits a fixed-size digest. The input can be a file, a pipe, a TCP stream, or the contents of memory. The output is always the same width for a given algorithm.

        flowchart LR
    input["input bytes<br/>(any length)"] --> fn["hash function"]
    fn --> digest["digest<br/>(fixed width)"]
    digest --> hex["hex string<br/>(operator-facing)"]

Algorithm

Digest size

Posture

MD5

128 bit

Broken for adversarial use. Fine as a non-crypto checksum.

SHA-1

160 bit

Broken (SHAttered, 2017). Treat as MD5.

SHA-256

256 bit

Standard for integrity. The operator’s default.

SHA-512

512 bit

Same family, larger digest. Faster than SHA-256 on 64-bit hardware.

BLAKE2b

up to 512 bit

Faster than SHA-2, same security posture. Native b2sum.

BLAKE3

256 bit (extendable)

Faster still, tree-parallel. Separate binary, not in coreutils.

Concretely, md5sum over a file emits a single line of thirty-two hex characters, a space, a mode marker, and the path.

$ md5sum /etc/hostname

b6d81b360a...3e2c  /etc/hostname

sha256sum produces the same format, with sixty-four hex characters instead of thirty-two.

$ sha256sum /etc/hostname

9eec28886f...04efd  /etc/hostname

GNU coreutils hashing#

The GNU coreutils ship a family of small one-job binaries the operator reaches for first. md5sum, sha1sum, sha256sum, sha384sum, sha512sum, and b2sum all share the same flags, the same output format, and the same -c verify mode that reads back a previously written list of digest+path lines. The operator picks the digest by name; everything else is identical.

Hash one file. The default mode the operator runs ten times a day.

$ sha256sum payload.bin

3a9d4e...c102  payload.bin

Verify a downloaded archive against the project’s published digest.

$ echo '3a9d4e...c102  payload.bin' | sha256sum -c -

payload.bin: OK

Hash every regular file in the current directory, write to a manifest.

$ sha256sum * 2>/dev/null > SHA256SUMS

The -c flag is the verify side of the same coin. Hand it a manifest, get a per-file OK / FAILED report and a non-zero exit on any miss. This is the pattern packagers and operators use to detect tampering at rest.

openssl dgst#

openssl dgst exposes the same algorithms plus a wider catalog (SHA-3, RIPEMD, Whirlpool, SM3), HMAC mode for keyed integrity, and detached-signature workflows. Reach for it when the algorithm the operator wants is not in coreutils, when an HMAC is needed instead of a plain hash, or when signing or verifying with a key.

Command

Effect

openssl dgst -sha256 FILE

Plain SHA-256, BSD-style output.

openssl dgst -sha3-256 FILE

SHA-3 family (FIPS 202).

openssl dgst -sha256 -hmac KEY FILE

Keyed HMAC-SHA-256 over the file.

openssl dgst -sha256 -binary FILE | xxd -p

Raw digest piped to hex for inline use.

openssl dgst -sha256 -sign key.pem -out sig FILE

Detached signature over the digest with a private key.

openssl dgst -sha256 -verify pub.pem -signature sig FILE

Verify a detached signature.

Plain digest.

$ openssl dgst -sha256 payload.bin

SHA2-256(payload.bin)= 3a9d4e...c102

Keyed HMAC. A hash alone authenticates content against a publisher’s digest; an HMAC authenticates against a shared secret, so an adversary who can replace the file cannot also replace the expected digest.

$ openssl dgst -sha256 -hmac "$KEY" payload.bin

HMAC-SHA2-256(payload.bin)= 7c12...91a4

Closing pointer. For an MD5/SHA-1/SHA-256 file fingerprint the operator wants every day, stick to coreutils. For everything else, openssl dgst is the broader knife.

Bundled hash tools#

What ships in the box without apt or dnf install. Most Linux distros come with coreutils, openssl, perl, and gnupg out of the box; together they cover every common hash family the operator needs.

Tool

Algorithms

From

md5sum

MD5

coreutils

sha1sum

SHA-1

coreutils

sha224sum / sha256sum / sha384sum / sha512sum

SHA-224, SHA-256, SHA-384, SHA-512

coreutils

b2sum

BLAKE2b

coreutils

cksum

CRC32 (POSIX); -a picks md5/sha/sm3/blake2b on newer builds

coreutils

sum

BSD checksum (legacy compatibility)

coreutils

shasum

SHA-1, SHA-224, SHA-256, SHA-384, SHA-512, SHA-512/224, SHA-512/256

perl (Digest::SHA)

openssl dgst

MD4, MD5, SHA-1, SHA-2, SHA-3, RIPEMD-160, Whirlpool, SM3, BLAKE2; HMAC mode

openssl

gpg --print-md

MD5, SHA-1, SHA-224, SHA-256, SHA-384, SHA-512, RIPEMD-160

gnupg

openssl passwd

sha256crypt, sha512crypt, yescrypt, MD5 crypt (/etc/shadow-style)

openssl

mkpasswd

yescrypt, sha512crypt, bcrypt (/etc/shadow-style)

whois (Debian/Ubuntu) or expect (RHEL)

Common add-ons that live in the default repos but are not always installed on a fresh image.

Tool

Algorithms

Package

xxhsum

XXH32, XXH64, XXH128, XXH3 (fast non-crypto)

xxhash

b3sum

BLAKE3

b3sum / rust-b3sum

rhash

30+ algorithms in one binary (CRC32, MD5, SHA family, BLAKE2, ED2K, Tiger, Whirlpool, GOST)

rhash

hashdeep / md5deep / sha1deep / sha256deep / tiger-deep / whirlpool-deep

Recursive, multi-algorithm forensic hashing

hashdeep (or per-algorithm md5deep)

Two takeaways. The operator who knows only sha256sum and openssl dgst covers 95% of integrity work on any Linux host. When the algorithm is exotic (Tiger, Whirlpool, ED2K, GOST), reach for openssl dgst first; if that does not have it, rhash is the catch-all add-on.

Hashing trees#

A whole-tree hash is two operations. First, walk the tree in a deterministic order and hash every file. Second, hash the list of per-file hashes so the tree as a whole has one digest the operator can compare. find ... -print0 | sort -z | xargs -0 sha256sum gives the per-file pass; piping the result through sha256sum collapses it to a single digest.

Command

Effect

find DIR -type f -print0 \| sort -z \| xargs -0 sha256sum

Per-file digest list, deterministic order.

The above \| sha256sum

One digest summarizing the whole tree.

sha256sum -c TREE_SUMS

Verify the tree against a saved manifest.

rsync -a --checksum src/ dst/

Sync by content hash, not by mtime+size.

Baseline a directory. Run before the engagement, store the manifest off the target, compare on return.

$ find /opt/svc -type f -print0 | sort -z | \
    xargs -0 sha256sum > /tmp/svc.sums

Compare a tree against its baseline.

$ sha256sum -c /tmp/svc.sums --quiet

/opt/svc/bin/agent: FAILED

Reduce a tree to a single digest for at-a-glance comparison.

$ find DIR -type f -print0 | sort -z | xargs -0 sha256sum | sha256sum

The sort step matters. find does not promise an order; without sort the same content can produce different digests on different runs. -print0 and sort -z together keep filenames with spaces and newlines from breaking the pipeline.

Constraints#

  • MD5 and SHA-1 are collision-broken. Use them only as non-adversarial checksums where an attacker has no incentive to forge.

  • Hashes are not MACs. A plain SHA-256 digest authenticates content against a value the operator trusts out-of-band; it does not authenticate the source. Use HMAC or a signature when an adversary can replace both the content and the posted digest.

  • The shell output of sha256sum is hex; the algorithm itself emits bytes. Use -binary (openssl) or convert carefully when scripting against the raw digest.

  • sha256sum -c is line-oriented. Filenames with newlines or backslashes need --strict and the \ escape prefix coreutils writes automatically; do not edit those manifests by hand.

  • On most Linux distros b2sum is present but BLAKE3 is not. Install b3sum separately if BLAKE3 is required.

Common Tasks#

Verify a downloaded archive against a posted digest.

$ echo 'EXPECTED_HEX  payload.tar.gz' | sha256sum -c -

payload.tar.gz: OK

Hash a payload streamed from the network without writing to disk.

$ curl -sL https://example.org/payload | sha256sum

Fingerprint every binary in a tree against an IOC list.

$ find /usr/bin -type f -print0 | xargs -0 sha256sum | \
    awk '{print $1}' | grep -F -f /tmp/ioc.sha256

Find content-duplicate files in a collection.

$ find . -type f -print0 | xargs -0 sha256sum | \
    sort | uniq -d -w 64

Build a baseline of a directory and verify against it later.

$ find /etc -type f -print0 | sort -z | \
    xargs -0 sha256sum > /tmp/etc.sums
$ sha256sum -c /tmp/etc.sums --quiet

Hash only the visible content of a file, ignoring metadata (useful when names or paths have changed).

$ sha256sum < file.bin

Compare two trees by content without copying anything.

$ rsync -an --checksum --itemize-changes src/ dst/ | head

HMAC a payload with a shared secret (authenticated integrity).

$ openssl dgst -sha256 -hmac "$KEY" payload.bin

Roll up a whole directory to one digest.

$ find DIR -type f -print0 | sort -z | xargs -0 sha256sum | sha256sum

Verify a detached signature over a digest.

$ openssl dgst -sha256 -verify pub.pem -signature payload.sig payload.bin

Verified OK

References#

Contents