Data

Contents

Data#

Codec round-trips and one-pass extractions for the standard serialisation formats. json, csv, tomllib, base64, and hashlib cover most needs without leaving the standard library.

Parse a JSON file.

import json, pathlib
obj = json.loads(pathlib.Path("scan.json").read_text())

Pretty-write a JSON file.

pathlib.Path("out.json").write_text(json.dumps(obj, indent=2, sort_keys=True))

Stream JSON Lines (one object per line).

import json
with open("events.ndjson") as f:
    for line in f:
        handle(json.loads(line))

Read a CSV into a list of dicts.

import csv, pathlib
with pathlib.Path("hosts.csv").open(newline="") as f:
    rows = list(csv.DictReader(f))

Write a CSV from a list of dicts.

with pathlib.Path("out.csv").open("w", newline="") as f:
    w = csv.DictWriter(f, fieldnames=["host", "port", "open"])
    w.writeheader()
    w.writerows(rows)

Parse a TOML file (Python 3.11+ stdlib).

import tomllib, pathlib
cfg = tomllib.loads(pathlib.Path("config.toml").read_text())

Base64-encode bytes for safe transport.

import base64
token = base64.urlsafe_b64encode(payload).rstrip(b"=").decode()

Decode a base64 token.

payload = base64.urlsafe_b64decode(token + "=" * (-len(token) % 4))

Hex-encode bytes.

hexstr = payload.hex()

Hash a byte string.

import hashlib
digest = hashlib.sha256(payload).hexdigest()

HMAC for keyed integrity.

import hmac
sig = hmac.new(key, payload, "sha256").hexdigest()

Constant-time compare of two MAC strings.

if hmac.compare_digest(expected, received):
    accept()

References#

  • Snippets for the snippets catalogue.

  • I/O for the deeper formatting and parsing surface.

  • Hashing for the hashing recipes.

Contents