JSON

JSON#

The format the operator parses on every collection: REST APIs from recon and enrichment services, cloud audit logs, EDR telemetry, container manifests, configuration drop on a deployed asset. JSON (JavaScript Object Notation) is a small, ubiquitous text format, originally a subset of JavaScript, now its own spec (RFC 8259 / ECMA-404) and the default for HTTP APIs, configuration, and inter-process data. Fluency here is mandatory; jq is on the operator’s box.

Types#

JSON has six value types, a deliberately small surface area that makes the format trivial to parse and easy to map to almost any language. The trade-off is that JSON’s primitive types don’t always cover what users want (no dates, no binary, single number type).

  • string, "hello", double-quoted, with backslash escapes.

  • number, 42, 3.14, -1e10. No separate integer type.

  • boolean, true or false.

  • null, the absence of a value.

  • array, [1, "two", null], ordered, heterogeneous.

  • object, {"a": 1, "b": 2}, unordered (in spec) keys to values.

Example#

{
  "id": 42,
  "name": "Ada Lovelace",
  "active": true,
  "tags": ["math", "computing"],
  "address": {
    "city": "London",
    "zip": null
  }
}

Strict Rules#

JSON is unforgiving by design; the strictness is what makes it easy to interoperate across languages, and the source of the most-cited “JSON has no comments” complaints. Any tool that “parses JSON with comments” is parsing a superset, not JSON.

  • Double quotes only on strings; no single quotes.

  • No trailing commas.

  • No comments (the format spec excludes them).

  • Keys must be strings.

  • Numbers are decimal; no leading zeros, no Infinity or NaN.

  • UTF-8 is the recommended encoding.

Tools that “parse JSON with comments” (jsonc, json5) are not parsing JSON; they’re parsing supersets.

JSON Schema#

JSON Schema describes the structure of valid JSON documents. The ecosystem now has mature validators in every major language; many API platforms (OpenAPI 3.1, JSON-LD, AsyncAPI) build on it as the standard “structure of this object” vocabulary.

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "type": "object",
  "required": ["id", "name"],
  "properties": {
    "id":    { "type": "integer", "minimum": 1 },
    "name":  { "type": "string",  "minLength": 1 },
    "tags":  {
      "type": "array",
      "items": { "type": "string" }
    }
  },
  "additionalProperties": false
}

Validators exist in every major language: Ajv (JS), jsonschema (Python), gojsonschema (Go), valico (Rust).

JSON Pointer / JSONPath#

Three query languages for navigating JSON documents. JSON Pointer is the simple slash-path syntax (single value); JSONPath is the XPath-style query language (sets of values); JMESPath is the AWS-flavored alternative used by aws --query.

  • JSON Pointer – /address/city selects the string "London".

  • JSONPath – $.tags[*] selects every tag; XPath-style.

  • JMESPath, alternative query language; used by AWS CLI --query.

Tooling#

The day-to-day JSON CLI tools. jq is the standard for querying and transforming JSON pipelines; gron flattens into greppable assignments; fx is the interactive viewer. Editor support is universal, so most one-off inspection happens in your editor or a browser DevTools panel.

  • jq, the standard CLI for querying and transforming JSON.

    $ curl -fsS https://api.example.com/users | jq '.[] | {id, name}'
$ jq -r '.items[].id' data.json
$ jq '. + {ts: now}' data.json

  • gron, flatten JSON into greppable assignments.

  • fx, interactive JSON viewer.

  • Editor support is universal: format, validate, fold.

Variants and Supersets#

JSON’s strict spec spawned a family of supersets and binary near-equivalents. Most “I want JSON but with comments” needs end up in JSON5 or JSONC; “I want JSON but smaller” goes to CBOR or MessagePack; “I want streaming JSON” goes to NDJSON.

  • JSON5, adds comments, trailing commas, single-quoted strings, unquoted keys. Common in config (.babelrc, some tsconfig variants).

  • JSONC, JSON with comments. Used by VS Code config and TypeScript tsconfig.json.

  • NDJSON / JSON Lines, one JSON object per line. Streaming-friendly.

    {"id": 1, "name": "operator"}
{"id": 2, "name": "alan"}

  • CBOR, binary, semantic-equivalent to JSON; smaller, faster.

  • MessagePack, another compact binary alternative.

  • BSON, MongoDB’s binary JSON variant.

Where JSON Wins#

The use cases that have made JSON the default cross-language data format. Most boil down to “ubiquitous, human-readable, no platform-specific quirks”, which is enough that JSON beats more capable formats almost everywhere.

  • HTTP APIs (REST, JSON-RPC).

  • Browser fetch payloads.

  • Cross-language data interchange.

  • Logs (especially structured logs).

  • Configuration where YAML / TOML aren’t preferred.

Where JSON Loses#

The cases where another format would serve better. Configuration, big numbers, streaming, schema, and binary data are JSON’s known weak spots; each one has a workaround, but the workarounds add up.

  • Configuration, no comments. YAML, TOML, or JSONC handle this better.

  • Number precision, 64-bit integers don’t survive round-trips through JavaScript without care (use strings for IDs).

  • Streaming, not naturally streamable; use NDJSON when needed.

  • Schema evolution, no built-in story; pair with JSON Schema.

  • Binary, always base64, which is cheap to encode but fattens the payload.

Common Pitfalls#

The bugs that come up over and over in production JSON handling. Each one is well-known but easy to miss – JavaScript’s number precision, missing date type, and spec-incompatible NaN behavior catch new teams every year.

  • 64-bit IDs in JavaScript, precision loss past 2^53. Encode as strings.

  • Date / time, not a native type; convention is ISO 8601 strings ("2026-04-26T12:00:00Z").

  • Trailing commas, valid in JSON5 / JSONC, invalid in plain JSON.

  • NaN / Infinity, not in the spec; libraries differ on what they do.

  • Unicode escapes, \uXXXX for BMP only; supplementary characters use surrogate pairs.

Workflow#

Extract, parse, filter, save. jq is the workhorse on the command line; Python json for anything beyond one-liners.

Extract and inspect.

$ jq . events.json | head                  # pretty-print to confirm shape
$ jq 'length' events.json                  # record count (top-level array)
$ wc -l events.ndjson                      # record count (NDJSON)

Parse and filter.

# select records with high severity
$ jq '.[] | select(.severity == "high")' events.json

# project specific fields
$ jq '.[] | {time, source, severity}' events.json

# NDJSON streaming pipeline
$ jq -c 'select(.severity == "high")' events.ndjson > high.ndjson

Save.

$ jq '[.[] | select(.severity == "high")]' events.json > high.json
$ jq -c '.[] | select(.severity == "high")' events.json > high.ndjson

Python when jq is not enough:

import json
from pathlib import Path

events = json.loads(Path("events.json").read_text())
high   = [e for e in events if e["severity"] == "high"]
Path("high.json").write_text(json.dumps(high, indent=2))
Contents