Scripting

A script is the operator’s force multiplier: a short program that automates work the operator would otherwise type by hand. Glue between commands, a pipeline of text transformations, a one-shot action on target. Scripts trade raw performance for reach; they run anywhere the interpreter is installed, with no build step, and stay readable to whoever has to debug them at three in the morning.

The shells section (Shells) covers shells as interactive environments. This section is scripting as a discipline: when the operator writes one, what language to reach for, and the idioms that keep a script safe when it runs unattended on a production host.

When to Script

The decision to script is mostly about cadence and audience. Scripts earn their keep when a task repeats, when somebody else needs to re-run it, and when “the steps Alice ran last week” is the only existing documentation. The flip side is knowing when a problem has outgrown the form:

• Three-times rule: the third time you type the same sequence, capture it. Earlier is premature; later is wasted typing.

• Glue, not engineering: scripts shine when stitching tools together. Once a script grows real domain logic, control flow, and tests, it has outgrown the form, and it’s time to promote it to a proper program.

• Operator-readable: scripts that run on production hosts must be legible by whoever is on call. Clarity over cleverness.

• Reproducible: a script you commit beats a command history someone has to remember.

When not to script:

• Long-running services: write a program, not a script.

• Anything that needs threads, complex error recovery, or rich concurrency. The cost of getting these right in shell exceeds the cost of switching languages.

• Data pipelines beyond awk-scale: reach for Python or a real data tool.

Languages at a Glance

The right language depends on what the script has to do and where it has to run. The table below sketches each contender’s sweet spot; for anything more than a one-liner, picking deliberately saves rework later.

Language	Where it fits	Notes
Bash	System glue, ops, CI	Ubiquitous on Linux. Strict mode (set -euo pipefail) is table stakes. Bashism-vs-POSIX matters when /bin/sh is dash.
POSIX sh	Cross-Unix portability	Lowest common denominator. Test with dash or shellcheck --shell=sh.
Python	Scripts > 100 lines, data, HTTP	Standard library covers most of what you’d reach for. Use argparse or click; ship a shebang and run it.
Lua	Embedded scripting, Neovim, Redis	Tiny interpreter; the language inside other tools.
AWK	Line-oriented text transforms	One-liners for column math, field extraction, simple reports.
sed	In-place edits, stream rewrites	When you need a regex applied across a stream and nothing more.
PowerShell	Windows ops, Azure, cross-platform	Object pipeline, not text. See PowerShell.
jq	JSON pipelines	Filter, transform, and reshape JSON without writing a real program.

Anatomy of a Good Script

The conventions every well-behaved operator script follows. None of them are difficult; together they are the difference between a script that runs cleanly on a fresh host and one that paged somebody at 3 a.m. on its first invocation.

• Shebang: #!/usr/bin/env bash (or python3). Portable across distros.

• Strict mode (shell): set -euo pipefail and IFS=$'\n\t'. Fail fast on errors, undefined vars, and pipeline failures.

• Usage on no args: print a one-screen usage block and exit non-zero.

• Arguments parsed once: getopts or argparse, not ad-hoc $1 $2 plucking.

• Logging to stderr: progress and errors on stderr; only the intended output on stdout. Pipelines depend on it.

• Idempotent where possible: re-running shouldn’t double-apply.

• Cleanup on exit: trap 'rm -f "$tmp"' EXIT for temp files; mktemp for the path itself.

• Linted in CI: shellcheck for shell, ruff for Python. Catch quoting and unset-variable bugs before they bite.

Example: a minimal, safe Bash script skeleton.

Example:

$ #!/usr/bin/env bash
$ set -euo pipefail
$ IFS=$'\n\t'

$ usage() {
  $ cat <<'EOF'
$ Usage: backup.sh <source> <destination>
  $ Copies <source> to <destination> with rsync, preserving attrs.
$ EOF
$ }

$ [[ $# -ne 2 ]] && { usage; exit 2; }

$ src=$1
$ dst=$2
$ tmp=$(mktemp -d)
$ trap 'rm -rf "$tmp"' EXIT

$ rsync -a --delete "$src/" "$dst/" >&2
$ echo "ok"

Example: a Python script with argparse and a clean main.

Example:

#!/usr/bin/env python3
"""Count lines per file under a directory."""
import argparse
import pathlib
import sys


def main(argv: list[str]) -> int:
    p = argparse.ArgumentParser(description=__doc__)
    p.add_argument("root", type=pathlib.Path)
    args = p.parse_args(argv)

    for path in sorted(args.root.rglob("*")):
        if path.is_file():
            n = sum(1 for _ in path.open(errors="ignore"))
            print(f"{n}\t{path}")
    return 0


if __name__ == "__main__":
    sys.exit(main(sys.argv[1:]))

Common Idioms

The patterns that come up so often in operator scripts they’re worth memorizing. Each one is a bug-avoidance shortcut: NUL-safe file iteration, safe line reading, default values, cleanup traps, and atomic file replace all show up in any non-trivial shell script.

Need	Idiom
Loop over files safely	find . -print0 | xargs -0 -n1 -- ... (NUL-delimited; spaces safe).
Read a file line by line	while IFS= read -r line; do ...; done < file.
Default a variable	: "${VAR:=default}".
Trap for cleanup	trap 'rm -f "$tmp"' EXIT.
Run only if available	command -v jq >/dev/null || { echo "need jq" >&2; exit 1; }.
Compose pipelines	set -o pipefail so a failing middle stage propagates.
JSON in / JSON out	curl -s ... | jq '.items[] | select(.active)'.
Atomic file replace	mv "$tmp" "$final" (after writing into $tmp on the same filesystem).
Argument parsing	getopts (POSIX) or argparse (Python).

Safety

Scripts on production systems are operator-class code; they run unattended, often as root, and a quoting bug or unvalidated input can take down the host. Treat them with the same care as any other production code, even when they are only a dozen lines.

• Quote everything: unquoted $var splits on whitespace and globs. shellcheck catches most of these.

• Validate inputs at the boundary: arguments, env vars, stdin. Reject early; never paste them into commands unescaped.

• Never ``eval`` user input: if the answer involves eval, there is almost always a better one.

• Avoid ``curl | sh``: pin a checksum or commit, fetch, verify, then run.

• Least privilege: run as the smallest account that works; drop privileges in the script when possible.

• Secrets stay out of args: arguments are visible in /proc/<pid>/cmdline. Use environment, files, or a secret store.

Testing

Scripts deserve tests proportional to their blast radius. A disposable one-off does not need pytest; a deployment script that will run on every host in production does. The four tools below cover most of the spectrum, plus the cheapest test of all, a --dry-run flag that prints actions without performing them:

• Bats: a TAP-style harness for Bash. Good for shell scripts of any size.

• shellcheck: static linter; mandatory in CI.

• Pytest: for Python scripts; the same rigor as application code.

• Dry-run flag: a --dry-run switch that prints actions without performing them is the cheapest test of all.

• Idempotence test: run twice; second run should be a no-op.

See Testing for the broader testing taxonomy.

References

External material that pays back the time spent reading it. The Wooledge wiki and the Google style guide are the standard references for shell-quality discussions; shellcheck is the linter that catches most of what those documents warn about.

• Greg Wooledge. “Bash Pitfalls.” https://mywiki.wooledge.org/BashPitfalls

• Google Shell Style Guide. https://google.github.io/styleguide/shellguide.html

• shellcheck: shell linter. https://www.shellcheck.net/

• Bash: Bash as a language.

• Python: Python.

• AWK: text-processing one-liners.

• sed: stream editing.

• jq: JSON pipelines.

• Shells: shells as interactive environments.

Scripts

Anatomy of a script. Shebangs, mode bits, idioms that keep it safe when it runs unattended.

Topics shebang, strict mode, usage block, argument parsing, logging.

Scripts

AWK

Line-oriented text transforms. Column math, field extraction, simple reports.

Topics patterns, actions, fields, BEGIN/END, NR/NF.

AWK