AWK

AWK#

AWK is a pattern-action language for line-oriented text. Designed in 1977 by Aho, Weinberger, and Kernighan (the names supply the letters); specified by POSIX; available everywhere.

Unfair characterization: a 200-line awk program is a tiny programming language; a 5-line one is the right tool for half the text problems you’ll encounter on the command line.

The Mental Model#

An AWK program is a list of pattern { action } rules. AWK reads input one record (line) at a time, splits each into fields, and runs each matching rule.

# /pattern/ { action }
/ERROR/ { print }                    # print lines containing ERROR

$1 == "operator"     { print $2, $3 }     # split-by-whitespace then match
$3 > 100        { count++ }
END             { print "found", count, "rows" }

Default Variables#

A small set of automatic variables drives every AWK program. Fields are pre-split into $1..``$N``; counters track the record and field positions; field and record separators configure how lines and columns are recognized. The table below is the operator’s quick reference.

Variable

Meaning

$0

the whole record (line)

$1..``$N``

the Nth field

NF

number of fields in the current record

NR

current record number (line counter)

FNR

record number within the current file

FS

field separator (default whitespace)

OFS

output field separator

RS

record separator (default newline)

ORS

output record separator

FILENAME

current input file

The Three Special Patterns#

  • BEGIN { ... }, runs once before any input.

  • END   { ... }, runs once after all input.

  • Everything else, runs per record where the pattern matches.

awk 'BEGIN { FS=":" } { print $1, $7 }' /etc/passwd

One-Liners#

A starter library of AWK invocations that show up in everyday log analysis and CSV wrangling. Sums, averages, max, deduplication, JSON projection, range filters, and group-counting; each one is a few characters longer than useless and many lines shorter than Python.

$ awk '{ print $2 }' file

$ awk '{ sum += $3 } END { print sum }' file

$ awk '{ sum += $3; n++ } END { print sum/n }' file

$ awk '$3 > max { max = $3 } END { print max }' file

$ awk 'BEGIN { FS="\t"; OFS="," } { print $1, $2, $3 }' file

$ awk 'NF' file

$ awk 'NR >= 5 && NR <= 10' file

$ awk '!seen[$0]++' file

$ awk '{ count[$1]++ } END { for (k in count) print k, count[k] }' file

$ awk -F '\t' 'NR>1 { printf "{\"id\":%s,\"name\":\"%s\"}\n", $1, $2 }' file

Programming Constructs#

AWK is a real programming language. Conditionals, loops, functions, associative arrays.

The ternary cond ? a : b yields one of two values without writing a full if statement.

        flowchart TD
    A[start] --> B{"x < 0?"}
    B -->|true| C["result = -x"]
    B -->|false| D["result = x"]
    C --> Z[return result]
    D --> Z
    
function abs(x) { return x < 0 ? -x : x }

A pattern-action rule fires the action when the pattern matches the current record; AWK loops the rule across every input line itself.

        flowchart TD
    A[next record] --> B{"$1 equals 'purchase'?"}
    B -->|true| C["total[$2] += $3"]
    C --> D["count[$2]++"]
    D --> E[next record]
    B -->|false| E
    
BEGIN { FS = "," }
$1 == "purchase" {
  total[$2] += $3
  count[$2]++
}

The for (k in array) loop walks the keys of an associative array; order is unspecified unless PROCINFO["sorted_in"] is set in gawk.

        flowchart TB
    A[start END block] --> B["it = keys(total)"]
    B --> C{"more keys?"}
    C -->|yes| D["user = next key"]
    D --> E["printf \"%s: ...\""]
    E --> C
    C -->|no| Z[end]
    
END {
  for (user in total) {
    printf "%s: %d orders, $%.2f total\n", user, count[user], total[user]
  }
}

Variants#

A handful of AWK implementations ship across systems, with real performance and feature differences. POSIX awk is the safe target for portable scripts; gawk is the richest; mawk is the fastest; busybox is the smallest. Knowing which one /usr/bin/awk points to is sometimes important.

  • awk, the original; on every system; minimal feature set.

  • nawk, “new awk”, the AT&T descendant.

  • gawk, GNU awk; richest feature set: gensub, lint mode, full Unicode, networking, multiple-precision math.

  • mawk, Mike’s awk; very fast; common as /usr/bin/awk on Debian / Ubuntu.

  • busybox awk, minimal; for embedded systems.

POSIX awk is the safe target; gawk features beyond POSIX get flagged by gawk --posix.

Field-Splitting Subtleties#

How AWK splits a record into fields depends on FS, and the defaults have surprises. Single-space FS is treated specially; single comma does not parse real CSV; tab-separated is exact; regex separators only work in gawk. For real CSV with quoted fields, reach for a CSV-aware tool.

  • FS=" ", the default; splits on any run of whitespace, ignores leading / trailing.

  • FS="\t", exactly one tab.

  • FS=",", exactly one comma; "a,b","c,d" does not parse the way you’d want.

  • FS="[,\t]", regex separator (gawk supports it).

For real CSV with quoted fields, use a CSV-aware tool (mlr, csvkit, Python csv).

Multi-Char Separators / Records#

$ awk 'BEGIN { RS="" } /pattern/' file

$ awk 'BEGIN { RS="^$" } { ... }' file

What AWK Does Best#

  • Filtering and projecting columnar data.

  • Sums, counts, simple aggregations on log files / CSVs.

  • Quick exploratory analysis where loading into pandas is overkill.

  • Building reports from text output.

When the program crosses ~50 lines, switch to Python or another general-purpose language. AWK is the best 5-line tool in your toolbox; it stops being best around 20 lines.

Modern Alternatives#

Tools that solve overlapping problems in friendlier forms. mlr (Miller) is AWK reshaped for CSV / TSV / JSON; dasel queries multi-format files; Polars-CLI and DuckDB-CLI bring SQL to flat files. For structured data, these clear AWK’s quirks; for line-oriented text, AWK still wins.

  • mlr (“miller”), AWK / sed / cut / sort / join for CSV / TSV / JSON / Pandas-DataFrame-shaped data. The spiritual successor for many use cases.

  • dasel, multi-format query (JSON / YAML / XML / TOML / CSV).

  • Polars CLI, SQL on CSV / Parquet at the command line.

  • DuckDB CLI, SQL on files in many formats.

For line-oriented unstructured text, AWK still wins. For structured data, the alternatives are usually clearer.

Pitfalls#

The traps that catch AWK users. Floating-point precision, locale-dependent number parsing, the off-by-one between $0 and $1, shell quoting, and AWK’s regex dialect all bite at different times. Each is a one-line fix once recognized.

  • Floating-point precision in default integer-looking math: 0.1 + 0.2 != 0.3.

  • Locale-dependent number parsing, a comma decimal separator (German / French locales) silently truncates parsed numbers.

  • Field 0 vs. 1, $0 is the whole line, fields start at 1.

  • Quoting in shell, single-quote your AWK programs; double-quoting invites the shell to expand $1 etc.

  • Regex escaping, AWK regex differs from PCRE; \d is not a digit class.

Contents