AWK

AWK#

AWK is the operator’s line-oriented text-transformation language. A program is a list of pattern { action } rules; AWK reads input one record at a time, splits each record into fields, and runs every rule whose pattern matches. The fastest path from “a table of text” to “the operator’s answer”.

For language-level detail see AWK. This page is the operator’s working subset, what to type at the prompt or drop into a pipeline.

Setup#

AWK is part of every POSIX userland. The implementation differs across distros:

Implementation

Detail

gawk

GNU AWK. Default on Debian / Ubuntu / RHEL. Extensions on top of POSIX (associative array tricks, gensub, mktime, networking).

mawk

Small, fast, POSIX-strict. Default on minimal Debian containers.

nawk / awk

One True AWK from Bell Labs. Default on BSDs and macOS.

busybox awk

Default on Alpine and most embedded estates. POSIX subset.

The operator usually does not care which is installed; non-portable GNU extensions are obvious in the code. Pin gawk with which awk when scripts depend on it.

$ awk --version
$ which awk

BEGIN { actions before the first record }
/pattern/ { action on matching lines }
$1 == "foo" { action on field match }
length($0) > 80 { action on computed match }
END { actions after the last record }

  • BEGIN runs once before any input is read.

  • END runs once after the last record.

  • In between, every rule whose pattern matches the current record fires. Multiple rules can fire per record, in source order.

  • $0 is the whole record; $1, $2, … are fields; NF is the field count; NR is the record number.

Records and fields#

By default, records are split on newlines and fields on runs of whitespace. Both separators are configurable.

Variable

Meaning

FS

Input field separator. Set with -F or BEGIN { FS = "," }.

OFS

Output field separator (between fields when printing).

RS

Input record separator. Set to "" for paragraph mode (blank-line separated records).

ORS

Output record separator (after each print).

NR

Current record number across all input.

FNR

Current record number in the current file.

NF

Number of fields in the current record.

FILENAME

Name of the current input file.

 
# CSV: comma-separated
$ awk -F, '{ print $1, $3 }' users.csv

# tab-separated
$ awk -F'\t' '{ print $2 }' table.tsv

# paragraph mode
$ awk 'BEGIN { RS = ""; FS = "\n" } /OPERATOR/ { print }' notes.txt

Patterns#

Patterns gate when an action fires. AWK takes plain regex, field comparisons, range expressions, and the special words BEGIN and END.

# regex pattern
$ awk '/ERROR/' app.log
$ awk '/^[0-9]+$/' input.txt

# field comparison
$ awk '$1 == "operator" { print $2 }' input.txt

# numeric comparison
$ awk '$3 > 100' input.txt

# negation
$ awk '$0 !~ /debug/' app.log

# range (NR-based)
$ awk 'NR == 1, NR == 10' input.txt
$ awk '/START/, /END/' input.txt

# compound
$ awk '$1 == "user" && $3 > 100' input.txt

Common idioms#

The operator’s daily AWK working set.

Idiom

Result

awk 'NR==42'

Print line 42.

awk '$2 ~ /^bob/'

Lines where field 2 starts with bob.

awk '{ print $NF }'

Print the last field ($NF references the count field).

awk '!seen[$0]++'

Stream dedupe (preserves first occurrence; small memory for small data).

awk 'END { print NR }'

Count lines (equivalent to wc -l, sometimes faster).

awk '{ s += $1 } END { print s }'

Sum column 1.

awk '{ s += $1 } END { print s / NR }'

Mean of column 1.

awk '{ print NR, $0 }'

Number every line.

awk 'BEGIN { FS=OFS="," } { $3 = $3 * 1.07 } 1'

Rewrite column 3 of a CSV.

awk '/start/{flag=1; next} /end/{flag=0} flag'

Print lines between markers, exclusive.

awk 'NF'

Drop blank lines.

awk '{ a[$1]++ } END { for (k in a) print a[k], k }'

Frequency count of field 1.

awk -v t="$threshold" '$3 > t'

Pass a shell variable in via -v.

Variables and arithmetic#

Variables are untyped (strings and numbers, with implicit conversion). Arrays are associative. No declarations.

{
  total += $3
  count++
  by_user[$1] += $3
}
END {
  printf "rows: %d  total: %.2f  mean: %.2f\n", count, total, total/count
  for (u in by_user) printf "%-20s %.2f\n", u, by_user[u]
}

Format with printf for column-aligned output; print for quick checks. The format specifiers mirror C.

Built-in functions#

Function

Use

length(s)

String length (or array size).

substr(s, start, len)

Slice. 1-indexed.

split(s, a, sep)

Split s into array a by sep. Returns count.

index(s, sub)

Position of sub in s; 0 if not found.

sub(re, repl, s) / gsub(re, repl, s)

Replace first / all matches of re in s.

match(s, re)

Sets RSTART and RLENGTH.

toupper(s) / tolower(s)

Case conversion.

sprintf(fmt, ...)

Like printf but returns the string.

system(cmd)

Run a shell command. Returns exit status.

getline

Pull the next record. Useful for multi-file or lookahead patterns.

GNU AWK adds strftime, mktime, gensub (group references), asort, networking via /inet/..., and extension loading. Stick to POSIX unless gawk is guaranteed on every target.

Operator examples#

Apache / nginx access-log triage:

# top 20 source IPs
$ awk '{ a[$1]++ } END { for (k in a) print a[k], k }' access.log \
    | sort -rn | head -20

# bytes served per URI
$ awk '$9 == 200 { a[$7] += $10 } END { for (k in a) print a[k], k }' access.log \
    | sort -rn | head

# request rate per minute
$ awk -F'[][]' '{ t=$2; sub(/:..[+ -].*/, "", t); a[t]++ } END { for (k in a) print k, a[k] }' access.log

ss / netstat parsing:

# listening TCP ports, deduped
$ ss -tln | awk 'NR>1 { print $4 }' | awk -F: '{ print $NF }' | sort -u

CSV report on a column:

# mean response time grouped by status
$ awk -F, 'NR>1 { n[$3]++; s[$3]+=$5 } END { for (k in n) printf "%s\t%.1f\n", k, s[k]/n[k] }' app.csv

Compose with find for multi-file rollups:

$ find /var/log -name '*.log' -print0 \
    | xargs -0 awk 'FNR==1 { print "==", FILENAME, "==" } /ERROR/'

Common Tasks#

Sum a column.

$ awk '{ s += $3 } END { print s }' data.tsv

Drop blank lines and comments.

$ awk 'NF && $1 !~ /^#/' config.txt

Strip lines between markers.

$ awk '/BEGIN-SECRET/,/END-SECRET/{next}1' file > sanitised

Pivot a long-form table to wide.

$ awk -F, 'NR>1 { v[$1,$2]=$3; k[$2]=1; r[$1]=1 }
$          END   { for (rk in r) { line=rk; for (ck in k) line=line FS v[rk,ck]; print line } }' long.csv

Rewrite a CSV column in place.

$ awk 'BEGIN { FS=OFS="," } NR>1 { $4 = toupper($4) } 1' users.csv > users.fixed.csv
$ mv users.fixed.csv users.csv

References#

Contents