Standard I/O

The Unix idea is that each program does one thing well. Programs are composed like Lego blocks using stdin, stdout, and stderr as the connectors between them.

Streams

Every process starts with three open file descriptors that connect it to the world: standard input for incoming data, standard output for results, and standard error for diagnostics. Keeping output and errors on separate streams is what lets shell pipelines work without mixing the two:

FD	Name	Meaning
0	stdin	Input stream the process reads from
1	stdout	Normal output stream the process writes to
2	stderr	Error / diagnostic output stream

Every block has the same three connectors: input on the left, normal output on the right, errors out the bottom.

        flowchart LR
  IN["stdin (0)"] --> P[program]
  P --> OUT["stdout (1)"]
  P --> ERR["stderr (2)"]
    

The kernel exposes each stream as a file under /dev, and a shell’s open file descriptors are visible under /proc/$$/fd.

By default echo writes to FD 1 (stdout):

$ echo "to stdout"

to stdout

>&2 redirects the command’s stdout onto FD 2; the message arrives via stderr instead:

$ echo "to stderr" >&2

to stderr

1>&2 is the explicit form of “send my FD 1 to FD 2”; useful inside scripts that already redirect things:

$ printf "error: %s\n" "boom" 1>&2

error: boom

read is the Bash builtin that pulls one line from stdin into a variable:

$ read line

(waits silently for input; press Enter to finish)

cat echoes stdin back to stdout until EOF (Ctrl-D):

$ cat

hello
hello
(Ctrl-D ends input)

< FILE feeds a file as stdin instead of the keyboard:

$ cat < /etc/os-release

PRETTY_NAME="Ubuntu 24.04 LTS"
NAME="Ubuntu"
VERSION_ID="24.04"

The kernel exposes each FD as a file under /dev; writing to /dev/stdout is identical to writing to FD 1:

$ echo hello > /dev/stdout

hello

/dev/stderr is the explicit equivalent of >&2:

$ echo oops > /dev/stderr

oops

< /dev/stdin paired with <<< (here-string) sends a single string in via FD 0:

$ wc -l < /dev/stdin <<< "one two three"

1

/proc/$$/fd exposes the current shell’s open file descriptors as symlinks:

$ ls -l /proc/$$/fd

lrwx------ 1 operator operator 64 May  1 14:38 0 -> /dev/pts/0
lrwx------ 1 operator operator 64 May  1 14:38 1 -> /dev/pts/0
lrwx------ 1 operator operator 64 May  1 14:38 2 -> /dev/pts/0

Redirects

A redirect (>, <, 2>) re-points one of a block’s connectors to or from a file instead of a terminal or another block. The shell performs the redirect before the command starts, which is why cmd > out 2>&1 and cmd 2>&1 > out produce different results; order matters.

Form	Effect
cmd > out.txt	Truncate, write stdout to file
cmd >> out.txt	Append stdout to file
cmd 2> err.txt	Send stderr to a file
cmd > out 2>&1 / &> out	Both streams to the same file
cmd < in.txt	Feed file as stdin
cmd <<EOF / EOF	Heredoc: multi-line stdin

Redirecting stdout to a file leaves stderr on the terminal (cmd > out.txt):

        flowchart LR
    cmd["cmd"] -->|stdout| out["out.txt"]
    cmd -->|stderr| term["terminal"]
    

Redirecting stderr to a file leaves stdout on the terminal (cmd 2> err.txt).

        flowchart LR
    cmd["cmd"] -->|stdout| term["terminal"]
    cmd -->|stderr| err["err.txt"]
    

Merging stderr into stdout sends both to the same place (cmd > out.txt 2>&1).

        flowchart LR
    cmd["cmd"] -->|stdout| out["out.txt"]
    cmd -->|"stderr (2>&1)"| out
    

2> captures stderr to a file; stdout still flows to the terminal (or wherever > would send it):

$ ls /no/such/dir 2> errors.log

(no terminal output; errors.log now holds the error message)

2>&1 | tee merges stderr into the pipe and tee writes the merged stream to both screen and file:

$ make 2>&1 | tee build.log

gcc -O2 -o app main.c
main.c: warning: implicit declaration of function 'foo'
(also written verbatim to build.log)

> /dev/null 2>&1 discards both streams, the standard “I just want the exit code” pattern:

$ command > /dev/null 2>&1

(no output; check ``$?`` for the exit code)

A heredoc (<<EOF) sends a multi-line block as stdin until the matching delimiter, with normal expansion in scope:

$ cat <<EOF > /tmp/note.txt
$ line 1
$ line 2 (today is $(date +%F))
$ EOF

(no output; /tmp/note.txt now contains the two lines)

Split the streams by sending each FD to its own file:

$ ls /etc /no/such > out.log 2> err.log

(no terminal output; out.log holds /etc, err.log holds the error)

Merge stderr into the pipe so grep sees both streams:

$ ls /etc /no/such 2>&1 | grep -i 'no'

ls: cannot access '/no/such': No such file or directory

Order matters: this puts both streams in all.log because the merge happens before the file redirect:

$ ls /etc /no/such > all.log 2>&1

(no output; all.log has both stdout and stderr)

This duplicates FD 2 onto the terminal’s FD 1, then redirects stdout to the file; stderr stays on the terminal:

$ ls /etc /no/such 2>&1 > only-stdout.log

ls: cannot access '/no/such': No such file or directory
(only-stdout.log has the /etc listing)

Pipes

A pipe (|) snaps two blocks together by connecting the stdout of the first to the stdin of the second. Both processes run concurrently and data flows through a kernel buffer between them.

Form	Effect
a | b	Send a’s stdout to b’s stdin
a 2>&1 | b	Merge a’s stderr into the pipe
a |& b	Bash shorthand for a 2>&1 | b
a | tee f | b	Tap the pipe to a file without breaking it

Two blocks snapped together with a pipe:

        flowchart LR
    a["a"] -->|stdout| b["b"]
    b -->|stdout| out["stdout"]
    a -->|stderr| ea["stderr"]
    b -->|stderr| eb["stderr"]
    

By default only stdout flows through the pipe. Each block’s stderr still drops out the bottom to the terminal unless explicitly merged.

Filter then re-filter; the second grep removes the line about grep itself:

$ ps -ef | grep sshd | grep -v grep

root      873     1  0 09:12 ?        00:00:00 /usr/sbin/sshd -D
sshd    19842   873  0 14:02 ?        00:00:00 sshd: operator [priv]

Pipe an HTTP response into jq to extract one field:

$ curl -s https://api.example.com/x | jq '.items[0]'

{
  "id": 4711,
  "name": "alpha"
}

The classic “top repeated lines” pipeline (sort → count adjacent duplicates → re-sort numerically → take the head):

$ sort access.log | uniq -c | sort -rn | head

4821 GET /index.html 200
1043 GET /favicon.ico 200
 902 GET /api/v1/items 200

Extract the first colon-separated field of every passwd entry:

$ cut -d: -f1 /etc/passwd | sort

_apt
bin
daemon
nobody
operator
postgres
root
sshd

Composition

Streams, redirects, and pipes are the connectors. Composition is what you build with them. The following pipeline retrieves the process ID of the running bash shell by snapping three blocks together:

        flowchart LR
    ps["ps"] -->|stdout| grep["grep bash"]
    grep -->|stdout| awk["awk '{print $1}'"]
    awk -->|stdout| out["stdout"]
    ps -->|stderr| e1["stderr"]
    grep -->|stderr| e2["stderr"]
    awk -->|stderr| e3["stderr"]
    

$ ps | grep bash | awk '{print $1}'
$ 18259

The returned value 18259 is the process ID of the bash shell we are running. Walking the pipeline one block at a time:

The ps block lists running processes on stdout:

        flowchart LR
    ps["ps"] -->|stdout| out["stdout"]
    ps -->|stderr| err["stderr"]
    

$ ps
  $ PID TTY          TIME CMD
$ 17811 pts/0    00:00:00 zsh
$ 18259 pts/0    00:00:00 bash
$ 18883 pts/0    00:00:00 ps

The grep block reads stdin and writes only the lines matching its pattern. Snapping it onto ps connects ps’s stdout to grep’s stdin:

        flowchart LR
    ps["ps"] -->|stdout| grep["grep bash"]
    grep -->|stdout| out["stdout"]
    ps -->|stderr| e1["stderr"]
    grep -->|stderr| e2["stderr"]
    

$ ps | grep bash
$ 18259 pts/0    00:00:00 bash

The awk block reads stdin and prints field $1 (the first column, the PID). Snapping it onto the end of the pipeline strips everything but the PID:

        flowchart LR
    ps["ps"] -->|stdout| grep["grep bash"]
    grep -->|stdout| awk["awk '{print $1}'"]
    awk -->|stdout| out["stdout"]
    ps -->|stderr| e1["stderr"]
    grep -->|stderr| e2["stderr"]
    awk -->|stderr| e3["stderr"]
    

$ ps | grep bash | awk '{print $1}'
$ 18259

Filters

These small programs end up in 90% of useful pipelines. sort orders lines, uniq collapses adjacent duplicates (so it almost always follows sort), wc counts, cut and awk slice fields, and xargs turns a stream of items on stdin into arguments to another command.

Command	Effect
sort / sort -u	Sort lines / sort + unique
uniq -c	Count adjacent duplicates (after sort)
wc -l	Count lines
cut -d, -f1	Pick fields by delimiter
awk '{print $1}'	Field-oriented mini-language
tr 'a-z' 'A-Z'	Translate / squeeze characters
xargs CMD	Build a command line from stdin items

A filter is just another block in the chain:

        flowchart LR
    src["source"] --> filt["filter"]
    filt --> sink["sink"]
    

awk -F: parses /etc/passwd colon-fields; the test $3 >= 1000 keeps only human users:

$ awk -F: '$3 >= 1000 {print $1}' /etc/passwd

nobody
operator
alice

wc -l counts lines in each named file:

$ wc -l *.py

 42 main.py
108 utils.py
 31 cli.py
181 total

tr translates one character set to another; here it replaces spaces with newlines, splitting words to lines:

$ echo "alpha bravo charlie" | tr ' ' '\n'

alpha
bravo
charlie

find -print0 emits NUL-separated paths; xargs -0 parses that NUL-separated stream, the safe pattern for filenames with spaces or newlines:

$ find . -name '*.log' -print0 | xargs -0 wc -l

4821 ./var/log/syslog
 902 ./var/log/auth.log
5723 total

Common Tasks

Capture command output to a file (stdout, stderr, both).

$ command > out.log 2>&1
$ command 2> err.log
$ command &> all.log

Tee output (log to disk and watch live).

$ command 2>&1 | tee run.log
$ command 2>&1 | tee -a run.log
$ command | tee >(grep ERROR > errs.log) >/dev/null

Process-substitute for diffs and merges (compare outputs without temp files).

$ diff <(ls /etc) <(ls /etc.bak)
$ comm -23 <(sort a.txt) <(sort b.txt)

Pipe to remote (exfil-ready output channels, within scope).

$ tar -cz /var/log | ssh user@host 'cat > logs.tgz'
$ command | ssh user@host 'cat >> /tmp/feed.log'
$ command | nc -N host 9000

Use named pipes for live coupling (decouple producer and consumer).

$ mkfifo /tmp/feed
$ producer > /tmp/feed &
$ consumer < /tmp/feed

Heredoc / herestring (inline content without temp files).

$ ssh host 'bash -s' <<'EOF'
id; hostname; uptime
EOF

$ grep -F 'pattern' <<<"$VAR"

Suppress output cleanly (when only the exit code matters).

$ command >/dev/null 2>&1
$ command &>/dev/null

Stream-process line by line (the Unix idiom for ad-hoc data).

$ tail -F app.log | grep --line-buffered ERROR | tee err.log
$ awk '/ERROR/ {print $1, $NF}' app.log
$ jq -c '.events[] | select(.level=="error")' input.json
$ ss -tnp | awk 'NR>1 && $1=="ESTAB" {print $5}' | sort -u

Pass stdin to a command that wants a file (the - and /dev/stdin idioms).

$ curl -s https://example.com/data.json | jq . -
$ echo "$payload" | openssl dgst -sha256
$ tar -xzf - < archive.tgz
$ diff <(echo "$a") <(echo "$b")

Show progress on a slow pipe (when a long task gives no output).

$ pv big.iso | sudo dd of=/dev/sdb bs=4M conv=fsync
$ tar -cf - /var/log | pv -s $(du -sb /var/log | awk '{print $1}') | gzip > logs.tgz

Buffer or unbuffer line output (pipes hide output until exit).

$ stdbuf -oL -eL <command> | tee live.log
$ unbuffer <command> | grep something      # from expect
$ script -fq -c 'long-cmd' run.log

Record an interactive session (replayable timing log).

$ script --timing=run.tim run.log
# ... run commands ...
$ scriptreplay --timing=run.tim run.log

Concatenate, split, transform (the pipeline building blocks).

$ paste -d, ids.txt names.txt
$ split -l 1000 big.csv chunk_
$ cut -d: -f1,3 /etc/passwd | sort -t: -k2 -n
$ tr -s '[:space:]' '\n' < file | sort | uniq -c | sort -rn | head

References

• man 1 bash (REDIRECTION section covers >, <, 2>, &>, heredocs).

• man 2 pipe, man 2 dup2, man 7 fifo (kernel pipe and FIFO primitives).

• man 1 tee, man 1 xargs, man 1 awk, man 1 sed, man 1 cut, man 1 tr, man 1 sort, man 1 uniq, man 1 wc (filter toolkit).

• man 1 stdbuf, man 1 unbuffer, man 1 script, man 1 scriptreplay, man 1 pv (buffering and recording helpers).

• The Terminal for quoting, expansion, and job control around these pipelines.

• Linux for the command quick-reference.