I/O and Pipelines

I/O and Pipelines#

Three byte streams (stdin, stdout, stderr) are the universal interface every Unix program agrees on. Bash’s job is to wire them up. Redirection points one stream at a file or another file descriptor. Pipes wire one program’s stdout into the next program’s stdin. Process substitution turns a command’s output into a path that looks like a file. Get these three right and most shell scripts stop being copy-paste magic and start being readable plumbing.

This page covers the wiring layer. For language-level reading and writing (read, printf, the positional parameters table), see Overview. For the long-running side of subprocesses (backgrounding, job control, wait), see Backgrounding near the bottom.

Streams#

Every process the kernel hands the operator inherits three open file descriptors. Bash assigns them the names below; tools rely on the convention without ever asking.

FD	Name	Used for
`0`	`stdin`	Input the program reads
`1`	`stdout`	Normal output (the result the operator wanted)
`2`	`stderr`	Diagnostics, warnings, errors (a separate channel so they can be filtered without affecting the result)

        flowchart LR
  IN[(stdin · fd 0)] --> P[process]
  P --> OUT[(stdout · fd 1)]
  P --> ERR[(stderr · fd 2)]

stdout is for the answer; stderr is for the commentary. The discipline matters because pipelines consume stdout; a script that prints “log: starting” to stdout pollutes whatever pipeline reads it.

$ echo "answer"           # to stdout
$ echo "log: starting" >&2  # to stderr, stays out of pipelines

Redirection#

Redirection rewires a file descriptor for one command, before the command runs. The operator sits between the command and its arguments. Spaces around the operator are optional; > file and >file parse the same.

Operator	Effect
`> file`	Write stdout to `file`, truncating it first
`>> file`	Append stdout to `file`
`< file`	Read stdin from `file`
`2> file`	Write stderr to `file`, truncating
`2>> file`	Append stderr to `file`
`2>&1`	Merge stderr into stdout (point fd 2 at the same target as fd 1)
`>&2`	Write stdout to stderr (the operator-voice “to stderr” idiom)
`&> file`	Bash shorthand for `> file 2>&1` (both streams to `file`)
`&>> file`	Same, appending
`< /dev/null`	Disconnect stdin (the program reads EOF immediately)
`> /dev/null`	Discard stdout
`2> /dev/null`	Discard stderr (swallow errors; use sparingly)
`n> file` / `n< file`	Redirect arbitrary fd `n`
`n>&m`	Point fd `n` at the same target as fd `m`
`n>&-` / `n<&-`	Close fd `n`
`<<< "text"`	Here-string: `text` becomes stdin
`<<EOF` … `EOF`	Heredoc: everything until `EOF` on its own line is stdin

Order matters. cmd > all.log 2>&1 sends both streams to all.log. cmd 2>&1 > all.log sends stderr to the terminal and only stdout to the file, because the shell processes redirections left to right and 2>&1 is evaluated before > all.log rebinds fd 1.

$ cmd > all.log 2>&1        # both streams → all.log
$ cmd 2>&1 > all.log         # stderr → terminal, stdout → all.log
$ cmd &> all.log             # bash shorthand for the first form

Capture vs discard vs split. The standard moves.

$ cmd                                 # stdout to terminal, stderr to terminal
$ cmd > out.log                       # stdout to file, stderr to terminal
$ cmd 2> err.log                      # stdout to terminal, stderr to file
$ cmd > out.log 2> err.log            # split each to its own file
$ cmd > all.log 2>&1                  # merged, both to one file
$ cmd 2>/dev/null                     # swallow errors
$ cmd >/dev/null 2>&1                 # silence everything

``exec`` persists a redirection for the whole shell. Once exec runs, the redirection applies to every later command in the same shell or script. Useful at the top of a script for logging.

$ exec 3< input.txt              # open fd 3 for reading
$ read -r first <&3              # read one line from fd 3
$ exec 3<&-                       # close fd 3

$ exec >script.log 2>&1           # redirect the rest of the script

Heredocs#

A heredoc uses the next lines of the script as the command’s stdin, up to a delimiter line. Variables and command substitution expand by default; quote the delimiter to turn that off.

$ cat <<EOF
  $ user is $USER
  $ today is $(date +%F)
$ EOF

user is operator
today is 2026-05-15

Quote the delimiter (<<'EOF' or <<"EOF") for a literal, unexpanded body. The standard way to embed config snippets, SQL, JSON, or scripts that the shell must not interpret.

$ cat <<'EOF' > /etc/cron.d/backup
  $ # variables left literal
  $ 0 3 * * * root /usr/local/bin/backup --target=$HOME
$ EOF

<<-EOF (note the dash) strips leading tabs from every line of the body and from the delimiter, so the heredoc can be indented to match the surrounding code. Spaces are not stripped; the indentation must be tabs.

$ if true; then
  $ cat <<-EOF
    $ all tabs in front of these lines
    $ are stripped before bash hands the
    $ heredoc to cat
  $ EOF
$ fi

Here-string (<<<) is the one-line cousin; the operator hands the command a single string as stdin.

$ wc -w <<< "the operator has the conch"
$ grep -E '^[0-9]+$' <<< "$candidate"

Pipes#

A pipeline chains commands so the stdout of each stage feeds the stdin of the next. Bash sets up the kernel pipes, forks one process per stage, and waits for all of them. The pipeline’s exit code defaults to the last stage; set -o pipefail makes it the first non-zero exit.

$ ps -ef | grep sshd | grep -v grep

        flowchart LR
  A[ps -ef] -->|stdout → stdin| B[grep sshd]
  B -->|stdout → stdin| C[grep -v grep]
  C --> OUT[(terminal stdout)]

Exit code of a pipeline. By default, $? is the exit code of the last stage. ${PIPESTATUS[@]} holds every stage’s exit code. set -o pipefail rewrites $? to the rightmost non-zero stage so silent failures stop hiding.

$ false | true; echo "$?"           # 0 — last stage masked the failure
$ false | true; echo "${PIPESTATUS[@]}"   # 1 0

$ set -o pipefail
$ false | true; echo "$?"           # 1 — pipefail picks up the failure

Merge stderr into the pipeline. |& is bash shorthand for 2>&1 |; the next stage sees both streams.

$ build.sh |& tee build.log         # stdout + stderr captured

The subshell trap. Each stage of a pipeline runs in its own subshell, so variables a while or read loop sets on the right of a pipe vanish when the pipe ends. See the While section of Overview for the standard two fixes (process substitution or shopt -s lastpipe).

$ count=0
$ printf 'a\nb\nc\n' | while read -r _; do
  $ count=$((count + 1))
$ done
$ echo "$count"                     # prints 0

Process Substitution#

Process substitution turns a command’s output (or input) into a path that looks like a file. The shell creates a named-pipe endpoint at /dev/fd/N and substitutes that path on the command line. The standard reach for commands that demand file arguments rather than reading stdin.

Form	Effect
`<(cmd)`	`cmd` runs; its stdout appears as a file path the caller can read
`>(cmd)`	`cmd` runs reading from the path the caller writes to

        flowchart LR
  subgraph SUB["<(cmd)"]
    direction TB
    C[cmd] --> FIFO["/dev/fd/N"]
  end
  FIFO --> CALLER[caller reads it as a file]

Compare two pipelines without temp files.

$ diff <(sort a.txt) <(sort b.txt)

Tee to multiple consumers. tee >(...) is the standard fan-out form.

$ build_output |& tee >(grep -E 'ERROR|WARN' > issues.log) \
$                    >(gzip > full.log.gz) \
$                  > /dev/null

Feed a loop without losing variables to a subshell.

$ count=0
$ while read -r _; do count=$((count + 1)); done < <(printf 'a\nb\nc\n')
$ echo "$count"                     # 3

Named Pipes (FIFOs)#

mkfifo creates an on-disk named pipe. Reads block until a writer arrives and vice versa. The standard reach when two independent processes (not a single pipeline) need to talk and the operator wants the kernel to handle buffering and synchronisation.

$ mkfifo /tmp/work.fifo
$ ( while read -r job; do                    # consumer
  $   echo "processing $job"
  $ done < /tmp/work.fifo ) &

$ for j in alpha bravo charlie; do
  $ echo "$j" > /tmp/work.fifo               # producer
$ done

$ rm /tmp/work.fifo

Backgrounding#

Trailing & runs a command in the background; the shell prints [job-id] pid and returns to the prompt immediately. jobs lists background work; wait blocks until the named job (or all of them) finishes; $! is the PID of the most recent backgrounded command.

$ long-task &
$ jobs
$ wait $!                                    # wait for the last one
$ wait                                       # wait for every job

Fan out work across a list and wait for the whole batch.

$ pids=()
$ for h in web01 db01 cache01; do
  $ ssh "$h" 'uptime' &
  $ pids+=($!)
$ done
$ for p in "${pids[@]}"; do wait "$p"; done

Common Tasks#

Tee output to a file while keeping it on the terminal.

$ cmd | tee out.log

Tee both streams to a file.

$ cmd |& tee out.log

Silence stdout but keep errors.

$ cmd > /dev/null

Silence everything.

$ cmd > /dev/null 2>&1

Send stdout to a file, stderr to a different file.

$ cmd > out.log 2> err.log

Send a script’s whole output to a logfile from line one.

$ exec >script.log 2>&1
$ # everything below this line goes to script.log

Read a file line by line without spawning a subshell.

$ while IFS= read -r line; do
  $ echo "$line"
$ done < input.txt

Compare two sorted streams without temp files.

$ diff <(sort a.txt) <(sort b.txt)

Stream both stderr and stdout into ``grep``.

$ make 2>&1 | grep -E 'error|warning'
$ make |& grep -E 'error|warning'             # bash shorthand

Probe a pipeline for which stage failed.

$ set -o pipefail
$ cmd1 | cmd2 | cmd3; echo "${PIPESTATUS[@]}"

References#

man 1 bash (the REDIRECTION and PIPELINES sections cover the operator surface in full).
man 7 fifo, man 2 pipe (kernel-level documentation of pipes and named pipes).
Overview for the language-level reading and writing (read, printf, IFS, the positional parameters).
Patterns for strict-mode and trap idioms that pair with pipeline error handling.
Standard I/O for the platform-level view of standard streams.
Coreutils for the utilities that live on the ends of most pipelines (tee, grep, sort, cut, awk).
Bash Hackers Wiki, Redirection
Greg’s Wiki, BashGuide / InputAndOutput