I/O and Pipelines

I/O and Pipelines#

Nushell pipelines carry typed values between Nu commands and bytes between Nu and external programs. The boundary matters: a Nu→Nu pipe never serializes the value; a Nu→external pipe flattens it to text first. This page covers the wiring layer. For records, tables, and lists as data structures, see Structures.

Streams#

Two streams to think about. The value stream carries Nu values (scalars, lists, records, tables) between Nu commands. The byte stream carries raw bytes between Nu and externals.

        flowchart LR
  A[ls] -->|table| B[where]
  B -->|table| C[grep]:::ext
  C -->|bytes| D[wc]:::ext
  D --> OUT[(host)]

  classDef ext fill:#1f2933,stroke:#637381,color:#e4e7eb,stroke-dasharray:3 3

The boundary between Nu and external programs is the moment the data stops being structured. The operator usually wants to either keep everything inside Nu (no externals) or convert explicitly at the boundary.

Convert to bytes for an external program.

ls | to csv | ^grep '\.go,'                  # serialize to CSV first
ls | to text | ^head -5                       # serialize to text

Convert bytes back to structure.

^cat data.json | from json                    # parse the bytes
^uname -a | str trim                          # treat as a string

Redirection#

Nushell has explicit redirection for stdout (out>), stderr (err>), and both (out+err>). The left-hand operand is whatever produces the data; the right-hand operand is a path.

Operator

Effect

out> file

Write stdout to file (overwrite)

out>> file

Append stdout

err> file

Write stderr to file

err>> file

Append stderr

out+err> file

Both streams, overwrite

out+err>> file

Both streams, append

e>|

Pipe stderr to the next stage

o+e>|

Pipe stdout + stderr to the next stage

out> /dev/null

Discard stdout (use platform-appropriate null device)

 
^./build.sh out+err> build.log
^./build.sh err> errors.log out> /dev/null

Nu-side commands return values; redirection mostly applies to external programs (which actually emit text on stdout / stderr). For Nu-side output, the cleaner alternative is save.

ls | to csv | save inventory.csv             # Nu-native write
ls | save --raw output.txt                   # raw bytes

Pipes#

The pipeline operator | chains commands. Each Nu command streams its result into the next; the data stays structured as long as both sides are Nu.

ps | where name =~ nginx | get pid

When the pipeline crosses into an external program, Nu serializes the value to bytes. Coming back, the bytes are treated as text unless the next stage parses them.

ls | to csv | ^grep '\.go,'                  # Nu → external
^cat data.json | from json | get .users      # external → Nu

The same $in placeholder lets a closure refer to the current pipeline value when calling something that does not take pipeline input directly.

"/tmp/x" | (ls $in)
42 | print $"answer: ($in)"

Parallelism#

par-each is the parallel cousin of each. Runs the closure on every input concurrently, capped at the system thread budget.

[web01 db01 cache01] | par-each { |h|
    ssh $h "uptime" | str trim
}

For order preservation, par-each --keep-order. For more explicit control, par-each --threads 8.

open and save#

open reads a file and parses by extension. save writes the value back to disk in the format implied by extension or the preceding to step.

Extension

Behavior of open

.json

Parse as JSON, return record or list

.yaml / .yml

Parse as YAML

.toml

Parse as TOML

.csv / .tsv

Parse as CSV / TSV, return table

.xlsx / .ods

Parse as spreadsheet (with appropriate plugin)

.parquet

Parse as Parquet (with polars plugin)

.sqlite / .db

Open as database, return a connection

.xml

Parse as XML

other

Read as text

 
open users.json | where age > 30
open db.sqlite | query db "select * from users limit 5"
open data.csv | where region == "us-east" | save filtered.csv

--raw bypasses parsing.

open --raw image.png | save copy.png         # byte-copy a file

Strings#

lines splits a byte stream on newlines. str is the namespace for the rest of the verbs.

open input.txt | lines | length              # line count
"a,b,c,d" | split row ","                    # ["a" "b" "c" "d"]
"abc" | str upcase                            # ABC
"hello" | str length                          # 5

Common Tasks#

Filter a table by predicate.

ls | where size > 1MB | sort-by size --reverse

Parse a JSON HTTP response.

http get https://api.example.com/users
    | where age > 30
    | select name email

Save a pipeline result as CSV.

ls | to csv | save inventory.csv

Read CSV, filter, write back.

open users.csv
    | where region == "us-east"
    | save filtered.csv

Pipe a Nu table to an external program.

ls | to text | ^less

Pipe an external command’s output through Nu.

^dmesg | lines | where $it =~ "error"

Capture stderr separately.

^./build.sh err> errors.log

Run work in parallel.

$hosts | par-each --threads 8 { |h| ssh $h uptime }

References#

Contents