I/O

I/O#

Every Python process starts with three open file descriptors connecting it to the world: standard input, standard output, and standard error. Above that, open plus the pathlib module cover regular files; socket and urllib cover the network; json / csv / struct cover the formats the operator parses every day.

This page is the day-to-day reference for the streams the operator reads and writes. For the broader networking surface (HTTP clients, sockets, asyncio loops) see Networking. For concurrency primitives that compose with I/O see Concurrency.

Standard streams#

The three streams you can always count on. stdin for incoming data, stdout for results, stderr for diagnostics. Keep results and diagnostics separate so shell pipelines work.

import sys

for line in sys.stdin:           # read until EOF
    sys.stdout.write(line.upper())

print("oops", file=sys.stderr)
sys.exit(1)                       # exit code 1

print("a", "b", "c", sep=" | ", end="\n")

input() is a thin wrapper that prompts and reads one line. sys.stdin.isatty() distinguishes piped from interactive. sys.stdout.flush() forces output now (line-buffered to a TTY, block-buffered to a pipe by default).

For the shell-side framing of pipes, redirects, and FDs, see Standard I/O.

File I/O#

open returns a file object; combine with with for deterministic cleanup. Modes select read/write/append/exclusive and text/binary.

Mode	Meaning
`r`	Read; file must exist (default)
`w`	Write; truncate or create
`a`	Append; create if needed; writes go to EOF
`x`	Exclusive create; fail if the file exists
`b`	Binary suffix (combines with the above)
`t`	Text suffix (default)
`+`	Read-write (extends `r`, `w`, `a`, `x`)

Read a whole text file into memory.

with open("notes.txt", "r", encoding="utf-8") as f:
    text = f.read()

Stream a text file line-by-line; iterating the file object is cheaper than readlines() for anything large.

with open("notes.txt", "r", encoding="utf-8") as f:
    for line in f:
        handle(line)

Read a fixed-size header from a binary file, then jump to an offset and read to the end.

with open("dump.bin", "rb") as f:
    header  = f.read(16)
    f.seek(0x100)
    payload = f.read()

Write text. f.write takes a string and adds no newline; print(..., file=f) appends the default line terminator.

with open("/tmp/out.txt", "w", encoding="utf-8") as f:
    f.write("hello\n")
    print("line2", file=f)

Always specify encoding=; the default depends on the locale and bites in containers. newline="" matters when round- tripping CSV. Open in binary (rb / wb) when bytes are the operator’s truth.

pathlib#

pathlib is the modern path API. Prefer it over os.path; it composes cleanly with the operator’s other tools.

Compose a path with /.

from pathlib import Path
p = Path("/var/log") / "auth.log"

One-shot read of a small file; the file handle opens and closes inside the call.

text = Path("/var/log/auth.log").read_text(encoding="utf-8")
data = Path("/tmp/dump.bin").read_bytes()

One-shot write.

Path("/tmp/note.txt").write_text("hello\n", encoding="utf-8")
Path("/tmp/dump.bin").write_bytes(payload)

Glob for files matching a pattern.

for cfg in Path("/etc").glob("*.conf"):
    print(cfg.name, cfg.stat().st_size)

Query methods on a path, grouped by purpose.

Call	Returns
`p.exists()`	`True` if any filesystem entry exists at `p`.
`p.is_file()`	`True` if `p` is a regular file.
`p.is_dir()`	`True` if `p` is a directory.
`p.is_symlink()`	`True` if `p` is a symlink (no follow).
`p.name`	Final path component (`"auth.log"`).
`p.stem`	Final component without suffix (`"auth"`).
`p.suffix`	Final extension including the dot (`".log"`).
`p.parent`	Parent directory as a `Path`.
`p.parts`	Tuple of every path component.
`p.absolute()`	Absolute path without resolving symlinks.
`p.resolve()`	Absolute path with symlinks followed.
`p.with_suffix(".json")`	New `Path` with the suffix replaced.

Formatting and parsing#

Standard-library codecs for common formats. Reach for these before adding a dependency.

JSON#

import json
data = json.loads('{"a": 1, "b": [2, 3]}')
text = json.dumps(data, indent=2, sort_keys=True)

with open("config.json") as f:
    cfg = json.load(f)
with open("out.json", "w") as f:
    json.dump(cfg, f, indent=2)

For non-stdlib speed reach for orjson (fastest) or ujson.

CSV#

import csv
with open("hosts.csv", newline="") as f:
    for row in csv.DictReader(f):
        print(row["ip"], row["asn"])

with open("out.csv", "w", newline="") as f:
    w = csv.DictWriter(f, fieldnames=["ip", "asn"])
    w.writeheader()
    w.writerows(rows)

pandas for any non-trivial CSV / TSV / Excel.

TOML / YAML / INI#

import tomllib                       # 3.11+; read-only
with open("pyproject.toml", "rb") as f:
    data = tomllib.load(f)

# YAML needs a dep
import yaml; data = yaml.safe_load(open("config.yml"))

# INI / .conf
import configparser
cp = configparser.ConfigParser()
cp.read("/etc/myapp.conf")

For writing TOML use tomli_w or tomlkit; the stdlib provides reader only.

Binary structs#

import struct
# !  network byte order; H = uint16; I = uint32
src, dst, length = struct.unpack("!HHI", header)
blob = struct.pack("!HHI", 80, 443, 4096)

Base64 / hex / urlsafe#

import base64
base64.b64encode(b"shellcode")
base64.b64decode("YWJj")
base64.urlsafe_b64encode(token)
base64.b85encode(payload)

b"\xde\xad\xbe\xef".hex()                # 'deadbeef'
bytes.fromhex("deadbeef")

References#

Syntax for the lexical layer file operations sit on.
Errors for with / context managers, the cleanup primitive open and friends compose with.
Networking for the network half of I/O (HTTP, sockets, asyncio).
Libraries for io, os, pathlib, shutil, tempfile, json, csv, configparser, base64, struct, pickle.
Standard I/O for the shell-side framing (pipes, redirects, FDs).

I/O

Contents