Libraries

Python ships “batteries included”. The standard library covers most everyday needs; PyPI hosts everything else. This chapter is reference for the stdlib modules and third-party packages worth reaching for, organized by job.

For installing and managing dependencies, see Setup and Tooling. For frameworks (Django, Flask, FastAPI, orchestration, security toolkits), see Frameworks. For networking-specific libraries (sockets, HTTP, DNS, SSH, scapy), see Networking.

Standard library

Filesystem and process

Module	Job
os	Process environment, os.environ, os.getcwd, os.fork, file system calls (use pathlib for paths).
pathlib	The operator’s path API. Path("/etc/passwd") reads, writes, iterates, globs without string-mangling.
shutil	High-level file ops, copytree, rmtree, which, disk_usage.
subprocess	Run external commands. subprocess.run is the modern default; never use shell=True on untrusted input.
tempfile	NamedTemporaryFile, TemporaryDirectory (use as context managers).
glob / fnmatch	Shell-style globbing.

Data formats

Module	Job
json	JSON encode / decode. loads / dumps.
csv	Read / write CSV with DictReader / DictWriter.
configparser	Read INI files.
tomllib (3.11+)	Read TOML (pyproject.toml etc.). Read-only; write with tomli-w if needed.
xml.etree.ElementTree	Parse XML. For untrusted XML use defusedxml.
html.parser	Pull-style HTML parser. For real work use beautifulsoup4 or lxml.
base64 / binascii	Encoding primitives.
struct	Fixed-layout binary records.

Text and dates

Module	Job
re	Regular expressions. compile once if reused.
string	String constants, Template, Formatter.
textwrap	Wrap and dedent.
unicodedata	Normalization, category lookup.
difflib	SequenceMatcher, get_close_matches.
datetime	Dates, times, deltas, timezone-aware via zoneinfo.
zoneinfo (3.9+)	IANA timezones (stdlib successor to pytz).
time	Low-level monotonic / perf counters, sleep.

Collections and algorithms

Module	Job
collections	deque, Counter, defaultdict, ChainMap, OrderedDict, namedtuple.
itertools	Lazy combinators, chain, islice, groupby, product, combinations, cycle, count.
functools	cache / lru_cache, partial, reduce, wraps, cached_property.
heapq	Binary min-heap on a list. heappush, heappop, nlargest, nsmallest.
bisect	Binary search and ordered-list insertion.
graphlib (3.9+)	TopologicalSorter for dependency ordering.
operator	itemgetter, attrgetter, methodcaller (faster than lambdas for keys).
contextlib	contextmanager, suppress, ExitStack, closing.

Concurrency and async

Module	Job
threading	Threads. GIL-bound; use for I/O-bound concurrency only.
multiprocessing	Processes. Use for CPU-bound work. Pool, Queue.
concurrent.futures	High-level ThreadPoolExecutor and ProcessPoolExecutor.
asyncio	Async event loop, coroutines, tasks, queues, locks.
queue	Thread-safe Queue, LifoQueue, PriorityQueue.

Networking (stdlib)

The deeper detail lives in Networking.

Module	Job
socket	TCP / UDP / Unix sockets. The operator’s primitive.
ssl	TLS wrap for sockets.
urllib.request / urllib.parse	HTTP and URL handling (use httpx for real work).
http.client / http.server	Low-level HTTP, ad-hoc dev server.
ipaddress	Parse and manipulate IPv4 / IPv6 addresses and networks.

Crypto and hashing

Module	Job
hashlib	Cryptographic hashes (SHA-2, SHA-3, BLAKE2). Streaming updates.
hmac	Keyed-hash MAC; compare_digest for constant-time equality.
secrets	Cryptographically strong random for tokens, passwords, URL-safe IDs. Replaces naive random for security.
ssl	TLS contexts, certificate verification.
uuid	UUID v1, v3, v4, v5.

Persistence

Module	Job
sqlite3	Embedded SQL. Zero-config. The operator’s local store.
shelve	dict-like persistent storage on disk.
pickle	Object serialisation. Never unpickle untrusted data.
copy	Shallow and deep copies.

Observability and tools

Module	Job
logging	Standard logger. Operator preference, configure once and use logging.getLogger(__name__) per module.
argparse	Command-line parsing.
warnings	Issue deprecation / runtime warnings.
traceback	Format / print tracebacks programmatically.
dis	Bytecode disassembler (operator’s introspection tool).
inspect	Introspect functions, classes, signatures.
typing	Type hints (Protocol, TypeVar, Literal, Annotated).
dataclasses	Plain-record class generator.

Third-party core

The packages every operator project tends to use. Only the stable, actively maintained, broadly adopted ones; deprecated and superseded packages are dropped from the catalog.

HTTP and networking

Library	Purpose
httpx	sync + async HTTP. Modern default for new code.
requests	sync HTTP. The classic; still rock-solid for one-off scripts.
aiohttp	async HTTP client + server.
websockets	RFC 6455 client and server.

CLI and UX

Library	Purpose
typer	CLI from type hints. The default for new CLIs.
click	Declarative CLI. What typer builds on.
rich	Colour, tables, progress bars, tracebacks for terminal output.
prompt-toolkit	Interactive shells with completion, history, mouse.

Data

Library	Purpose
pydantic	Runtime validation from type hints. The default for parsing JSON into models.
attrs	Dataclasses with more flexibility (validators, slots, converters).
polars	Fast columnar dataframe. The default for single-host data work over pandas in 2026.
sqlalchemy	ORM and query builder. The default RDBMS abstraction.
alembic	Schema migrations for sqlalchemy.

Logging and observability

Library	Purpose
structlog	Structured logging. Default for new services.
loguru	Zero-config logging for scripts.
sentry-sdk	Error reporting + performance traces to Sentry.
opentelemetry-api	Vendor-neutral traces and metrics.

Resilience

Library	Purpose
tenacity	Retry decorators with backoff. The modern default.

Testing

Library	Purpose
pytest	The test framework. Fixtures, parametrize, plugins.
hypothesis	Property-based testing.
coverage	Line and branch coverage.

OSINT libraries

API clients and enumeration tools for open-source intelligence.

API clients

Library	Purpose
shodan	Python client for Shodan internet-wide search.
censys-python	Python client for Censys.
dnspython	DNS queries (also covered in Networking).
tweepy	Twitter / X API client.
praw	Reddit API client.

Enumeration

Library	Purpose
python-whois	Domain WHOIS lookups.
sherlock	Username search across hundreds of sites; CLI and library.
phonenumbers	Parse, validate, geocode phone numbers.
geopy	Geocoding clients for Nominatim, Google, Mapbox.

Scraping primitives

Library	Purpose
beautifulsoup4	Permissive HTML parser. The default for messy real-world pages.
lxml	Fast XML / HTML parser. XPath plus XSLT; backend for many other tools.
trafilatura	Article-body extraction from arbitrary web pages. Modern, maintained successor to newspaper3k.
feedparser	RSS / Atom feed parsing.

Cryptography

Library	Purpose
cryptography	The modern Python crypto library. hazmat layer for primitives, recipes layer for Fernet. Default for new work.
pycryptodome	Drop-in for the abandoned pycrypto. Older API; reach for it only when inheriting code that uses it.
passlib	Password hashing with multiple backends (bcrypt, scrypt, argon2).
argon2-cffi	Argon2 password hashing (the default for new deployments).

Don’t roll your own crypto. Use library primitives, never raw hashlib for password storage.

Security linting

The operator’s audit toolkit for both source code and installed packages.

Tool	Catches
bandit	Python-specific security smells (eval, pickle, yaml.load, hardcoded secrets, weak random, SSL verify-off).
semgrep --config=p/python	Same surface plus framework-aware rules (Django, Flask, FastAPI).
pip-audit / safety	Vulnerable installed packages (CVE feed).
trivy fs . / grype	Same, lockfile-aware; container-aware.
ruff check --select=S	Bandit rules ported to ruff (fast).

References

• Setup and Tooling for installing these libraries.

• Networking for the network-specific libraries (sockets, HTTP, DNS, SSH, scapy).

• Frameworks for the framework-class libraries (Django, FastAPI, pandas, Celery, MCP).

• The Python Standard Library, the reference documentation.

• PyPI, the package index.

• Awesome Python, a curated list of Python libraries by topic.