Logging

The logging module is the standard way to leave a trail an operator can follow when the code is running in production without a debugger attached. print is for ad-hoc local shell work; everything that ships emits log records through logging. The module is in the standard library, requires no dependencies, and integrates with every observability stack in use.

Anatomy

Four objects do the work. A logger is the callsite; a handler decides where the record goes; a formatter turns the record into a string; a filter drops records that fail a predicate. Records flow upward through a tree of loggers (by dotted name) until either a handler accepts them or the root logger is reached.

        flowchart LR
    A["logger.info(...)"] --> B["Logger 'app.api'"]
    B --> C{"propagate?"}
    C -->|yes| D["Logger 'app'"]
    D --> E["Root logger"]
    B --> F["Handler(s)"]
    D --> G["Handler(s)"]
    E --> H["Handler(s)"]
    F --> I["Formatter"]
    I --> J["stderr / file / socket / queue"]
    

Component	Role
Logger	Named callsite. logging.getLogger(__name__) is the standard pattern; the dotted hierarchy mirrors the import graph.
Handler	Sink. StreamHandler for stderr, FileHandler for disk, RotatingFileHandler for rotation, SysLogHandler for syslog, QueueHandler for async dispatch.
Formatter	Renders a LogRecord to a string. Plain text by default; JSON via python-json-logger or a custom formatter.
Filter	Drops records that fail a predicate. Attached to a logger or a handler. Rarely the right tool; usually the level is what the operator wants.
LogRecord	The runtime object. Carries the message, level, logger name, thread, process, exception info, and any extra fields the caller attached.

Levels

Six numeric levels; only the top five are commonly used. Setting a logger or handler to WARNING drops every record below that level.

Level	Value	When
NOTSET	0	Inherit from parent; the default on a new logger.
DEBUG	10	Detail for tracing flow. Off in production.
INFO	20	Routine progress. On in production.
WARNING	30	Something unexpected, but the operation continued. Default level on the root logger.
ERROR	40	The operation failed. Caller-recoverable.
CRITICAL	50	The process is in trouble. Pageable.

The pattern at the top of every module.

import logging
log = logging.getLogger(__name__)

def scan(host: str) -> None:
    log.info("scan start: %s", host)
    try:
        ...
    except Exception:
        log.exception("scan failed: %s", host)   # logs + traceback
        raise

log.exception is log.error plus the active traceback; call it from inside an except block.

Configuring

Three ways to configure logging. Pick one per process; never mix.

basicConfig

The one-liner. Useful for scripts and tests.

import logging, sys
logging.basicConfig(
    level=logging.INFO,
    format="%(asctime)s %(levelname)-8s %(name)s: %(message)s",
    datefmt="%Y-%m-%dT%H:%M:%S%z",
    stream=sys.stderr,
)

It is a no-op if the root logger already has handlers; safe to call at the top of main.

dictConfig

The production form. A dict (often loaded from YAML or TOML) configures every logger, handler, formatter, and filter in one declaration.

import logging.config

CONFIG = {
    "version": 1,
    "disable_existing_loggers": False,
    "formatters": {
        "plain": {
            "format": "%(asctime)s %(levelname)-8s %(name)s: %(message)s",
            "datefmt": "%Y-%m-%dT%H:%M:%S%z",
        },
        "json": {
            "()": "pythonjsonlogger.jsonlogger.JsonFormatter",
            "fmt": "%(asctime)s %(levelname)s %(name)s %(message)s",
        },
    },
    "handlers": {
        "stderr": {
            "class": "logging.StreamHandler",
            "level": "INFO",
            "formatter": "plain",
            "stream": "ext://sys.stderr",
        },
        "file": {
            "class": "logging.handlers.RotatingFileHandler",
            "level": "DEBUG",
            "formatter": "json",
            "filename": "/var/log/op/app.log",
            "maxBytes": 10_000_000,
            "backupCount": 5,
        },
    },
    "loggers": {
        "":         {"level": "INFO",  "handlers": ["stderr", "file"]},
        "app":      {"level": "DEBUG", "propagate": True},
        "httpx":    {"level": "WARNING"},
        "urllib3":  {"level": "WARNING"},
    },
}

logging.config.dictConfig(CONFIG)

The empty-string logger ("") is the root; quiet noisy third-party loggers there.

Programmatic

Build the tree by hand. Reach for this in libraries that ship a default but let the application override.

import logging, sys

handler = logging.StreamHandler(sys.stderr)
handler.setFormatter(logging.Formatter(
    "%(asctime)s %(levelname)s %(name)s: %(message)s"
))

root = logging.getLogger()
root.setLevel(logging.INFO)
root.addHandler(handler)

Handlers

The stdlib ships handlers for every common sink. Pick the right one rather than building over StreamHandler.

Handler	Use
StreamHandler	File-like object (stderr, stdout, io.StringIO).
FileHandler	Append-only file. No rotation.
RotatingFileHandler	File with size-based rotation (maxBytes + backupCount).
TimedRotatingFileHandler	File with time-based rotation (when="midnight", "H", "D").
SysLogHandler	Local or remote syslog daemon (UDP / TCP / Unix socket).
SMTPHandler	Email alerts. Rarely the right answer; alerting belongs in the metrics pipeline.
HTTPHandler	POST records to an HTTP endpoint. Synchronous; risky on the request path.
QueueHandler / QueueListener	Decouple the log call from the slow handler; the queue owns the I/O on a background thread.
NullHandler	Discard. Add to library loggers so the host application owns the configuration.

Custom handlers

Subclass logging.Handler and implement emit. Always guard with self.handleError(record) so a broken handler never raises out of the logging path.

import logging
import http.client
import json

class SlackHandler(logging.Handler):
    def __init__(self, webhook: str, level=logging.WARNING):
        super().__init__(level=level)
        self.webhook = webhook

    def emit(self, record: logging.LogRecord) -> None:
        try:
            body = json.dumps({"text": self.format(record)})
            conn = http.client.HTTPSConnection("hooks.slack.com")
            conn.request("POST", self.webhook,
                         body=body,
                         headers={"Content-Type": "application/json"})
            conn.getresponse().read()
        except Exception:
            self.handleError(record)

slack = SlackHandler("/services/T0/B0/XX", level=logging.ERROR)
slack.setFormatter(logging.Formatter("[%(levelname)s] %(name)s: %(message)s"))
logging.getLogger().addHandler(slack)

For network sinks always wrap with QueueHandler / QueueListener so a slow Slack call does not stall the request path.

Formatters

A formatter renders a LogRecord into a string. The default format string covers most fields; the most common attributes are in the table below.

Attribute	Value
%(asctime)s	Human-readable timestamp; datefmt controls the format.
%(levelname)s	String level (INFO, ERROR).
%(name)s	Logger name (the dotted callsite).
%(message)s	The interpolated message body.
%(filename)s	Source filename.
%(lineno)d	Source line number.
%(funcName)s	Calling function name.
%(process)d	PID.
%(thread)d	Thread ID; useful with %(threadName)s.
%(exc_info)s	Traceback if log.exception was used.

Structured JSON

Aggregators (Loki, Splunk, ELK, Datadog) prefer JSON over delimited text. Two paths.

• python-json-logger — drop-in formatter, no code changes.

• structlog — composable processor chain; binds context values to every record without manual extra= plumbing.

import logging
from pythonjsonlogger import jsonlogger

handler = logging.StreamHandler()
handler.setFormatter(jsonlogger.JsonFormatter(
    "%(asctime)s %(levelname)s %(name)s %(message)s"
))
logging.getLogger().addHandler(handler)
logging.getLogger().setLevel(logging.INFO)

log = logging.getLogger("app")
log.info("scan complete", extra={"target": "1.2.3.4", "ms": 412})

Output.

{"asctime": "2026-05-19T14:35:00", "levelname": "INFO", "name": "app",
 "message": "scan complete", "target": "1.2.3.4", "ms": 412}

The extra= keyword adds fields to the record without interpolating them into the message.

Context

Attaching context (request id, target, run id) to every record without threading it through call signatures.

LoggerAdapter

Wrap a logger to inject fixed context.

log = logging.LoggerAdapter(
    logging.getLogger("scan"),
    {"target": "1.2.3.4", "run_id": "r-9f3"},
)
log.info("opening connection")
# → fields target and run_id appear on every record

contextvars + filter

For async or threaded code where context is per-task, use a contextvars.ContextVar and a filter that copies its value onto the record.

import contextvars, logging

request_id: contextvars.ContextVar[str] = contextvars.ContextVar("request_id", default="-")

class RequestIdFilter(logging.Filter):
    def filter(self, record: logging.LogRecord) -> bool:
        record.request_id = request_id.get()
        return True

logging.getLogger().addFilter(RequestIdFilter())

Add %(request_id)s to the format string; every record now carries the active value.

structlog

structlog makes context binding first-class: log = log.bind(target="1.2.3.4") returns a new logger with the binding attached. Recommended for any project that emits structured logs as a matter of course.

Files

Path	Purpose
/var/log/<service>/*.log	Conventional service log directory on Linux.
~/.cache/<tool>/<tool>.log	User-mode CLI tools that log to the user cache.
/dev/log	Local syslog Unix socket; SysLogHandler default.
logging.yaml / logging.toml	External configuration file consumed by dictConfig.

Common Tasks

Configure logging from a YAML file at startup.

import logging.config, yaml, pathlib
cfg = yaml.safe_load(pathlib.Path("logging.yaml").read_text())
logging.config.dictConfig(cfg)

Silence a noisy third-party logger.

logging.getLogger("urllib3").setLevel(logging.WARNING)
logging.getLogger("botocore").setLevel(logging.WARNING)

Log a traceback for a caught exception.

try:
    risky()
except Exception:
    log.exception("risky() failed")    # level=ERROR + traceback

Route DEBUG and INFO to stdout, WARNING+ to stderr.

stdout = logging.StreamHandler(sys.stdout)
stdout.addFilter(lambda r: r.levelno < logging.WARNING)
stderr = logging.StreamHandler(sys.stderr)
stderr.setLevel(logging.WARNING)
for h in (stdout, stderr):
    h.setFormatter(fmt)
    root.addHandler(h)

Rotate a log at 10 MB, keep 5 backups.

from logging.handlers import RotatingFileHandler
h = RotatingFileHandler("/var/log/op/app.log",
                        maxBytes=10_000_000, backupCount=5)

Capture log records in a test.

def test_warns_on_empty(caplog):
    with caplog.at_level(logging.WARNING):
        run_with_empty()
    assert "empty input" in caplog.text

Pitfalls

• Don’t use the root logger directly in library code. logging.getLogger(__name__) lets the host configure noise per-module.

• Don’t ``basicConfig`` twice. It’s a no-op when handlers exist; call it once in main or use force=True.

• Don’t ``%`` interpolate the message yourself. Pass arguments to the logger (log.info("hit %s", url)); the formatter skips the interpolation when the level is filtered out.

• Don’t share file handlers across processes. The RotatingFileHandler is not multi-process safe; rotation races. Use concurrent-log-handler or a separate writer process behind QueueHandler.

• Don’t log secrets. Tokens, keys, and personal data in log lines outlive the process. Strip at the formatter or the filter.

References

• Errors for exception handling and raise from.

• Testing for caplog and log assertions in pytest.

• Libraries for structlog and python-json-logger placement.

• logging — Logging facility for Python.

• logging.handlers.

• Logging HOWTO.

• structlog.

• python-json-logger.