Pipeline

A pipeline architecture moves data through a sequence of stages, each transforming the data and passing it to the next. Common in data engineering, build systems, image / signal processing, compilers, and shell pipelines.

        flowchart LR
  Source[(Source)] --> Extract[Extract]
  Extract --> Validate[Validate]
  Validate --> Enrich[Enrich]
  Enrich --> Transform[Transform]
  Transform --> Load[Load]
  Load --> Sink[(Warehouse)]
  Validate -->|"bad rows"| DLQ[(Dead Letter)]
    

Structure

source → transform → transform → ... → sink

The structure is simple, data flows through a series of stages, each doing one transformation, each unaware of its neighbors beyond the data contract they share. Stages run sequentially, concurrently, or distributed across machines depending on the pipeline’s needs. Each stage.

• Has a single, well-defined responsibility.

• Reads from its input.

• Writes to its output.

• Doesn’t know about the stages on either side beyond the data contract.

Stages can run sequentially, concurrently, or distributed across machines.

Familiar Examples

The pipeline pattern shows up everywhere; shell commands, build systems, CI, compilers, ETL, stream processing. Recognizing the pattern is what lets you reuse the same toolkit (composability, parallelism, backpressure) across very different domains.

• Unix pipes, grep ERROR app.log | awk '{print $5}' | sort | uniq -c.

• Build systems, compile → link → package → sign.

• CI/CD, checkout → build → test → publish → deploy.

• Compilers, lex → parse → typecheck → optimize → emit.

• Data ETL, extract → transform → load.

• Stream processing, ingest → enrich → aggregate → sink.

Strengths

What pipelines do well. Composability and reuse let stages travel between projects; parallelism and backpressure handle load smoothly; per-stage testability and observability make operating the system tractable.

• Composability, mix and match stages.

• Parallelism, independent stages run concurrently; backpressure controls flow.

• Reusability, the same stage works in different pipelines.

• Testability, each stage is testable in isolation.

• Observability, per-stage metrics fall out naturally.

Costs

What pipelines are bad at. Logic scatters across stages; debugging requires reconstructing context from upstream inputs; schema drift between stages quietly corrupts data; backpressure becomes a real engineering problem the moment any stage falls behind.

• Scattered logic, a single business action spans many stages.

• Debug context, when stage 5 fails, what was the input to stage 1?

• Schema drift, each stage assumes a schema; changing it ripples forward.

• Backpressure is a real concern; if stages can’t keep up, data piles up between them.

Batch vs. Streaming

The two main flavours of data pipeline. Batch operates on finite chunks with minutes-to-hours latency and easy reprocessability; streaming operates on unbounded events with millisecond-to-seconds latency and harder stateful semantics. The boundary blurs in modern systems that offer both modes.

Batch

• Operates on a finite chunk of data (a file, a partition, a day’s worth).

• Latency: minutes to hours.

• Strong reprocessability; run the same job on the same input, get the same output.

• Tools: Apache Spark, Beam (in batch mode), dbt, Airflow.

Streaming

• Operates on an unbounded sequence of events.

• Latency: milliseconds to seconds.

• Stateful operations (windowing, joins) are hard problems.

• Tools: Kafka Streams, Flink, Beam (in streaming mode), ksqlDB, Materialize.

The boundary blurs; modern systems often offer both modes from the same API.

Lambda and Kappa Architectures

Two well-known data-pipeline architectural patterns from the late 2010s. Lambda runs batch and speed pipelines side by side and merges the results; Kappa unifies them into a single streaming pipeline that can be replayed for reprocessing. In 2026, Kappa-flavoured architectures dominate where streaming is mature.

Lambda Architecture (Nathan Marz)

• Batch layer, processes all data, periodically; produces standard results.

• Speed layer, processes recent data quickly; produces approximate up-to-the-minute results.

• Serving layer, merges batch + speed views.

Strengths: combines accuracy with low latency. Costs: two pipelines to build, test, deploy, and keep consistent.

Kappa Architecture (Jay Kreps)

• Single streaming pipeline.

• Reprocessing is just rewinding the stream and replaying.

Strengths: one pipeline. Costs: requires the streaming pipeline to be powerful enough to do all the work; storage costs to keep enough history.

Both have specific use cases. In 2026, Kappa-flavoured architectures dominate when streaming infrastructure is good enough; Lambda still appears in environments where batch tooling is mature and streaming isn’t.

DAG-Based Pipelines

Real pipelines are often DAGs rather than linear chains; a stage fans out to multiple downstream stages, multiple upstream stages converge, branches join. The orchestration tools below model the DAG explicitly, schedule stages, retry on failure, and surface lineage.

• Apache Airflow, schedules and monitors task DAGs.

• Prefect, modern alternative.

• Dagster, asset-centric.

• Argo Workflows, Kubernetes-native.

• Temporal, durable workflow execution; good for long-running multi-step processes.

ETL vs. ELT

• ETL (Extract, Transform, Load), transform before loading into the warehouse.

• ELT (Extract, Load, Transform), load raw data first, then transform with SQL inside the warehouse.

Modern data architectures lean ELT thanks to cloud warehouses with abundant compute. Tools like dbt are designed for it.

Idempotency and Replays

Data pipelines fail; the question is what happens then. The discipline below makes failure recoverable: idempotent stages that handle re-runs cleanly, checkpointing for long stages, and the ability to replay history when a downstream issue demands a backfill.

• Idempotent stages, safe to re-run on the same input.

• Checkpointing, a stage records its progress so it can resume after a crash.

• Exactly-once semantics, often achieved by idempotent writes plus per-event keys; “exactly once” without those is mostly marketing.

• Backfills, the ability to rerun history is a fundamental capability of any data pipeline.

Backpressure

When a downstream stage can’t keep up, the pipeline has to do something other than buffer forever. Unbounded buffering looks fine until the day a slow downstream takes the whole system out. The patterns below cap the buffer instead.

• Bounded queues with overflow policies (drop, reject, oldest-out).

• Reactive streams style (request-N) for explicit demand.

• Rate limiting at the source.

Unbounded buffering looks fine until the day it doesn’t.

Pipeline Design Heuristics

The rules of thumb that separate pipelines that age well from pipelines that turn into a maintenance nightmare. One concern per stage, stable schemas, idempotent by default, observability per stage, fail-loud on bad data, and replayable from any reasonable starting point.

• One concern per stage. Two responsibilities is a smell.

• Stable schemas between stages, with versioning.

• Idempotent by default.

• Observability per stage, input rate, output rate, latency, error rate.

• Fail loud, bad data should fail visibly, not silently corrupt later stages.

• Replayable from any reasonable starting point.

When to Pick a Pipeline

The signals that a pipeline is the right fit. Data with clear sequential transformations, multiple owners with different concerns, parallelism along stage boundaries, and reprocessing as a first-class capability all point toward the pipeline pattern.

• Data flowing through a series of clear transformations.

• Multiple owners of different stages, each with their own concerns.

• Need for parallelism along stage boundaries.

• Reprocessing / backfilling as a first-class capability.

When to Skip

The cases where forcing the pipeline structure adds friction without paying back. Two-stage transformations are just functions; tightly coupled logic is harder to read split across stages; and if even the best streaming option doesn’t meet the latency target, no pipeline tooling will save you.

• The “pipeline” has only two stages and no real concurrency benefit; call them functions.

• The transformations are tightly coupled; splitting them obscures the logic.

• Latency requirements aren’t met by even the best streaming option.