Storage

Storage#

Pick storage by access pattern, not by the first option that comes to mind. Different kinds of data want different stores. On a target the storage mix is a map of where the data lives, and the operator reasons about it the same way whether designing collection infrastructure or auditing what a target already runs.

        flowchart TB
  Q{What's the access pattern?}
  Q -->|"Transactional, joins, ACID"| RDB[Relational]
  Q -->|"Single-key get, sub-ms"| KV[Key-Value]
  Q -->|"Self-contained docs"| Doc[Document]
  Q -->|"Wide rows, write-heavy"| WC[Wide-Column]
  Q -->|"Traversal"| Graph[Graph]
  Q -->|"Full-text / similarity"| Search[Search / Vector]
  Q -->|"Big aggregates"| OLAP[Analytical / Warehouse]
  Q -->|"Blobs / files"| Obj[Object Storage]
  Q -->|"Streaming events"| Stream[Streams / Queues]

  RDB   --> Postgres[(PostgreSQL)]
  KV    --> Redis[(Redis / DynamoDB)]
  Doc   --> Mongo[(MongoDB)]
  WC    --> Cass[(Cassandra)]
  Graph --> Neo[(Neo4j)]
  Search --> Elastic[(Elasticsearch / pgvector)]
  OLAP  --> BQ[(BigQuery / Snowflake / ClickHouse / DuckDB)]
  Obj   --> S3[(S3 / GCS / Azure Blob)]
  Stream --> Kafka[(Kafka / Pub-Sub / SQS)]

Object Storage#

An HTTP key/value store for blobs up to terabytes per object. Cheap, durable, and effectively infinite, but not a filesystem. Reach for it for backups, media, logs, build artifacts, and data-lake files. On a target it is the log sink and the staging ground for exfil, and a world-readable bucket is a classic foothold.

Service

Provider

Note

Amazon S3

AWS

The de facto API standard

Cloud Storage

Google

S3-compatible interop

Blob Storage

Azure

Hot, cool, and archive tiers

MinIO

Self-hosted

S3-compatible, on-prem

R2

Cloudflare

No egress fees

B2

Backblaze

Low-cost archive

Patterns:

  • Lifecycle rules transition cold data to infrequent-access and archive tiers.

  • Server-side encryption with default-deny bucket policies and explicit grants.

  • Versioning with MFA delete for irreplaceable data.

Block Storage#

Raw disks attached over the network and formatted with a filesystem. Latency is dominated by the IOPS limit, so size carefully; snapshots are the backup primitive, and an unguarded snapshot hands whoever can reach it a clean copy of the volume.

Service

Provider

Note

EBS

AWS

gp3 and io2 volume types trade IOPS for cost

Persistent Disk

GCP

Balanced, SSD, and extreme tiers

Managed Disks

Azure

Standard, Premium, and Ultra tiers

File Storage#

POSIX filesystems exposed over NFS or SMB. Easy to drop in for legacy apps, with more throughput than object and less than block. A shared mount is a lateral-movement surface, one writable export reaches every host that mounts it.

Service

Provider

Protocol

EFS

AWS

NFS

Filestore

GCP

NFS

Azure Files

Azure

SMB or NFS

Relational Databases#

The default storage tier for transactional workloads, with strong consistency, joins, mature query optimizers, and ACID transactions. It usually holds the crown-jewel data, the store the operator hardens first, and the one an attacker maps first on contact.

Engine

Role

PostgreSQL

General-purpose, extensible, the modern default

MySQL

High read throughput, broad ecosystem, LAMP heritage

MariaDB

MySQL-compatible community fork

SQLite

Embedded, file-per-database, great for edge and read-heavy

Managed flavors handle the operational pieces:

Provider

Managed service

AWS

RDS

AWS

Aurora

GCP

Cloud SQL

GCP

AlloyDB

GCP

Spanner

Azure

Azure Database

Operational concerns:

  • Connection pooling, pgbouncer for Postgres.

  • Replication, async streaming replicas for read-scaling and DR.

  • Migrations, forward-only and online, via Flyway, Liquibase, sqlx, Prisma migrate, or Alembic.

  • Backups, both logical (pg_dump) and physical (PITR and WAL).

NoSQL / Document#

Non-relational stores chosen by access pattern, key-value for sub-millisecond gets, document for self-contained records, wide-column for write-heavy multi-datacenter workloads. The data model a target chose is a tell for how its data is keyed and where it lives.

Store

Model

DynamoDB

Key-value with secondary indexes, serverless, predictable cost

MongoDB

Document store with a rich query language

Cassandra

Wide-column, multi-DC writes, no single primary

ScyllaDB

Wide-column, Cassandra-compatible, low latency

Firestore

Document store (GCP)

Bigtable

Wide-column (GCP)

Caches#

The fastest tier of storage. In-process caches hold the hottest data; a shared store takes over when state crosses process boundaries. A cache also holds live sessions and tokens, a high-value, low-latency target for an attacker who reaches it.

Cache

Note

Redis

In-memory data structures, pub/sub, streams, persistence

Valkey

Open-source Redis fork

Memcached

Pure cache, no persistence

In-process

Fastest tier for the hottest data, no network hop

Invalidation strategies:

  • TTL, simple, eventually correct.

  • Cache-aside, read miss reads the DB then writes the cache.

  • Write-through, write the DB and cache atomically.

  • Write-behind, write the cache and flush to the DB async, with crash-loss risk.

Queues and Streams#

Three messaging patterns worth distinguishing, by retention and consumer count. The broker is a critical dependency and a single point of failure, and its backlog is a window into what the system is doing.

Pattern

Semantics

Example

Queue

One message to one consumer, workers compete

SQS

Stream

Replayable log, many consumers read independently

Kafka

Pub/Sub

Broadcast to every subscriber

NATS

Pick by retention and consumer model:

  • Need replay or multiple independent consumers? Stream or event log.

  • One worker pool drains tasks? Queue.

  • Real-time fanout? Pub/sub.

Analytics / Warehouse#

Columnar stores optimized for analytical queries on large data. The warehouse pools the whole organization’s data into one place, the richest single collection target and the widest blast radius if breached.

Store

Type

BigQuery

Managed columnar warehouse (GCP)

Snowflake

Managed columnar warehouse

Redshift

Managed columnar warehouse (AWS)

Databricks

Lakehouse on the data lake

ClickHouse

Open-source columnar, very fast

DuckDB

Embedded analytical, in-process

Iceberg + Trino

Open table format with a distributed query engine
Contents