Testing

Tests are the only reliable mechanism for keeping software working as it changes. The challenge isn’t writing tests; it’s writing the right tests and running them at the right times.

The Test Pyramid

A heuristic, not a law. The pyramid says you should have many fast unit tests at the base, fewer integration tests in the middle, and very few end-to-end tests at the top; inverting that ratio (the “ice cream cone”) produces flaky, slow CI that nobody trusts. From bottom to top.

1. Unit tests, one function or class, no I/O, milliseconds each.

2. Integration tests, a few components together, often with a real database or HTTP server, hundreds of milliseconds each.

3. End-to-end tests, the whole system, real browser / real environment, seconds to minutes each.

The pyramid pushes back on the inverse “ice cream cone” anti-pattern (many slow E2E tests, few unit tests) which produces flaky, slow CI.

Unit Tests

Test one unit of behavior in isolation. The ones at the base of the pyramid, thousands per second on a laptop, deterministic, targeted at one behavior. Most pyramid imbalances are caused by not writing enough of these.

• Fast: thousands per second on a laptop.

• Deterministic: no time, no network, no filesystem (or sandboxed filesystems).

• Targeted: one test per behavior, descriptive name.

• Few external doubles: prefer real collaborators within the unit; mock only at the boundary.

Common smells.

• Mocks of everything -> the test exercises the mocks, not the code.

• Tests of implementation details -> brittle to refactoring.

• Setup that’s longer than the assertion -> the unit is too big.

Integration Tests

A few real pieces working together. The middle of the pyramid. slower than unit tests, but exercise the wiring that unit tests mock away. Per-test setup is the bottleneck; use shared fixtures, parallelize, and isolate state per test to keep them fast. Typical patterns.

• The application + a real database (Testcontainers, embedded Postgres).

• The application + a stubbed external HTTP service (WireMock, MSW).

• Cross-module logic that unit tests can’t cover honestly.

Per-test setup is the bottleneck; share fixtures, parallelize, isolate state per test.

End-to-End Tests

Drive the whole system from outside. The top of the pyramid. expensive to maintain, prone to flake, slow to run, but the only tests that prove the deployed system actually works. Keep the count small and focus on the critical user journeys.

• Browser-based: Playwright, Cypress, Selenium.

• API-based: real HTTP calls against the deployed system.

• Mobile: Detox, Maestro, Appium.

Keep the count small; cover the critical user journeys, not every button. E2E tests are expensive to maintain and prone to flake; use retries with exponential backoff and quarantine flaky tests fast.

Contract Tests

Verify that a service’s outputs match what its consumers expect, without running the consumers. Especially valuable in microservices, where the cost of full E2E tests across N services explodes; contract tests catch the same drift much earlier and much cheaper.

• Pact, consumer-driven contracts.

• Schema-based: OpenAPI / JSON Schema / Avro / Protobuf with compatibility rules.

Especially valuable in microservices, where E2E across N services explodes in cost.

Property-Based Tests

Generate inputs from a specification and assert properties hold across all of them. Strong at finding edge cases that example-based tests miss, especially in parsers, codecs, math, and state machines. Mainstream libraries by language.

• Hypothesis (Python).

• proptest / quickcheck (Rust).

• fast-check (TypeScript).

• gopter (Go).

• Hedgehog (Haskell, Scala, F#, …).

Strong at finding edge cases that example-based tests miss, especially in parsers, codecs, math.

Mutation Testing

Inject small changes (“mutants”) into the code and verify tests catch them. If a mutant survives, the tests don’t actually cover the code despite what coverage reports show. Slow to run; valuable on critical code paths where coverage quality matters more than coverage quantity.

Tools: Stryker (JS / TS / .NET), PIT (Java), mutmut (Python), cargo-mutants (Rust).

Slow; valuable on critical code paths, less useful as a default.

Performance Tests

The four kinds of load test, each answering a different question. Load tests check the SLO at expected traffic; stress tests find the breaking point; soak tests catch leaks; spike tests verify autoscaling. Run regularly and capture results as time series so regressions are visible.

• Load, expected traffic; does the system meet SLOs?

• Stress, beyond expected; where does it break?

• Soak, sustained load over hours; do leaks or queues build up?

• Spike, sudden bursts; does autoscaling react?

Tools: k6, Gatling, JMeter, Locust, Artillery.

Run regularly, not just before launch. Capture results as time series so regressions are visible.

Security Tests

Five categories of security check that run alongside functional testing. SAST and dependency scanning go in CI; DAST runs against staged deploys; IaC and secret scanning catch the infrastructure / config side. Each is cheap individually – gaps tend to be in coverage and follow-through, not tooling.

• SAST, static analysis (Semgrep, CodeQL, SonarQube, Bandit).

• DAST, dynamic analysis against the running app (ZAP, Burp).

• Dependency scanning, known-CVE scanning (Snyk, Dependabot, Trivy).

• IaC scanning, Checkov, tfsec.

• Secret scanning, gitleaks, trufflehog.

See vuln-management.

Accessibility Tests

Automated linters catch the easy stuff, missing alt text, contrast violations, unlabeled inputs. Real users with assistive tech find the rest. Both layers matter; neither is sufficient on its own.

• axe-core, the standard automated accessibility engine.

• Lighthouse, end-to-end accessibility audit alongside performance.

• Manual testing: VoiceOver, NVDA, JAWS, Narrator, TalkBack.

Snapshot Tests

Capture a fixed output (HTML, JSON, image) and assert it doesn’t change. Useful for fast regression detection on output that’s expected to be stable; easy to overuse to the point that snapshots become noise instead of signal.

• Useful for fast regression detection on output that’s expected to be stable.

• Easy to overuse, snapshots that always need updating are a smell.

• Tools: insta (Rust), jest --snapshot, vitest snapshots, approvaltests.

Test Data

The four sources of data tests run against. Fixtures and factories cover most needs; anonymized production data is realistic but legally fraught; synthetic data is the safer substitute when production-structure data matters.

• Fixtures, canned test data, version-controlled.

• Factories, programmatically generate test data (factory_boy, factory-bot, fishery).

• Anonymized production data, realistic, but legally fraught; sanitize.

• Synthetic data, generated to match production structure; safer.

Test Doubles

The five flavors of stand-in for a collaborator, named in Gerard Meszaros’s xUnit Test Patterns. Knowing the distinctions lets a code review catch the most common mocking mistakes, pre-programming expectations where a fake would serve the test better.

• Dummy, placeholder; never used.

• Stub, canned answers.

• Spy, records calls.

• Mock, pre-programmed expectations.

• Fake, working implementation, simplified (in-memory DB).

Prefer fakes for collaborators that have meaningful behavior; reserve mocks for true boundaries.

What to Test

The priorities for where tests pay off the most. Pure logic always; integration tests at wiring points; coverage on boundaries and critical paths; and a regression test for every bug fix. The list afterwards is what not to obsess over, which is just as important.

• Pure logic, always.

• Side-effecting code, enough integration tests to catch wiring bugs.

• Boundaries, everywhere data crosses a trust line.

• Critical paths, the user journeys that, if broken, the company notices in 5 minutes.

• Bug fixes, a regression test, every time.

What not to obsess over.

• 100% line coverage. Coverage is a leading indicator at low values (low → bad) and noisy at high (95% vs. 100% says little).

• Tests that mirror the implementation; refactor them when the code structure changes legitimately.

CI Integration

Where each tier of test belongs in the CI pipeline. Fast tests on every push; slower ones at PR time; the slowest (E2E, performance, security) on a schedule or pre-release. The required-to-merge gate set here shapes engineering culture more than any process document.

• Unit tests on every push, fast feedback.

• Integration tests on PR, caught before merge.

• E2E tests, subset on PR; full on a schedule.

• Performance / security tests, nightly or pre-release.

• Required to merge, the right gates set culture more than process documents do.