DevSecOps

DevSecOps#

DevSecOps folds security into the same loop as development and operations. Instead of a separate review gate at the end of the pipeline, security is a property of every commit, every build, every deploy, every running workload. For the operator, this is the defended side of the same automation that ships features; the tools and practices below are how a team keeps the blast radius small without slowing the ship rate to zero.

The discipline grew out of two pressures, the shift to continuous delivery (which made end-of-cycle reviews impossible) and the rise of software supply chain attacks (which made the build pipeline itself a target). The answer was to push security checks left into the developer’s editor and pipeline, and to push security telemetry right into production runtime.

Shift Left#

The slogan that defined the movement. Find defects earlier on the timeline, where the cost of fixing is orders of magnitude lower than in production. The pipeline is the natural place to enforce this, every stage gets the security check appropriate to the artifact it produces.

Stage	Artifact	Security check
Editor	Source under edit	Linters, IDE plugins, pre-commit hooks.
Commit	Git diff	Secret scanning, license headers, branch protection.
Build	Source tree	SAST, dependency (SCA) scan, license compliance.
Package	Container image, JAR, binary	Image scan, SBOM generation, signing.
Deploy	Manifest, Terraform plan	IaC scan, policy as code, admission control.
Runtime	Running workload	Falco / eBPF detection, RASP, anomaly detection.

The lower the stage, the cheaper the fix, and the smaller the audit surface that ever has to defend it.

Shift Right#

The complement to shift left. Some classes of risk only surface under real traffic, real data, and real adversaries; the operator needs telemetry from production to catch them. Shift-right work includes runtime detection, chaos and resilience drills, red-team exercises, bug bounties, and continuous attack surface monitoring.

A mature programme runs both. Shift left keeps the easy classes out; shift right catches the rest before the adversary does.

Pipeline Scanners#

The core class of tools that wraps the build. Each looks at a different artifact and finds a different class of defect; the operator wires them in as pipeline stages with fail conditions tied to severity.

SAST#

Static Application Security Testing. Reads source code without running it, finds vulnerable patterns (SQL injection, command injection, insecure deserialisation, hardcoded secrets, weak crypto). Best at code-level defects in code the team owns.

Semgrep, rule-based, fast, supports custom rules in YAML.
CodeQL (GitHub), query the codebase as data.
SonarQube, broad coverage, established in enterprise pipelines.
Bandit (Python), gosec (Go), Brakeman (Rails), language- specific scanners that ship cheap wins.

SCA#

Software Composition Analysis. Walks the dependency tree and matches versions against vulnerability databases (NVD, GHSA, OSV). Best at catching known CVEs in third-party libraries, which is where most modern breaches live.

Dependabot (GitHub), automated PRs to update vulnerable deps.
Renovate, self-hosted alternative, more configurable.
Trivy, scans dependencies, containers, IaC, and SBOMs.
Snyk, commercial; broad language coverage.
OSV-Scanner, Google-backed, OSV database.

DAST#

Dynamic Application Security Testing. Black-box probe against a running instance; sends crafted requests and observes responses. Catches classes SAST misses (auth flaws, broken access control, business-logic issues).

OWASP ZAP, the workhorse open-source scanner.
Burp Suite, paid; the manual-tester’s tool, with CI plugins.
Nuclei, template-based; covers known-CVE proofs of concept.

IaC Scanning#

Catches misconfigured Terraform, CloudFormation, Kubernetes manifests, and Dockerfiles before they ship. Open S3 buckets, public security groups, privileged containers, missing encryption, all detectable from the manifest alone.

Checkov, broad coverage across Terraform, K8s, Cloudformation, Dockerfile.
tfsec, Terraform-focused.
KICS, multi-format scanner.
Trivy config, the same Trivy as above, with IaC rules.

Container Scanning#

Inspects the layered image filesystem for vulnerable packages, embedded secrets, and bad configuration. Run on every push to the registry; fail the build above a severity threshold.

Trivy, the default open-source pick.
Grype (Anchore), pairs with Syft for SBOM generation.
Clair, registry-side scanner; integrates with Quay and Harbor.

Secret Scanning#

Finds credentials, tokens, and keys committed to the repo, often historically. Run as a pre-commit hook and as a pipeline stage, hooks catch most of it, the pipeline stage covers anyone who bypassed the hook.

gitleaks, fast, configurable patterns.
trufflehog, history-aware; can verify whether a found token still works.
detect-secrets (Yelp), baselines and audit workflow.

Each scanner is a stage in the pipeline. Severity thresholds wire into pass / fail conditions; a high-severity finding either blocks the merge or files a ticket with an SLA, depending on policy.

Threat Modeling#

The design-time discipline. Before code is written, the team walks through the proposed change asking what an attacker would target, which trust boundaries the change crosses, and what mitigations exist or are missing. STRIDE (Spoofing, Tampering, Repudiation, Information disclosure, Denial of service, Elevation of privilege) is the standard checklist; LINDDUN extends it for privacy.

For the operator, threat modeling is the cheapest control on the list. An hour at design time prevents a class of defect from ever reaching the scanners. Lightweight versions (a 30-minute whiteboard session per feature, a one-page doc per service) tend to stick; heavyweight processes do not.

Supply Chain#

Modern attackers do not write zero-days when they can corrupt a build. Supply chain security treats the path from source to running artifact as a target in its own right, with controls covering authorship, provenance, dependencies, and the build environment.

SBOM#

Software Bill of Materials. A machine-readable inventory of every component (and version) that went into an artifact. Two competing formats, SPDX and CycloneDX, both produced by tools like Syft, Trivy, and language-native plugins.

The SBOM is the input to a later question, “I just learned about CVE-X, which of my running artifacts are affected?”. Without an SBOM, the answer is a frantic grep.

SLSA#

Supply-chain Levels for Software Artifacts. A framework that ranks build-system integrity from L0 (no controls) to L3 (hardened, non-falsifiable provenance). Targets the class of attack where the source is clean but the build server was compromised.

L1, scripted build with provenance.
L2, signed provenance, hosted build service.
L3, hardened build, non-falsifiable provenance.

Signing#

Provenance is only useful if the operator can verify it. Sigstore (cosign, rekor, fulcio) provides keyless signing of containers, binaries, and SBOMs using short-lived certificates and a public transparency log. in-toto attests the full build process, not just the final artifact.

Dependency Hygiene#

Pin versions and lockfiles; rebuilds should be byte-identical.
Vendor or mirror critical dependencies so an upstream takedown does not break the build.
Restrict the dependency set; every transitive dep is a new surface.
Watch for typosquats and dependency confusion (private package names that shadow public ones).

Policy as Code#

The control plane for “what is allowed to ship”. Express security and compliance rules as code, evaluate them against manifests and provenance, fail the pipeline when a rule trips.

OPA (Open Policy Agent) with Rego policies, the general- purpose policy engine. Used by Kubernetes admission control, Terraform plan checks, API gateways, microservice authz.
Conftest, runs Rego against config files (YAML, JSON, HCL, Dockerfile) inside a pipeline.
Kyverno, Kubernetes-native policy engine, YAML rules instead of Rego.
Gatekeeper, the original OPA admission controller for Kubernetes.

Examples of policies the operator codifies:

No container runs as UID 0.
No service exposes a privileged port.
All images come from the approved registry.
All Terraform changes touching IAM require a second reviewer.
All S3 buckets enable encryption at rest.

The advantage of code is the same as everywhere else, the rule is versioned, reviewable, testable, and applies uniformly.

See Rego for the Rego language itself.

Secrets#

Secret material has its own pipeline, generation, distribution, rotation, revocation, audit. The shift-left answer is “no secrets in git, no secrets in environment files committed to the repo”.

HashiCorp Vault, the dominant secret broker; dynamic creds, PKI, transit encryption, leases.
AWS Secrets Manager / SSM Parameter Store, Google Secret Manager, Azure Key Vault, cloud-native equivalents.
External Secrets Operator, syncs secrets from a broker into Kubernetes Secret objects.
SOPS (Mozilla) and age, encrypt secrets-in-git with KMS keys when a broker is overkill.
sealed-secrets (Bitnami), one-way-encrypt manifests so they can live in git safely.

Rotation matters more than people expect. A secret that never rotates is one breach away from being permanently compromised.

Runtime#

Detection and response at the running-workload layer. Static checks catch known patterns; runtime catches the rest, the novel exploit, the misconfigured container that bypassed the pipeline, the compromised dependency phoning home.

Falco (CNCF), eBPF-based runtime threat detection on Linux. Watches syscalls; alerts on suspicious patterns (a shell spawned in a container, an unexpected network connection, a sensitive file read).
Tetragon (Isovalent), eBPF-based, more general; observes and optionally enforces.
Tracee (Aqua), eBPF tracing for security events.
Sysdig Secure, commercial; Falco roots, with a UI and policy hub.
Wazuh, open-source SIEM / EDR; host-based.

For Kubernetes specifically, the Pod Security Standards (Privileged / Baseline / Restricted) replaced the deprecated PodSecurityPolicy. Enforce Restricted in production; Baseline is the floor.

Workflow#

A pipeline that fails noisily on every dependency CVE quickly trains the team to ignore it. The workflow around the tooling is where the real practice lives.

Severity thresholds, fail the build on high / critical; file a ticket on medium; ignore low without context.
Triage queue, every finding has an owner and an SLA. A finding with no owner is a finding that will be there in six months.
Exception process, sometimes a finding is a false positive or a known acceptable risk. Document, expire, review.
Suppression with a date, suppressing a finding “forever” is the same as ignoring it. Suppress for a quarter, force re-review.
Metrics, mean time to remediate, % of services with a current SBOM, % of deployments with signed provenance, % of incidents caught pre-prod.

The DORA reliability metrics extend naturally with security counterparts (sometimes called the “DORA security metrics”), how fast the team patches, how many critical findings reach prod, mean time to detect a compromise.

Compliance#

Compliance frameworks (ISO 27001, SOC 2, PCI DSS, HIPAA, FedRAMP, the EU AI Act, NIS2, the EU Cyber Resilience Act) overlap heavily with DevSecOps controls. The pipeline can produce most of the evidence auditors ask for, who built it, what was scanned, what was signed, what was deployed, who approved.

The trick is to design the pipeline so the evidence falls out as a byproduct, not as a separate exercise. SBOMs, signed provenance, policy decisions, change records, all into a queryable audit log.

Common Pitfalls#

The traps that keep DevSecOps a slogan rather than a practice.

Scanner sprawl, ten tools, no triage. Pick the few that fit the team’s languages and stack; ignore the rest.
Vulnerability-count theatre, dashboards that count findings without acting on them. The number that matters is “open high-severity findings older than the SLA”.
Security as veto, the gate that says “no” with no path to “yes”. A useful security team ships exceptions with expiry dates, not blockers without remediation.
Shift-left without shift-right, the editor catches the easy classes; the production runtime catches the rest. Skip either and the gap is the attacker’s lane.
Tooling without process, a scanner with no triage is a noise generator. The workflow around the tool is the practice.
Cargo-cult zero-trust, deploying a mesh, calling it zero-trust, leaving the underlying authz unchanged.

Reading#

Building Secure & Reliable Systems (Google, 2020), free online.
Securing DevOps (Julien Vehent, 2018).
Container Security (Liz Rice, 2020).
Cloud Native Security Cookbook (Josh Armitage, 2022).
NIST SP 800-218 (SSDF), the secure software development framework most compliance regimes map back to.
OWASP SAMM, a maturity model the operator can use to self-assess.