Projects

Projects#

A Kubernetes cluster is a runtime, not a product. The product is whatever the operator is shipping on top, a coding agent, a SIEM backend, a target range, a redirector mesh, a data pipeline. This page covers the patterns and the moving parts that turn a cluster into a project the operator can iterate, deploy, observe, and hand off.

        flowchart LR
  DEV[Developer commit] --> CI[CI build + test]
  CI --> REG[(Container registry)]
  CI --> CFG[Manifests / Helm chart]
  CFG --> GIT[(Git repo, env overlays)]
  GIT --> GITOPS[GitOps controller]
  REG --> GITOPS
  GITOPS --> K8S[(Kubernetes cluster)]
  K8S --> OBS[Observability stack]
  OBS --> ON[On-call alerts]
  OBS --> DASH[Dashboards]

Every block above is replaceable. The shape, source to artifact to declared state to running pod to observed signal, is universal.

Project layout#

A typical repository layout for an in-house service on Kubernetes:

my-service/
|-- src/                        # application source
|-- Dockerfile                  # builds the image CI pushes
|-- chart/                      # Helm chart (if using Helm)
|   |-- Chart.yaml
|   |-- values.yaml             # defaults
|   |-- values-dev.yaml         # env overrides
|   |-- values-prod.yaml
|   `-- templates/
|       |-- deployment.yaml
|       |-- service.yaml
|       |-- ingress.yaml
|       |-- hpa.yaml
|       |-- pdb.yaml
|       `-- networkpolicy.yaml
|-- kustomize/                  # alternative: kustomize
|   |-- base/
|   `-- overlays/{dev,stage,prod}/
|-- .github/workflows/ci.yaml   # build + push + (optionally) deploy
`-- README.md

Pick Helm when the workload has parameters (registries, replica counts, secrets) that vary per environment. Pick Kustomize when overlays are the primary axis of variation and the operator wants to avoid templating. The two are not mutually exclusive; some teams ship Helm charts and apply Kustomize patches on top.

Workload patterns#

Stateless web service#

The base case. Deployment plus Service plus Ingress. Add an HPA and a PDB.

        flowchart LR
  ING[Ingress] --> SVC[Service]
  SVC --> D[Deployment, N replicas]
  HPA[HPA] -. scales .-> D
  PDB[PDB] -. floors .-> D

Stateful service#

StatefulSet plus headless Service plus a backup workflow. Per-pod PVCs via volumeClaimTemplates; pods named <set>-0, <set>-1 addressed by DNS through the headless Service.

Use an operator (Strimzi for Kafka, postgres-operator, Cassandra, MongoDB Community Operator) when one is available. Operators encode the ordering, backup, failover, and rolling-update logic that a raw StatefulSet does not.

Batch and scheduled work#

Jobs for run-to-completion (nightly ETL, backups). CronJobs for recurring. Argo Workflows or Tekton when the pipeline has fan-out, fan-in, or DAG structure.

Long-running consumers#

Deployment scaled by queue depth via custom metrics or KEDA (Kubernetes Event-driven Autoscaling). Pods drain in-flight work on shutdown using terminationGracePeriodSeconds and a preStop hook that signals the consumer to stop pulling.

CI/CD and GitOps#

The pipeline that ships a commit to production has two halves. CI builds and tests the image; CD moves the new image into the cluster.

CI pipeline#

A minimal CI build for a Kubernetes-bound service:

# .github/workflows/ci.yaml
name: ci
on: [push, pull_request]
jobs:
  build:
    runs-on: ubuntu-latest
    permissions:
      contents: read
      id-token: write             # for OIDC into the registry
    steps:
      - uses: actions/checkout@v4
      - uses: docker/setup-buildx-action@v3
      - uses: docker/login-action@v3
        with:
          registry: ghcr.io
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}
      - run: make test
      - uses: docker/build-push-action@v6
        with:
          context: .
          push: true
          tags: ghcr.io/myorg/my-service:${{ github.sha }}
          cache-from: type=gha
          cache-to: type=gha,mode=max

The image tag is the commit SHA, never latest in production. Immutable tags mean rollbacks are a single object change instead of a registry race.

GitOps#

A controller in the cluster watches a git repository of manifests and reconciles the cluster against the repo. The operator never kubectl apply to production; the operator commits and the controller applies.

        flowchart LR
  DEV[Developer] --> PR[Pull request]
  PR --> MAIN[(main branch)]
  MAIN --> SYNC[Argo CD / Flux]
  SYNC --> CLUSTER[(Cluster)]
  CLUSTER --> STATUS[Sync status]
  STATUS --> SYNC

Tool

Detail

Argo CD

Pull-based. UI plus CLI; Application CRDs reference a git repo, a path, and a target cluster / namespace. Diff and sync visualization built in.

Flux

Pull-based. Lower-level; composes GitRepository, Kustomization, HelmRelease CRDs. Lighter UI, stronger automation seams.

Argo Rollouts

Pairs with Argo CD or any Deployment; adds canary and blue-green strategies on top of the standard rolling update.

Argo CD example:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata: { name: my-service, namespace: argocd }
spec:
  project: default
  source:
    repoURL: https://github.com/myorg/k8s-manifests
    targetRevision: main
    path: my-service/overlays/prod
  destination:
    server: https://kubernetes.default.svc
    namespace: prod
  syncPolicy:
    automated: { prune: true, selfHeal: true }
    syncOptions: ["CreateNamespace=true"]

Image update pipelines#

A common pattern: CI builds the image, commits the new tag to the manifests repo, the GitOps controller picks up the diff. Either hand-rolled (CI does yq plus git push) or via dedicated tools (Argo CD Image Updater, Flux Image Automation Controller).

Observability#

Three signals, three stacks. The operator picks one per signal and wires alerts off the back of them.

        flowchart LR
  APP[Workload pods] -->|metrics endpoint| PROM[Prometheus]
  APP -->|stdout / stderr| LOGS[Loki / Elasticsearch]
  APP -->|OTLP| OTEL[OpenTelemetry Collector]
  OTEL --> TEMPO[Tempo / Jaeger]
  OTEL --> PROM
  OTEL --> LOGS
  PROM --> GRAFANA[Grafana]
  LOGS --> GRAFANA
  TEMPO --> GRAFANA
  PROM --> ALERT[Alertmanager]
  ALERT --> PAGE[PagerDuty / Slack]

Signal

Open-source default

Cloud-managed option

Notes

Metrics

Prometheus + kube-prometheus-stack

CloudWatch, Cloud Monitoring, Azure Monitor, VMware Aria Operations

ServiceMonitor / PodMonitor CRDs auto-discover scrape targets.

Logs

Loki, Fluent Bit

CloudWatch Logs, Cloud Logging, Azure Monitor Logs

Index labels, not contents; cheap at scale.

Traces

OpenTelemetry Collector + Tempo / Jaeger

X-Ray, Cloud Trace, Application Insights

OTLP is the cross-vendor wire format.

Dashboards

Grafana

Provider-managed Grafana, CloudWatch dashboards

Pair with golden-signal templates (latency, traffic, errors, saturation).

Alerting

Alertmanager

CloudWatch Alarms, Cloud Monitoring alerts, Azure Monitor alerts

Page on symptoms (user-facing), not causes.

A practical bootstrap:

$ helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
$ helm install monitoring prometheus-community/kube-prometheus-stack \
    --namespace monitoring --create-namespace

That single chart pulls Prometheus, Alertmanager, Grafana, the Prometheus Operator, plus default kube-state-metrics and node-exporter DaemonSets.

Secrets management#

Three options, in increasing order of operator burden and security posture.

Pattern

Detail

Native Secret

Object in etcd. Enable encryption at rest; tight RBAC; audit reads. Acceptable for non-sensitive config; insufficient for credentials of real consequence.

Sealed Secrets

Bitnami’s tool; the operator commits an encrypted SealedSecret to git, the cluster’s controller decrypts to a Secret. Lets GitOps own secrets without leaking them.

External secrets manager

HashiCorp Vault, AWS Secrets Manager, GCP Secret Manager, Azure Key Vault. Pulled into the cluster via External Secrets Operator or CSI Secrets Store. Single source of truth, rotates without re-deploying.

Example with External Secrets Operator pulling from AWS:

apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata: { name: aws, namespace: prod }
spec:
  provider:
    aws:
      service: SecretsManager
      region: us-east-1
      auth:
        jwt: { serviceAccountRef: { name: external-secrets } }
---
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata: { name: db, namespace: prod }
spec:
  refreshInterval: 1h
  secretStoreRef: { name: aws, kind: SecretStore }
  target: { name: db }
  data:
    - secretKey: url
      remoteRef: { key: prod/db, property: url }

Multi-tenancy#

Two shapes the operator picks between, often based on the threat model the workloads sit in.

Namespace per tenant#

Cheaper. One cluster, one namespace per tenant, RBAC plus NetworkPolicy plus ResourceQuota plus LimitRange enforcing boundaries. Pod Security Standards (restricted profile) reduce breakout risk.

Caveats: a kernel CVE that escapes a container reaches every tenant. Service mesh sidecars can leak metadata across namespaces if not configured carefully. Not appropriate when tenants do not trust each other.

Cluster per tenant#

One cluster per tenant. Stronger isolation, much higher operator burden (every cluster needs its own upgrade, backup, addon set). The standard choice when tenants are external customers, hostile, or regulated separately. Tooling: Cluster API, vCluster, Karmada.

Sample end-to-end project#

A minimum-viable shape the operator can lift from. A single service (“api”) with a Postgres dependency, dev and prod environments, GitOps deploy.

k8s-manifests/
|-- argocd/
|   |-- api-dev.yaml
|   `-- api-prod.yaml                       # Argo CD Applications
`-- api/
    |-- base/
    |   |-- kustomization.yaml
    |   |-- deployment.yaml
    |   |-- service.yaml
    |   |-- ingress.yaml
    |   |-- hpa.yaml
    |   |-- pdb.yaml
    |   |-- networkpolicy.yaml
    |   `-- externalsecret.yaml
    `-- overlays/
        |-- dev/
        |   |-- kustomization.yaml
        |   `-- replica-patch.yaml          # 1 replica, smaller resources
        `-- prod/
            |-- kustomization.yaml
            |-- replica-patch.yaml          # 4 replicas
            `-- hpa-patch.yaml              # higher ceiling

Build the image in CI, push to the registry, commit the new tag to the manifests repo on green. Argo CD picks up the change and syncs the target overlay into the matching cluster.

$ kustomize build api/overlays/prod | kubectl diff -f -
$ git commit -am "api: bump to abc1234"
$ git push                                  # Argo CD takes it from here

This pattern scales from one service to dozens without changing shape, the operator just adds more directories under api/ (one per service) and more Argo CD Applications pointing at them.

References#

Contents