Objects

Objects#

Every interaction with Kubernetes is reading or writing an object through the API server. An object is a YAML (or JSON) document with four top-level fields and a kind-specific spec underneath.

apiVersion: <group>/<version>     # apps/v1, v1, batch/v1, networking.k8s.io/v1
kind: <Kind>                      # Pod, Deployment, Service, ...
metadata:
  name: <name>
  namespace: <namespace>          # optional; defaults to "default"
  labels: { <key>: <value> }
  annotations: { <key>: <value> }
spec:
  <kind-specific desired state>
status:
  <kind-specific observed state>  # written by controllers, not the operator

The operator writes spec. Controllers write status. The distinction is rigid; reading the status of an object you just applied is how the operator knows the cluster agrees.

Pods#

The pod is the unit of scheduling and the smallest object the operator deploys. One or more containers that share a network namespace, a set of volumes, and a lifecycle. Containers in a pod can reach each other on localhost and see the same files in mounted volumes.

        flowchart LR
  subgraph pod[Pod]
    direction TB
    INIT[initContainer: migrate]
    C1[container: app]
    C2["container: sidecar (log shipper)"]
    VOL[(volumes: emptyDir, configMap, PVC)]
    NET[shared network namespace]
    INIT --> C1
    C1 --- C2
    C1 --- VOL
    C2 --- VOL
    C1 --- NET
    C2 --- NET
  end
    
apiVersion: v1
kind: Pod
metadata:
  name: web
  labels: { app: web }
spec:
  restartPolicy: Always
  initContainers:
    - name: migrate
      image: myorg/migrator:1.0
      command: ["./migrate.sh"]
  containers:
    - name: app
      image: myorg/app:1.0
      ports: [{ containerPort: 8080 }]
      env:
        - name: DB_URL
          valueFrom: { secretKeyRef: { name: db, key: url } }
      resources:
        requests: { cpu: 100m, memory: 128Mi }
        limits:   { cpu: 500m, memory: 512Mi }
      livenessProbe:
        httpGet: { path: /healthz, port: 8080 }
        initialDelaySeconds: 10
      readinessProbe:
        httpGet: { path: /ready, port: 8080 }
      startupProbe:
        httpGet: { path: /healthz, port: 8080 }
        failureThreshold: 30
      volumeMounts:
        - { name: data, mountPath: /var/lib/app }
    - name: log-shipper
      image: fluent/fluent-bit:3.0
      volumeMounts:
        - { name: data, mountPath: /var/log/app, readOnly: true }
  volumes:
    - name: data
      emptyDir: {}

The operator rarely creates Pods directly; a workload controller manages them. Bare pods are useful for one-off debugging (kubectl run --rm -it --image=alpine sh).

Concept

Detail

Init containers

Run to completion before app containers start. Used for migrations, asset downloads, secret-fetching.

Sidecar containers

Run alongside the app for the pod’s lifetime. Log shippers, service-mesh proxies, secret rotators.

Liveness probe

Restart the container if it fails. Catches deadlocks; do not use for “is the dependency up” checks.

Readiness probe

Toggle the pod’s Service endpoint membership. The right place for “is the dependency up” checks.

Startup probe

Like liveness but only runs at startup; lets slow-booting apps avoid premature liveness kills.

Resources requests

Scheduler reservation. Sum of requests on a node cannot exceed allocatable.

Resources limits

Hard cap; CPU is throttled past the limit, memory past the limit triggers OOMKill.

QoS class: Guaranteed

Requests equal limits on every resource. Last to be evicted under pressure.

QoS class: Burstable

At least one resource sets requests; one or more limits exceed requests. Evicted after BestEffort.

QoS class: BestEffort

No requests or limits. First to be evicted under pressure.

restartPolicy: Always

Default for Deployments; restart on exit regardless of status.

restartPolicy: OnFailure

Default for Jobs; restart only on non-zero exit.

restartPolicy: Never

One-shot; never restart.

Workload controllers#

Controllers wrap pod templates with replica management, rolling updates, and lifecycle policies.

Deployment#

The default for stateless workloads. Declares N identical replicas; rolls forward and back with a configurable strategy.

apiVersion: apps/v1
kind: Deployment
metadata: { name: web }
spec:
  replicas: 3
  strategy:
    type: RollingUpdate
    rollingUpdate: { maxSurge: 1, maxUnavailable: 0 }
  selector:
    matchLabels: { app: web }
  template:
    metadata: { labels: { app: web } }
    spec:
      containers:
        - { name: web, image: myorg/app:1.0 }

The Deployment owns a ReplicaSet for each version; rolling forward creates a new ReplicaSet and scales it up while scaling the old one down. kubectl rollout undo flips the ratio back.

StatefulSet#

For workloads where each pod has a stable identity. Pods are named <set>-0, <set>-1, …, brought up in order, and each pod gets its own PersistentVolumeClaim from a volumeClaimTemplates block. The right choice for Kafka, Cassandra, PostgreSQL, etcd, any clustered storage.

apiVersion: apps/v1
kind: StatefulSet
metadata: { name: pg }
spec:
  serviceName: pg-headless
  replicas: 3
  selector: { matchLabels: { app: pg } }
  template:
    metadata: { labels: { app: pg } }
    spec:
      containers:
        - name: pg
          image: postgres:16
          volumeMounts:
            - { name: data, mountPath: /var/lib/postgresql/data }
  volumeClaimTemplates:
    - metadata: { name: data }
      spec:
        accessModes: ["ReadWriteOnce"]
        resources: { requests: { storage: 50Gi } }
        storageClassName: fast-ssd

DaemonSet#

One pod per node (or per node matching a selector). Used for log shippers, monitoring agents, CNI / CSI components, node-level security daemons.

apiVersion: apps/v1
kind: DaemonSet
metadata: { name: node-exporter, namespace: monitoring }
spec:
  selector: { matchLabels: { app: node-exporter } }
  template:
    metadata: { labels: { app: node-exporter } }
    spec:
      hostNetwork: true
      containers:
        - name: node-exporter
          image: prom/node-exporter:1.8.2
          args: ["--path.rootfs=/host"]
          volumeMounts:
            - { name: rootfs, mountPath: /host, readOnly: true }
      volumes:
        - { name: rootfs, hostPath: { path: / } }
      tolerations:
        - { operator: Exists }   # run on tainted nodes too

Job and CronJob#

Run-to-completion work. A Job runs N parallel pods until completions have succeeded; a CronJob schedules a Job on a cron expression.

apiVersion: batch/v1
kind: CronJob
metadata: { name: nightly-export }
spec:
  schedule: "17 2 * * *"
  concurrencyPolicy: Forbid
  jobTemplate:
    spec:
      backoffLimit: 3
      template:
        spec:
          restartPolicy: OnFailure
          containers:
            - name: export
              image: myorg/export:1.0

Services and traffic#

Services and Ingress map external and internal traffic onto pods.

Service#

A stable virtual IP and DNS name in front of a set of pods chosen by label selector. Four types:

Type

Detail

ClusterIP (default)

In-cluster virtual IP, reachable from pods only. Backed by kube-proxy iptables / IPVS / nftables rules.

NodePort

Allocates a port (30000-32767) on every node; node IP plus the port reaches the service. Crude external access, mostly used in development.

LoadBalancer

Asks the cloud-controller-manager for a real external load balancer (ELB / NLB on AWS, Cloud LB on GCP, Azure LB, NSX Advanced LB on Tanzu).

ExternalName

DNS CNAME to an external host. No proxying, just name resolution.

Headless (clusterIP: None)

Returns pod IPs directly via DNS. Used by StatefulSets so each pod is independently addressable.

 
apiVersion: v1
kind: Service
metadata: { name: web }
spec:
  selector: { app: web }
  ports:
    - { port: 80, targetPort: 8080, protocol: TCP }
  type: ClusterIP

Endpoints / EndpointSlices are the data plane underneath; the service controller writes them as pods come and go.

Ingress#

HTTP-layer routing into the cluster. Rules match host plus path and forward to a Service. The Ingress object is just configuration; an ingress controller (ingress-nginx, Traefik, Contour, the provider’s managed controller) is what serves traffic.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: web
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod
spec:
  ingressClassName: nginx
  tls:
    - hosts: ["app.example.com"]
      secretName: app-tls
  rules:
    - host: app.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: web
                port: { number: 80 }

The Gateway API (gateway.networking.k8s.io) is the successor, splitting infrastructure (Gateway) from route ownership (HTTPRoute, GRPCRoute, TLSRoute). Use it on new clusters where the controller supports it.

NetworkPolicy#

Pod-level firewall. Default policy is permissive (every pod reaches every pod); a NetworkPolicy attached to a pod switches the selected directions to default-deny and only allows the explicit matches. Enforcement is the CNI plugin’s job (Calico, Cilium, Weave); Flannel out of the box does not enforce policy.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: { name: web-allow-from-gw, namespace: prod }
spec:
  podSelector: { matchLabels: { app: web } }
  policyTypes: [Ingress, Egress]
  ingress:
    - from:
        - podSelector: { matchLabels: { app: gateway } }
      ports:
        - { protocol: TCP, port: 8080 }
  egress:
    - to:
        - podSelector: { matchLabels: { app: db } }
      ports: [{ protocol: TCP, port: 5432 }]
    - to:                               # allow DNS
        - namespaceSelector: { matchLabels: { kubernetes.io/metadata.name: kube-system } }
          podSelector: { matchLabels: { k8s-app: kube-dns } }
      ports: [{ protocol: UDP, port: 53 }]

Configuration and secrets#

ConfigMap#

Key-value configuration. Mounted as files in a volume or projected as environment variables.

apiVersion: v1
kind: ConfigMap
metadata: { name: app-cfg }
data:
  LOG_LEVEL: info
  app.yaml: |
    server:
      port: 8080

Secret#

Same shape, base64-encoded data. Encryption at rest in etcd is off by default; turn it on with an EncryptionConfiguration. Most teams move sensitive material out of native Secrets to external secret managers (HashiCorp Vault, AWS Secrets Manager, GCP Secret Manager, Azure Key Vault) and reference them through external-secrets or CSI Secrets Store.

apiVersion: v1
kind: Secret
metadata: { name: db }
type: Opaque
stringData:                    # plain text; the API server base64-encodes
  url: postgres://user:pw@pg:5432/app

Storage#

Persistent storage is two-step. A PersistentVolume (PV) is the actual disk; a PersistentVolumeClaim (PVC) is the workload’s request for one. A StorageClass automates the provisioning side so the operator does not have to create PVs by hand.

        flowchart LR
  POD[Pod] --> PVC[PersistentVolumeClaim]
  PVC -->|bind| PV[PersistentVolume]
  SC[StorageClass] -. dynamic provision .-> PV
  PV --> CSI[CSI driver]
  CSI --> DISK[("Cloud disk, NFS, vSAN")]
    
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata: { name: fast-ssd }
provisioner: pd.csi.storage.gke.io        # or ebs.csi.aws.com, disk.csi.azure.com, csi.vsphere.vmware.com
parameters:
  type: pd-ssd
reclaimPolicy: Retain
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata: { name: data }
spec:
  accessModes: ["ReadWriteOnce"]
  storageClassName: fast-ssd
  resources: { requests: { storage: 50Gi } }

Access mode

Meaning

ReadWriteOnce

One node mounts the volume read-write. Standard for block storage.

ReadWriteOncePod

One pod mounts the volume read-write. Tighter than RWO for sensitive workloads.

ReadOnlyMany

Many nodes mount read-only.

ReadWriteMany

Many nodes mount read-write. Requires shared-filesystem storage (NFS, CephFS, EFS, Filestore, Azure Files, vSAN file).

Identity and RBAC#

ServiceAccount#

The identity a pod runs as inside the cluster. Every pod has one; the default service account is named default in its namespace. The kubelet projects a token for the service account into the pod at /var/run/secrets/kubernetes.io/serviceaccount/token so the workload can call the API server.

Role and RoleBinding#

Permissions live in Roles (namespaced) and ClusterRoles (cluster-wide). Bindings attach roles to subjects (users, groups, service accounts).

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata: { namespace: prod, name: read-pods }
rules:
  - apiGroups: [""]
    resources: ["pods", "pods/log"]
    verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata: { namespace: prod, name: web-read-pods }
subjects:
  - kind: ServiceAccount
    name: web
    namespace: prod
roleRef:
  kind: Role
  name: read-pods
  apiGroup: rbac.authorization.k8s.io

ClusterRole and ClusterRoleBinding are the cluster-wide variants. Use ClusterRole with a (namespaced) RoleBinding when the same rule set is needed in many namespaces.

Scaling#

HorizontalPodAutoscaler#

Watches a metric and scales a Deployment / StatefulSet between minReplicas and maxReplicas to keep the metric near the target. CPU and memory work out of the box (via Metrics Server); custom metrics need the Prometheus Adapter or a similar bridge.

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata: { name: web }
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: web
  minReplicas: 2
  maxReplicas: 20
  metrics:
    - type: Resource
      resource: { name: cpu, target: { type: Utilization, averageUtilization: 60 } }
    - type: Pods
      pods:
        metric: { name: rps }
        target: { type: AverageValue, averageValue: "200" }

VerticalPodAutoscaler#

Adjusts pod requests over time based on observed usage. Off by default; install separately. Useful for workloads whose resource profile drifts, harmful for tightly bin-packed clusters.

PodDisruptionBudget#

A floor on availability during voluntary disruptions (drain, upgrade). Keeps the cluster from evicting too many pods of a workload at once.

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata: { name: web }
spec:
  minAvailable: 2
  selector: { matchLabels: { app: web } }

Tenancy and limits#

Namespace#

Soft tenancy. Scopes names, RBAC, ResourceQuotas, NetworkPolicies. Not a security boundary on its own (pods in different namespaces share a kernel) but the unit of multi-tenancy in most clusters.

apiVersion: v1
kind: Namespace
metadata:
  name: prod
  labels:
    pod-security.kubernetes.io/enforce: restricted
    pod-security.kubernetes.io/audit: restricted

ResourceQuota#

Per-namespace ceiling on aggregate resource use.

apiVersion: v1
kind: ResourceQuota
metadata: { namespace: prod, name: prod-quota }
spec:
  hard:
    requests.cpu: "40"
    requests.memory: 80Gi
    limits.cpu: "80"
    limits.memory: 160Gi
    pods: "100"
    persistentvolumeclaims: "30"

LimitRange#

Per-namespace defaults and bounds for individual containers. Prevents pods from sneaking past quota by submitting requests of zero.

apiVersion: v1
kind: LimitRange
metadata: { namespace: prod, name: defaults }
spec:
  limits:
    - type: Container
      defaultRequest: { cpu: 100m, memory: 128Mi }
      default:        { cpu: 500m, memory: 512Mi }
      min:            { cpu: 50m,  memory: 64Mi }
      max:            { cpu: "4",  memory: 8Gi }

Extending the API#

CustomResourceDefinition#

A CRD teaches the API server a new resource kind. Once installed, that kind is a first-class object the operator reads and writes with kubectl like any built-in. Operators (controllers paired with CRDs) are how vendors ship products into Kubernetes (cert-manager, Argo CD, Prometheus Operator, Strimzi, Tanzu Mission Control).

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata: { name: certificates.cert-manager.io }
spec:
  group: cert-manager.io
  scope: Namespaced
  names:
    kind: Certificate
    plural: certificates
    singular: certificate
  versions:
    - name: v1
      served: true
      storage: true
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              properties:
                dnsNames:
                  type: array
                  items: { type: string }

The operator pattern (controller plus CRD) extends Kubernetes the same way kube-controller-manager extends the core API. See Projects for when to reach for one.

References#

Contents