Architecture

A Kubernetes cluster has two planes. The control plane holds the desired state, schedules work, and reconciles reality against the declaration. The data plane is the pool of worker nodes that run the actual pods. Every cluster, from a laptop kind cluster to a hyperscaler-managed regional fleet, has the same components doing the same jobs; only the deployment shape and the operator’s view into them change.

        flowchart TB
  subgraph CP[Control plane]
    direction TB
    API[kube-apiserver]
    ETCD[(etcd)]
    SCHED[kube-scheduler]
    CM[kube-controller-manager]
    CCM[cloud-controller-manager]
    API --- ETCD
    API --- SCHED
    API --- CM
    API --- CCM
  end
  subgraph N1[Worker node 1]
    K1[kubelet]
    P1[kube-proxy]
    R1["CRI runtime (containerd)"]
    K1 --- R1
  end
  subgraph N2[Worker node 2]
    K2[kubelet]
    P2[kube-proxy]
    R2["CRI runtime (containerd)"]
    K2 --- R2
  end
  CLI[kubectl / client] -->|HTTPS| API
  API -->|watch / exec| K1
  API -->|watch / exec| K2
  CCM -->|provider API| CLOUD[("cloud or vCenter API")]
    

The whole control plane talks through the API server. No component reads or writes etcd directly except the API server. No component talks to nodes directly except the API server. This single choke point is what makes the cluster auditable and what makes API server availability the binding constraint for the entire control plane.

The control plane

Five processes (four mandatory plus the optional cloud-controller- manager) make up the control plane. On a self-managed cluster they run on dedicated control-plane nodes; on a managed cluster (EKS, GKE, AKS, TKG) the provider hides them behind their own SRE team.

kube-apiserver

The stateless REST front end for the cluster. Every read, every write, every watch streams through it. It validates requests against schemas, runs admission controllers, persists accepted state to etcd, and serves watch streams to the controllers and kubelets that subscribe.

Property	Detail
Port	6443/tcp (HTTPS). The only network-exposed control-plane port the operator normally interacts with.
Auth	X.509 client certs, bearer tokens, service-account JWTs, OIDC, webhook. Authentication is pluggable.
Authz	RBAC (default), ABAC, Node, webhook. RBAC is what the operator writes against.
Admission	Mutating then validating admission plugins (PodSecurity, ResourceQuota, LimitRanger, ImagePolicyWebhook) run on every write.
Storage	All persisted state in etcd, keyed by resource path (/registry/pods/default/web-7d8c-xyz).
Scaling	Stateless. Add replicas behind a load balancer for HA; HA needs an HA etcd quorum to be useful.

The API server is the audit log. Enable structured audit policy (--audit-policy-file) and ship the log; everything the cluster ever did flows through this one process.

etcd

A distributed key-value store using the Raft consensus algorithm. Kubernetes stores every object here as protobuf-encoded JSON keyed by resource path. Cluster state, secrets, RBAC, configmaps, service account tokens, every CRD instance, all of it.

Property	Detail
Port	2379/tcp (client), 2380/tcp (peer).
Quorum	3 or 5 voting members; odd numbers tolerate the most failures (3 tolerates 1, 5 tolerates 2).
Auth	Mutual TLS between every client and peer. API server is the only client most of the time.
Backup	etcdctl snapshot save; back up at least daily and ship off-cluster. The only recovery path for a broken cluster.
Encryption at rest	Off by default. Enable via EncryptionConfiguration so leaked etcd snapshots do not leak every Secret.
Performance	Latency-sensitive. SSDs, low-latency disks, dedicated network if possible. fsync slowness is the most common cluster pathology.

Losing etcd quorum is a cluster outage. Losing all etcd state without a backup is a cluster rebuild. The operator’s first disaster-recovery drill is restoring from a snapshot.

kube-scheduler

A loop that watches for pods with an empty spec.nodeName and binds each one to a node. Two phases:

• Filtering, drop nodes that fail hard constraints (taints, node selectors, affinity, resource requests, port conflicts, topology spread).

• Scoring, rank the survivors by soft preferences (least requested, image locality, inter-pod affinity, topology spread) and bind to the top.

Property	Detail
Pluggability	Scheduling framework lets the operator add custom plugins or run a second scheduler in parallel (schedulerName per pod).
Preemption	High-priority pods can evict lower-priority pods to make room. Driven by PriorityClass.
Failure mode	If the scheduler is down, no new pods get placed. Running pods are unaffected; the cluster degrades only on the new-work axis.

Custom scheduling decisions (GPU-aware bin packing, locality to data, batch gang scheduling) are common reasons the operator writes plugins or runs a sidecar scheduler.

kube-controller-manager

A single binary that hosts dozens of independent control loops, each watching one or more resource types and reconciling state. Examples worth knowing:

Controller	Job
Deployment controller	Creates / updates ReplicaSets when a Deployment changes.
ReplicaSet controller	Maintains the desired pod count for a ReplicaSet.
StatefulSet controller	Manages stable identities and ordered rollout for stateful workloads.
DaemonSet controller	Creates one pod per node matching the selector.
Job / CronJob controllers	Run-to-completion and scheduled work.
Node controller	Marks nodes unhealthy when the kubelet stops heartbeating.
Service controller	Allocates ClusterIPs and updates Endpoints / EndpointSlices.
Endpoint(Slice) controller	Tracks which pods back which Service.
PV / PVC controllers	Binds claims to volumes; provisions dynamically through a StorageClass.
Service account / token controller	Creates default service accounts and bound tokens.
Namespace controller	Drives namespace termination through finalizers.

Every controller follows the same pattern, watch the API for changes, compute desired state, write back the difference. The operator writes custom controllers in the same shape when extending the cluster with operators and CRDs.

cloud-controller-manager

The piece that knows about the underlying infrastructure. Splits out of kube-controller-manager so a managed cluster’s provider ships their own without forking core Kubernetes. Responsible for:

• Node controller (cloud variant), tags nodes with provider zone, region, instance type; reacts when an instance disappears.

• Route controller, programs the underlying network’s route tables when nodes need pod-CIDR routes.

• Service controller, creates and tears down the cloud load balancer behind a type: LoadBalancer Service (AWS ELB / NLB, GCP Cloud Load Balancing, Azure Load Balancer, NSX Advanced LB on vSphere with Tanzu).

On AWS it talks to EC2 and ELB APIs; on GCP, Compute and LB; on Azure, the Azure Resource Manager; on VMware Tanzu Kubernetes Grid, the vCenter API plus NSX. Kubeadm clusters with no cloud integration ship without this binary.

The data plane

Every worker node runs three components plus whatever DaemonSet pods land there (log shippers, node exporters, CSI drivers).

kubelet

The node agent. Subscribes to the API server for pods bound to its own nodeName, talks to the local container runtime to bring them up, runs probes, reports status back. The kubelet is the single point of trust between the control plane and the node.

Property	Detail
Port	10250/tcp (HTTPS). API server calls in for exec, logs, port-forward, metrics.
Authentication	Node-scoped X.509 client cert; rotated via certificate signing requests when rotateCertificates: true.
Authorization	Defaults to the Node authorizer plus admission webhook (NodeRestriction) so a kubelet can only touch its own pods and node object.
Runtime interface	Container Runtime Interface (CRI) over gRPC on a Unix socket (/run/containerd/containerd.sock).
Storage and network	Drives CSI plugins for volume mounts and CNI plugins for pod network setup.
Probes	Runs liveness, readiness, and startup probes; restarts containers and toggles endpoint membership accordingly.
Eviction	Watches node-level pressure (memory, disk, PID) and evicts pods to keep the node healthy. Tunable via --eviction-hard and --eviction-soft.

If the kubelet stops heartbeating (Node.status not updated within the configured grace period) the node controller marks the node NotReady and, eventually, evicts its pods to other nodes.

kube-proxy

The per-node implementation of Service. Watches Endpoints / EndpointSlices and programs a load-balanced virtual IP for each ClusterIP. Three modes:

Mode	Behavior
iptables (default)	Programs DNAT rules that match the ClusterIP and rewrite to a randomly chosen endpoint pod. Scales to a few thousand services before rule evaluation cost shows up.
ipvs	Programs Linux IPVS rules; better scale, hash-based selection, supports more LB algorithms (round-robin, least conn, source hash).
nftables	Newer alternative on top of nftables; replaces iptables on modern distros.

eBPF-based dataplanes (Cilium, Calico-eBPF) replace kube-proxy entirely with eBPF programs attached to the pod network, lower latency and richer policy.

Container runtime

The component that actually pulls images and runs containers. The kubelet talks CRI (gRPC) to whichever runtime is installed.

Runtime	Detail
containerd	The current default. CNCF graduate, derived from Docker. Most distros ship this.
CRI-O	Red Hat / OpenShift default. Minimal, OCI-focused.
cri-dockerd	Shim that lets the old Docker engine satisfy CRI. Used where Docker is still mandated.
gVisor / Kata Containers	Sandboxed runtimes. Run a stripped kernel (gVisor) or a microVM (Kata) per pod, trading performance for stronger isolation.

The container runtime is what the operator’s threat model lives in on the data plane. A container escape is a runtime escape; pick the runtime accordingly when the workload is hostile.

Cluster addons

A working cluster needs more than the binaries above. The following addons are present in virtually every production cluster and are usually installed as DaemonSets or Deployments in kube-system.

CoreDNS

Cluster DNS. Resolves <service>.<namespace>.svc.cluster.local into ClusterIPs, plus pod DNS, headless service A records, and external forwarding. Sits behind a Service of its own that every pod’s /etc/resolv.conf points at via the kubelet’s pod-DNS config.

CNI plugin

The network plugin that wires up pod-to-pod connectivity. The kubelet calls a CNI binary when a pod starts to set up its network namespace, allocate an IP from the cluster CIDR, and program whatever overlay or routing the plugin needs. Choices:

Plugin	Detail
Calico	BGP routing or VXLAN overlay; NetworkPolicy support; eBPF dataplane available.
Cilium	eBPF-native. Pod networking, NetworkPolicy, service mesh, observability (Hubble), kube-proxy replacement.
Flannel	Simple VXLAN overlay; no policy support out of the box.
Weave Net	Mesh VXLAN; lightweight; policy support.
AWS VPC CNI / Azure CNI / GCP NetD	Provider-native plugins that assign pods real cloud-VPC IPs (no overlay), so cloud security groups apply directly.
VMware NSX	NSX-T integration on Tanzu Kubernetes Grid. Pod networks become NSX segments.

The CNI choice constrains NetworkPolicy enforcement, mTLS support, and the operator’s visibility into east-west traffic.

CSI plugin

The storage equivalent of CNI. A CSI driver implements three gRPC services (controller, node, identity) so kubelets can mount cloud disks, NFS shares, vSAN volumes, or any storage backend a vendor ships a driver for. Triggered by the PV / PVC controllers when a claim binds.

Metrics server

A small aggregator that scrapes kubelet.metrics from every node and exposes the data through the Metrics API (metrics.k8s.io). Required for kubectl top and for the Horizontal Pod Autoscaler’s CPU / memory rules. Not a long-term metrics store; pair with Prometheus for that.

Ingress controller

The data-plane piece behind an Ingress or Gateway resource. Common choices: ingress-nginx, Traefik, HAProxy Ingress, Contour, NSX Advanced LB (Avi) on Tanzu, plus the managed AWS / GCP / Azure ingress controllers fronted by their cloud load balancers. The controller watches Ingress / HTTPRoute / Gateway objects and programs its own dataplane (an nginx, a sidecar Envoy, a cloud LB) to match.

Request flow

A kubectl apply -f deploy.yaml traverses every component above.

        sequenceDiagram
  participant U as kubectl
  participant API as kube-apiserver
  participant ETCD as etcd
  participant DC as Deployment ctrl
  participant RSC as ReplicaSet ctrl
  participant SCH as kube-scheduler
  participant KL as kubelet
  participant CRI as CRI runtime
  U->>API: PUT /apis/apps/v1/.../deployments
  API->>API: auth, admission, validate
  API->>ETCD: write Deployment object
  ETCD-->>API: ack
  API-->>U: 200 OK
  DC->>API: watch Deployments
  DC->>API: create ReplicaSet
  RSC->>API: watch ReplicaSets
  RSC->>API: create N Pods
  SCH->>API: watch unscheduled Pods
  SCH->>API: bind Pod to Node
  KL->>API: watch Pods for myNode
  KL->>CRI: pull image, create container
  CRI-->>KL: container running
  KL->>API: update Pod status
    

Every arrow above is a watch or write against the API server. No shortcut paths, no direct etcd reads, no node-to-node coordination. This is what makes the cluster legible and the API server the choke point.

High availability

A production control plane runs every binary above with redundancy.

• etcd, a quorum of 3 or 5 across availability zones. Backups off-cluster, encryption at rest enabled.

• kube-apiserver, two or more replicas behind an L4 load balancer. Stateless so the LB is sufficient.

• kube-scheduler and kube-controller-manager, leader- elected; run two replicas, only one is active at a time.

• cloud-controller-manager, same leader-election shape on clusters that have one.

• Worker nodes, spread across zones; topologySpreadConstraints on workloads to enforce the spread.

Managed clusters (EKS, GKE, AKS, TKG) handle this layer for the operator. The operator’s HA burden on a managed cluster is the worker pool and the workloads on it.

References

• Objects for the API surface the operator writes against.

• Deploy for how to stand a cluster up.

• Projects for what to build on top.

• Kubernetes components

• etcd documentation

• Container Runtime Interface (CRI)

• CNI specification

• CSI specification

• Kubernetes audit logging