Deploy

Deploy#

A Kubernetes cluster can sit anywhere from the operator’s laptop to a multi-region managed fleet. The deployment shape is a function of who runs the control plane, who runs the worker pool, and what the substrate underneath is. Every option below produces the same API server on port 6443; the difference is the operator burden above and below it.

        flowchart TB
  subgraph local[Local, single binary]
    L1[kind]
    L2[minikube]
    L3[k3d]
  end
  subgraph self[Self-managed]
    S1[kubeadm]
    S2[k3s / k0s]
    S3[Talos]
  end
  subgraph managed[Managed control plane]
    M1[AWS EKS]
    M2[GCP GKE]
    M3[Azure AKS]
  end
  subgraph vmware[VMware]
    V1[Tanzu Kubernetes Grid]
    V2[vSphere with Tanzu]
  end
  OP[Operator] --> local
  OP --> self
  OP --> managed
  OP --> vmware

Concern	Local	Self-managed	Managed	VMware Tanzu
Control plane operator	operator	operator	provider	operator (TKG mgmt cluster)
Worker pool operator	operator	operator	operator	operator
Substrate	laptop	VM or bare metal	cloud	vSphere / vCenter
Right for	dev, CI	on-prem, edge, air-gapped	production in cloud	private / hybrid cloud
Time to a cluster	one minute	tens of minutes	five to ten minutes	one to several hours

Local clusters#

For development, CI, integration tests, and trying changes before they hit shared infrastructure. All three options below run the entire control plane and a node or two inside containers (or a single VM) on the operator’s box.

kind#

Kubernetes in Docker. Each node is a container running a full kubelet plus containerd. The fastest way to a standards-compliant cluster on a workstation.

$ brew install kind        # or: go install sigs.k8s.io/kind@latest
$ kind create cluster --name dev --config - <<'YAML'
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
  - role: control-plane
  - role: worker
  - role: worker
YAML
$ kubectl cluster-info --context kind-dev
$ kind delete cluster --name dev

minikube#

A single-node cluster in a VM (or container). The longest-running local option; ships an addon catalog (registry, ingress, dashboard, metrics-server) that flips on with one flag.

$ minikube start --driver=docker --cpus=4 --memory=8g
$ minikube addons enable ingress
$ minikube addons enable metrics-server
$ kubectl get pods -A

k3d#

Wraps k3s (Rancher’s lightweight distribution) in Docker. Light control plane, fast cluster create / destroy, useful for CI matrices that need multiple cluster versions in parallel.

$ curl -s https://raw.githubusercontent.com/k3d-io/k3d/main/install.sh | bash
$ k3d cluster create dev --agents 2 --port 8080:80@loadbalancer
$ kubectl get nodes

Self-managed clusters#

For on-prem, edge, air-gapped, or any environment where the operator owns the control plane. Three styles dominate.

kubeadm#

The reference bootstrap tool. Initialises the control plane on one node, joins workers, leaves the operator in charge of OS prep, networking, certs, and upgrades. The closest a production cluster gets to “vanilla” Kubernetes.

# On every node: install kubeadm, kubelet, containerd; disable swap.
$ swapoff -a && sed -i '/ swap / s/^/#/' /etc/fstab
$ apt-get install -y containerd kubelet kubeadm kubectl
$ systemctl enable --now containerd kubelet

# On the first control-plane node:
$ kubeadm init \
    --pod-network-cidr=10.244.0.0/16 \
    --control-plane-endpoint=k8s.example.com:6443 \
    --upload-certs
$ mkdir -p ~/.kube && cp /etc/kubernetes/admin.conf ~/.kube/config

# Install a CNI plugin (Calico, Cilium, Flannel) before nodes are Ready.
$ kubectl apply -f https://raw.githubusercontent.com/projectcalico/calico/v3.28.0/manifests/calico.yaml

# On every worker:
$ kubeadm join k8s.example.com:6443 --token <token> \
    --discovery-token-ca-cert-hash sha256:<hash>

For HA control planes, run kubeadm init on the first control plane and kubeadm join --control-plane on the others, fronted by a TCP load balancer.

k3s#

Rancher’s single-binary distribution. Bundles the control plane, the kubelet, a CNI (Flannel), an ingress (Traefik), and a SQLite backing store into one process. The default choice for edge and IoT, also fine on a small VM.

# Control plane node:
$ curl -sfL https://get.k3s.io | sh -
$ cat /var/lib/rancher/k3s/server/node-token       # share with agents

# Agent (worker) node:
$ curl -sfL https://get.k3s.io | \
    K3S_URL=https://master:6443 K3S_TOKEN=<token> sh -

Talos#

A minimal, immutable Linux distribution purpose-built for Kubernetes. No shell, no SSH; the OS is managed entirely through a gRPC API. The right choice when the operator wants the smallest possible attack surface on the host OS.

$ talosctl gen config dev https://k8s.example.com:6443
$ talosctl apply-config --insecure -n 10.0.0.10 --file controlplane.yaml
$ talosctl apply-config --insecure -n 10.0.0.11 --file worker.yaml
$ talosctl bootstrap -n 10.0.0.10
$ talosctl kubeconfig -n 10.0.0.10 .

Managed control planes#

The provider runs the API server, etcd, scheduler, and controllers; the operator brings the worker pool and the workloads. Trades control for a much shorter day-2 list.

EKS (AWS)#

The control plane is a managed service; node groups are either EC2-backed Auto Scaling Groups or Fargate (serverless pods). VPC CNI gives pods real VPC IPs.

$ eksctl create cluster \
    --name prod \
    --region us-east-1 \
    --version 1.30 \
    --nodegroup-name workers \
    --node-type m6i.large \
    --nodes 3 --nodes-min 3 --nodes-max 10 \
    --with-oidc \
    --managed
$ aws eks update-kubeconfig --name prod --region us-east-1

OIDC integration (--with-oidc) is what lets IAM Roles for Service Accounts (IRSA) work; turn it on at cluster create.

GKE (GCP)#

The longest-running managed option. Standard mode gives the operator a node pool; Autopilot lets GCP run nodes too, billing per pod.

$ gcloud container clusters create prod \
    --region us-central1 \
    --release-channel regular \
    --num-nodes 1 \
    --enable-autoscaling --min-nodes 1 --max-nodes 5 \
    --workload-pool=PROJECT_ID.svc.id.goog
$ gcloud container clusters get-credentials prod --region us-central1

Workload Identity (--workload-pool) is GCP’s IRSA equivalent, turn it on at create.

AKS (Azure)#

Azure’s managed offering. Integrates tightly with Entra ID for auth, Azure CNI for pod networking, and Azure Monitor for logs and metrics.

$ az group create -n prod -l eastus
$ az aks create -g prod -n prod \
    --kubernetes-version 1.30.2 \
    --node-count 3 \
    --enable-cluster-autoscaler --min-count 3 --max-count 10 \
    --enable-managed-identity \
    --enable-oidc-issuer --enable-workload-identity
$ az aks get-credentials -g prod -n prod

VMware Tanzu and vSphere#

VMware’s Kubernetes story has two arms, both managed through vCenter.

vSphere with Tanzu#

Enables Kubernetes natively inside a vSphere cluster. The cluster itself becomes a Supervisor cluster; the operator provisions “guest” Tanzu Kubernetes clusters as vSphere resources alongside VMs.

# Enable Workload Management in vCenter (Web UI):
#   Menu -> Workload Management -> Get Started
# Pick a vSphere cluster, choose networking (NSX-T or vSphere networking),
# set a content library for VM images, and a storage policy.

$ kubectl vsphere login --server=<supervisor-vip> \
    --insecure-skip-tls-verify \
    --vsphere-username administrator@vsphere.local

# Provision a workload cluster against the Supervisor:
$ kubectl apply -f - <<'YAML'
apiVersion: run.tanzu.vmware.com/v1alpha3
kind: TanzuKubernetesCluster
metadata:
  name: prod
  namespace: ns-prod
spec:
  topology:
    controlPlane: { replicas: 3, vmClass: best-effort-medium, storageClass: gold }
    nodePools:
      - name: workers
        replicas: 5
        vmClass: best-effort-large
        storageClass: gold
  distribution:
    version: v1.28.8
YAML

Tanzu Kubernetes Grid (TKG)#

A standalone distribution that runs on vSphere (or AWS / Azure) without requiring the Supervisor. The operator stands up a management cluster, which then creates and manages workload clusters.

$ tanzu login                                    # to TMC or local
$ tanzu management-cluster create mgmt \
    --file mgmt-cluster.yaml --ui                # interactive bootstrap
$ tanzu cluster create prod \
    --file prod-cluster.yaml
$ tanzu cluster kubeconfig get prod --admin
$ kubectl get nodes

The TKG installer talks to vCenter to provision VMs, attaches them to NSX-T or vSphere networking, and bootstraps the control plane with Cluster API under the hood.

Bootstrap addons#

A working cluster needs more than the binaries kubeadm or the provider installs. The operator’s day-1 checklist:

Addon	Why
CNI plugin	Pod networking. Calico, Cilium, Flannel, Weave, AWS VPC CNI, Azure CNI, NSX. Required for nodes to be Ready.
Ingress controller	HTTP routing. ingress-nginx, Traefik, Contour, NSX Advanced LB, or the cloud provider’s controller.
cert-manager	Automated TLS via ACME (Let’s Encrypt) or an internal CA. Required for serious HTTPS.
CSI driver	Persistent storage. `ebs-csi` on EKS, `pd-csi` on GKE, `disk-csi` on AKS, `vsphere-csi` on vSphere.
metrics-server	`kubectl top` and HPA CPU / memory metrics.
external-dns	Reconciles Ingress / Service hostnames into Route 53, Cloud DNS, Azure DNS, or Infoblox.
cluster-autoscaler / Karpenter	Add and remove nodes based on pending pods.

A common install order:

# CNI (Calico)
$ kubectl apply -f https://raw.githubusercontent.com/projectcalico/calico/v3.28.0/manifests/tigera-operator.yaml
$ kubectl apply -f https://raw.githubusercontent.com/projectcalico/calico/v3.28.0/manifests/custom-resources.yaml

# metrics-server
$ kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml

# ingress-nginx
$ kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.11.2/deploy/static/provider/cloud/deploy.yaml

# cert-manager
$ kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.15.2/cert-manager.yaml

# external-dns (example: AWS Route 53)
$ helm repo add external-dns https://kubernetes-sigs.github.io/external-dns/
$ helm install external-dns external-dns/external-dns \
    --set provider=aws \
    --set txtOwnerId=prod-cluster

Day-2 operations#

Task	Practice
Upgrades	Patch monthly; minor (1.x → 1.(x+1)) every 3 to 4 months, never skip more than two minors. Drain nodes before upgrading kubelet.
etcd backups	`etcdctl snapshot save` daily plus before every upgrade. Ship snapshots off-cluster. Test restores quarterly.
Certificate rotation	kubeadm clusters rotate on `kubeadm certs renew all`. Managed clusters handle this automatically. Watch the `kube-apiserver` cert expiry on dashboards.
Node patching	Cordon, drain, replace. With Karpenter or managed node groups, terminate and let the autoscaler recreate.
RBAC drift	Audit cluster role bindings quarterly; alert on new bindings to `cluster-admin`.
Audit logs	Enable structured audit policy; ship logs off-cluster. Required for any compliance posture.
Disaster recovery drill	Restore from an etcd snapshot into a fresh cluster at least quarterly. The first time should not be during an outage.

References#

Architecture for what is actually being deployed.
Objects for the API surface the cluster exposes.
Projects for what to run on the cluster.
kubeadm
kind
k3s
Talos
EKS, GKE, AKS
vSphere with Tanzu
Tanzu Kubernetes Grid
Cluster API