Nomad

Nomad#

HashiCorp Nomad is the operator’s lighter-weight scheduler. One binary, declarative HCL job specs, a built-in Raft-backed state store, and a workload model that handles containers, raw binaries, Java apps, and QEMU VMs from the same control plane. The bargain versus Kubernetes is less ecosystem and fewer batteries included, in exchange for one-tenth the operational surface.

For the operator, Nomad sits in two places, small clusters where Kubernetes is overkill (a handful of nodes, two or three services) and mixed-workload estates where legacy binaries have to live alongside containers without re-platforming first.

Architecture#

        flowchart TB
  OP["Operator<br/>nomad CLI / UI / API"]

  subgraph srv["Control plane (Raft, 3-5 servers)"]
    direction LR
    S1["server 1"]
    S2["server 2"]
    S3["server 3"]
    S1 --- S2 --- S3 --- S1
  end

  subgraph node["Client nodes"]
    direction LR
    C1["agent + drivers"]
    C2["agent + drivers"]
    C3["agent + drivers"]
  end

  CONSUL[("Consul<br/>(optional)")]
  VAULT[("Vault<br/>(optional)")]

  OP --> srv
  srv --> node
  srv -. service catalog .- CONSUL
  srv -. secrets .- VAULT

Component

Role

Server

Control plane. Holds the job state, runs the scheduler, elects a leader through Raft (3 or 5 servers in production).

Client (agent)

Per-node daemon. Reports resources, runs allocations, restarts failed tasks, drains on shutdown.

Task driver

Pluggable execution backend. docker, podman, exec (raw binary), raw_exec, java, qemu, custom plugins.

Job

The unit of work the operator submits. Contains one or more task groups; each group contains tasks.

Allocation

A scheduled instance of a task group on a specific node.

Consul

Optional service catalog and health checking. Provides Nomad’s service mesh and DNS.

Vault

Optional secret broker. Nomad templates secrets into tasks at start.

Job spec#

Jobs are HCL files. The same shape covers a Docker service, a batch job, or a system DaemonSet equivalent.

job "web" {
  datacenters = ["dc1"]
  type        = "service"

  group "web" {
    count = 3

    network {
      port "http" { to = 8080 }
    }

    service {
      name = "web"
      port = "http"
      check {
        type     = "http"
        path     = "/healthz"
        interval = "10s"
        timeout  = "2s"
      }
    }

    task "server" {
      driver = "docker"
      config {
        image = "myorg/app:1.0"
        ports = ["http"]
      }
      resources {
        cpu    = 200    # MHz
        memory = 256    # MiB
      }
    }
  }
}

Submit:

$ nomad job run web.nomad.hcl
$ nomad job status web
$ nomad alloc logs <alloc-id>
$ nomad alloc exec <alloc-id> /bin/sh

Job types#

Type

Detail

service

Long-running, restarted on failure. The default for web services and APIs.

batch

Run to completion; not restarted on exit.

system

One allocation per eligible node. The DaemonSet equivalent.

sysbatch

One run-to-completion allocation per eligible node. For one-off node-scoped tasks.

periodic

Cron-style schedule wrapped around any of the above.

parameterized

A job template the operator instantiates with parameters; runs once per dispatch. Used for ad hoc processing.

Drivers#

The driver is what actually runs the workload on a node. Mix them; one cluster can run containers next to raw binaries.

Driver

Detail

docker

The default container driver. Docker Engine on the node.

podman

Rootless container runtime. Works where Docker is not installed.

containerd

Direct containerd integration (no Docker daemon).

exec

Sandboxed binary. cgroups + chroot + namespaces, no container image.

raw_exec

Unsandboxed binary. The escape hatch; disabled by default for safety.

java

Run a JAR or Java class with JVM args from the job spec.

qemu

Boot a QEMU VM image. Edge / mixed-workload clusters.

Networking#

Nomad’s default networking is host networking, the task binds a port on the host. Optional bridge mode and CNI plugins give each allocation its own network namespace.

network {
  mode = "bridge"            # CNI plugin (Consul Connect, custom)
  port "http" { to = 8080 }
}

For service discovery the operator typically wires in Consul:

service {
  name     = "web"
  provider = "consul"
  port     = "http"
}

Consul DNS then resolves web.service.consul on every node.

Service mesh#

Consul Connect (now Consul Service Mesh) is the native option, sidecar Envoys, automatic mTLS, intentions for policy. Linkerd also runs on Nomad with manual sidecar configuration. Both are more deliberate than Kubernetes’ ambient meshes.

CLI quick reference#

$ nomad job run web.nomad.hcl
$ nomad job plan web.nomad.hcl                # dry-run + diff
$ nomad job status -short
$ nomad alloc status <id>
$ nomad alloc logs -f <id> <task>
$ nomad alloc exec <id> <task> /bin/sh
$ nomad node status
$ nomad node drain -enable <node-id>
$ nomad system gc                             # garbage-collect dead allocs

Deploy a cluster#

# Each control plane node:
$ sudo nomad agent -config /etc/nomad.d/server.hcl

# Each worker:
$ sudo nomad agent -config /etc/nomad.d/client.hcl

# server.hcl
server {
  enabled = true
  bootstrap_expect = 3
}
acl    { enabled = true }
ui     { enabled = true }

# client.hcl
client {
  enabled = true
  servers = ["nomad.example.com:4647"]
}
plugin "docker" {}

Bootstrap ACL:

$ nomad acl bootstrap
$ export NOMAD_TOKEN=<token>

When to pick Nomad#

  • Cluster is small (1 to 20 nodes) and Kubernetes feels like overkill.

  • Workload mix includes raw binaries, JVM apps, or VMs alongside containers.

  • Team is small and the operator wants one binary instead of a control plane with five.

  • HashiCorp stack already in place (Consul, Vault, Terraform).

When not to#

  • The ecosystem (operators, ingress controllers, CSI drivers, observability tooling) matters more than scheduler simplicity.

  • The cluster needs RBAC + admission control with the granularity of Kubernetes RBAC + Pod Security Standards + admission webhooks.

  • The team already knows Kubernetes; the learning cost is the same.

References#

Contents