Services

Services#

A service is a long-running program supervised by an init system and provides some sort of service to the system, user, and or network. often called a daemon, a service, or a process. The operator’s tools are short:

systemctl to control
journalctl to read logs
systemd-analyze to time the boot, and the unit files under
/etc/systemd/system/. On objective these answer “what is enabled, what is running, what is failing, and what could be hiding in a drop-in.” On a defended estate they answer “what did the intruder leave behind.”

Init System#

An init system is the first userspace program the kernel starts (PID 1). It brings the rest of the system up, supervises long-running services, and reaps orphans. Several implementations exist; the operator meets systemd on every modern desktop and server distribution, and the others mainly in containers, embedded gear, and a few opinionated distros.

Init	Where seen	Notes
systemd	Debian, Ubuntu, RHEL, Fedora, Arch, SUSE, Kali	Modern default; unit files, cgroups, journald, timers, sockets. The rest of this page assumes systemd.
SysV init	Legacy boxes, Devuan, some appliances	Shell scripts in `/etc/init.d/` driven by runlevels. Predates systemd.
OpenRC	Alpine, Gentoo, Artix	Dependency-based, parallel start; configured under `/etc/init.d/` + `/etc/conf.d/`.
Upstart	Old Ubuntu (6.10-15.04), legacy ChromeOS	Event-driven; retired upstream in 2014. Replaced by systemd.
runit	Void Linux, Artix variant	Tiny three-stage init; services are `run` scripts under `/etc/service/`.
s6	Alpine option, embedded, container PID-1	skarnet’s process supervisor; minimal, deterministic.
s6-rc	s6-based distros	Dependency layer on top of s6, equivalent to systemd targets.
launchd	macOS (not Linux)	Apple’s init + service manager; `.plist` units.
BusyBox init	Containers, embedded, recovery images	Single-binary init from BusyBox; reads `/etc/inittab`.

SystemD#

systemd is PID 1 on every modern Linux distribution: it parses unit files at boot, computes a dependency graph, drives the system to a target, supervises every process it starts, and restarts on failure. Everything below it is either a unit (declarative config), a tool that talks to it, or an artifact it owns (a cgroup, a socket, a journal entry).

Unit types#

A unit is the granularity of systemd’s world. Eleven types exist; .service is by far the most common.

Type	What it represents
`.service`	A long-running program (the default; what most people mean by “a service”)
`.timer`	A schedule that triggers another unit; the cron replacement
`.socket`	A listening socket; activates a service on first connection
`.path`	A filesystem watcher; activates a service on change / arrival
`.mount`	A mount point; equivalent to an `/etc/fstab` entry
`.automount`	Lazy mount; the actual mount happens on first access
`.target`	A grouping / synchronisation point (`multi-user.target`, `graphical.target`, `network-online.target`)
`.device`	A device exposed via udev
`.swap`	A swap space activation
`.slice`	A cgroup branch in the resource hierarchy
`.scope`	An externally-created cgroup attached to systemd

Dependencies and ordering#

Two orthogonal axes, what must be running and when. The operator-facing directives are below.

Directive	Meaning
`Wants=`	Pull in the named units; their failure doesn’t fail us. The most common dependency; `[Install] WantedBy=` adds this from the other direction.
`Requires=`	Hard dependency; if the named unit stops we stop.
`Requisite=`	Like `Requires=`, but doesn’t start the named unit; it must already be running.
`After=`	Start after the named units (ordering only, not a dep).
`Before=`	Start before the named units.
`Conflicts=`	Can’t run together; starting one stops the other.

A service moves through six states. The diagram below maps every state and the transitions that connect them. Each edge is labeled with the operator command or the systemd event that triggers it.

        stateDiagram-v2
    direction LR
    [*] --> inactive
    inactive --> activating: systemctl start
    activating --> active: ExecStart ok
    activating --> failed: start fails / timeout
    active --> reloading: systemctl reload
    reloading --> active: ExecReload ok
    active --> deactivating: systemctl stop / SIGTERM
    deactivating --> inactive: ExecStop ok
    active --> failed: process exits / watchdog miss
    failed --> activating: Restart= (auto-restart)
    failed --> inactive: systemctl reset-failed
    inactive --> [*]

Operator → state-machine cheat sheet:

Command	Path through the machine
`systemctl start X`	`inactive` → `activating` → `active`
`systemctl stop X`	`active` → `deactivating` → `inactive`
`systemctl restart X`	stop, then start (full cycle)
`systemctl reload X`	`active` → `reloading` → `active` (no process restart)
`systemctl reset-failed X`	`failed` → `inactive` (clears the failure latch)
`systemctl kill X`	`active` → `deactivating` (or `failed` if SIGKILL)

The two states the operator most often debugs are failed (read journalctl -u X --no-pager for the exit and reason) and a unit stuck in activating (likely a missing dependency, slow ExecStartPre, or Type=notify waiting for sd_notify(READY=1)).

Two more concepts worth naming up front:

Drop-ins, /etc/systemd/system/<unit>.d/*.conf snippets that override pieces of a vendor unit without copying the whole file. systemctl edit writes them.
cgroups, where every unit gets its own cgroup, so resource accounting (MemoryMax=, CPUQuota=) and kill semantics (“stop the whole tree”) are precise. systemd-cgls shows the tree.

systemctl#

systemctl is the front door for everything systemd. status to see what’s happening, start / stop / restart / reload to control, enable / disable to wire into boot, list-units and list-unit-files to enumerate.

Command	Effect
`systemctl status UNIT`	Show state, recent log lines, PID, memory
`sudo systemctl start UNIT`	Start now (does not survive reboot)
`sudo systemctl stop UNIT`	Stop now
`sudo systemctl restart UNIT`	Stop + start
`sudo systemctl reload UNIT`	Send the reload signal (no restart)
`sudo systemctl enable UNIT`	Wire into boot (creates `.wants/` symlink)
`sudo systemctl disable UNIT`	Remove from boot
`sudo systemctl enable --now UNIT`	Enable + start in one command
`systemctl is-active UNIT`	Exit 0 if running
`systemctl is-enabled UNIT`	Exit 0 if enabled
`systemctl list-units --type=service`	Every loaded service
`systemctl list-unit-files --state=enabled`	Every enabled unit

systemctl status shows state, the cgroup tree, recent log lines, and the PID:

$ systemctl status nginx

● nginx.service - A high performance web server and a reverse proxy server
     Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
     Active: active (running) since Wed 2026-04-30 09:14:02 UTC; 1 day 8h ago
    Process: 2807 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=exited, status=0/SUCCESS)
    Process: 2810 ExecStart=/usr/sbin/nginx -g daemon on; master_process on; (code=exited, status=0/SUCCESS)
   Main PID: 2811 (nginx)
      Tasks: 3 (limit: 9417)
     Memory: 8.2M
        CPU: 312ms

enable --now is the muscle-memory shortcut for “wire into boot and start it right now”:

$ sudo systemctl enable --now nginx

Created symlink /etc/systemd/system/multi-user.target.wants/nginx.service → /lib/systemd/system/nginx.service.

reload sends the configured reload signal (often SIGHUP) without dropping connections; preferred over restart when the service supports it:

$ sudo systemctl reload nginx

list-units --type=service is the system-wide service inventory:

$ systemctl list-units --type=service --state=running | head

UNIT                            LOAD   ACTIVE SUB     DESCRIPTION
nginx.service                   loaded active running A high performance web server and a reverse proxy server
ssh.service                     loaded active running OpenBSD Secure Shell server
systemd-journald.service        loaded active running Journal Service
systemd-logind.service          loaded active running User Login Management

Unit Files#

A unit file is INI-style configuration that tells systemd how to run something. .service is the default; other types include .timer, .socket, .mount, .target, .path. Operator-edited units belong in /etc/systemd/system/; distribution-shipped units live under /usr/lib/systemd/system/.

Section	Purpose
`[Unit]`	Description, dependencies (`After=`, `Wants=`)
`[Service]`	What to run, how, as whom, restart policy
`[Install]`	Where it goes when `enable`d (`WantedBy=`)

A minimal service. Restart on failure with a 5-second backoff, running as a dedicated user, started after the network is up:

[Unit]
Description=My App
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
User=myapp
ExecStart=/usr/local/bin/myapp --config /etc/myapp/config.toml
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target

After dropping the file in /etc/systemd/system/myapp.service, reload systemd to pick it up, then enable + start:

$ sudo systemctl daemon-reload
$ sudo systemctl enable --now myapp

systemctl cat prints the resolved unit file (vendor + drop-ins merged); show prints every property:

$ systemctl cat nginx

# /lib/systemd/system/nginx.service
[Unit]
Description=A high performance web server and a reverse proxy server
Documentation=man:nginx(8)
After=network-online.target remote-fs.target nss-lookup.target
Wants=network-online.target

[Service]
Type=forking
...

systemd-analyze verify is the lint step before reloading:

$ systemd-analyze verify /etc/systemd/system/myapp.service

Drop-ins (overrides)#

A drop-in is a fragment in /etc/systemd/system/<unit>.d/*.conf that overrides or extends an existing unit, the right way to tweak a vendor service without editing the shipped file. systemctl edit writes the drop-in for you and reloads.

Command	Effect
`sudo systemctl edit UNIT`	Open the drop-in editor (creates `override.conf`)
`sudo systemctl edit --full UNIT`	Edit a full copy of the unit (less common)
`systemctl revert UNIT`	Drop all overrides for `UNIT`

systemctl edit opens a stub for override.conf in $EDITOR. Add only the section you want to change:

$ sudo systemctl edit nginx

[Service]
LimitNOFILE=65535
Environment=NGINX_DEBUG=1

After save, systemd reloads automatically; restart the service to pick up the change:

$ sudo systemctl restart nginx

Timers#

A timer unit fires another unit on a schedule, the systemd replacement for cron. Pair a .service (what to run) with a .timer (when to run it) of the same base name.

File / command	Role
`backup.service`	The work (`Type=oneshot` is typical)
`backup.timer`	The schedule (`OnCalendar=` or `OnBootSec=`)
`systemctl list-timers`	Next-fire / last-fire across all timers
`systemctl status backup.timer`	Inspect one timer
`Persistent=true`	Catch up missed runs after downtime

A daily backup pair:

[Service]
Type=oneshot
ExecStart=/usr/local/bin/backup.sh

[Unit]
Description=Daily backup

[Timer]
OnCalendar=daily
Persistent=true

[Install]
WantedBy=timers.target

Enable the timer (not the service) so the schedule activates at boot:

$ sudo systemctl enable --now backup.timer
$ systemctl list-timers --all

NEXT                        LEFT          LAST                        PASSED       UNIT          ACTIVATES
Sat 2026-05-03 00:00:00 UTC 7h left       Fri 2026-05-02 00:00:00 UTC 16h ago      backup.timer  backup.service

journalctl#

The journal is systemd’s structured, indexed log store (binary on disk, queryable on the way out). -u filters by unit, -f follows live, --since and --until bound a window.

Command	Effect
`journalctl`	All entries (newest at the bottom)
`journalctl -u UNIT`	Entries from one unit
`journalctl -u UNIT -f`	Live tail for one unit
`journalctl -u UNIT --since '1 hour ago'`	Time-bounded
`journalctl -p err`	Filter by priority (`emerg`..``debug``)
`journalctl -k`	Kernel log only (`dmesg` equivalent)
`journalctl -b`	Current boot only; `-b -1` for previous
`journalctl --disk-usage`	How much disk the journal is using
`journalctl --vacuum-time=14d`	Delete entries older than 14 days

Live-tail one service while you change configuration:

$ journalctl -u nginx -f

May 02 09:14:02 web nginx[2811]: 2026/05/02 09:14:02 [notice] 2811#2811: signal process started
May 02 09:14:03 web nginx[2811]: 2026/05/02 09:14:03 [notice] 2811#2811: reloading configuration

Errors only, last hour:

$ journalctl -p err --since '1 hour ago'

Trim the journal when disk pressure hits:

$ sudo journalctl --vacuum-size=500M

Targets and boot#

A target is a synchronisation point, the systemd replacement for SysV runlevels. multi-user.target is the typical server boot target; graphical.target adds a display manager; rescue.target and emergency.target are recovery shells.

Target	When you reach it
`multi-user.target`	Non-graphical multi-user; default on servers
`graphical.target`	Multi-user + display manager
`rescue.target`	Single-user with most of the system online
`emergency.target`	Single-user with only `/` mounted (last resort)
`reboot.target` / `poweroff.target`	Reboot / shut down

Show the default boot target:

$ systemctl get-default

multi-user.target

Switch to a different default permanently:

$ sudo systemctl set-default multi-user.target

Drop into rescue mode without rebooting (kicks out other users):

$ sudo systemctl isolate rescue.target

systemd-analyze blame ranks units by how long they took to start at boot:

$ systemd-analyze blame | head

2.412s NetworkManager-wait-online.service
 812ms snapd.seeded.service
 410ms cloud-init.service
 312ms systemd-journal-flush.service

User services#

Services that run only while a user is logged in belong in the user-scoped systemd instance. Same unit-file syntax as system services, but located under ~/.config/systemd/user/ and managed with the --user flag on every systemctl and journalctl invocation.

Command	Effect
`systemctl --user start UNIT`	Start in the user instance
`systemctl --user enable --now UNIT`	Enable + start
`journalctl --user -u UNIT`	Logs for a user-scoped unit
`loginctl enable-linger USER`	Keep the user instance alive past logout

Drop the unit file in ~/.config/systemd/user/my-agent.service, then enable + start:

$ systemctl --user enable --now my-agent.service
$ journalctl --user -u my-agent -f

If the user logs out and the service should keep running, enable lingering (otherwise systemd kills the user instance at logout):

$ sudo loginctl enable-linger $(whoami)

Common Tasks#

Map the unit landscape (what is enabled, what is running, what is failing).

$ systemctl list-units --type=service --state=running
$ systemctl list-unit-files --state=enabled
$ systemctl --failed

Hunt persistence. User units, drop-ins, and timers are common quiet places.

$ systemctl --user list-unit-files --state=enabled
$ ls -la /etc/systemd/system/*.d/ 2>/dev/null
$ systemctl list-timers --all
$ find /etc/systemd /lib/systemd /usr/lib/systemd -name '*.service' -newer /etc/hostname

Read a unit end to end (the merged view, including drop-ins).

$ systemctl cat <unit>
$ systemctl show <unit>
$ systemctl status <unit>

Pull logs cleanly (by unit, by time, by priority).

$ journalctl -u <unit> -n 200 --no-pager
$ journalctl -u <unit> --since '1 hour ago'
$ journalctl -p err -b
$ journalctl -k -b

Toggle without rebooting (start, stop, mask, reload).

$ sudo systemctl restart <unit>
$ sudo systemctl mask <unit>
$ sudo systemctl daemon-reload

Audit boot impact (which units cost the boot).

$ systemd-analyze
$ systemd-analyze blame | head -20
$ systemd-analyze critical-chain

Enable or disable on boot (distinct from start/stop now).

$ sudo systemctl enable --now <unit>
$ sudo systemctl disable --now <unit>
$ systemctl is-enabled <unit>; systemctl is-active <unit>

Reload after editing a unit (pick up changes without reboot).

$ sudo systemctl daemon-reload
$ sudo systemctl reload <unit>           # SIGHUP, no restart
$ sudo systemctl reload-or-restart <unit>

Edit a unit safely. Drop-ins beat full overrides.

$ sudo systemctl edit <unit>             # creates override.conf
$ sudo systemctl edit --full <unit>      # copy-and-modify
$ sudo systemctl revert <unit>

Create a one-off service or timer (minimum-viable unit).

$ sudo tee /etc/systemd/system/myjob.service <<'EOF'
[Unit]
Description=My job
[Service]
Type=oneshot
ExecStart=/usr/local/bin/myjob.sh
[Install]
WantedBy=multi-user.target
EOF
$ sudo systemctl daemon-reload && sudo systemctl enable --now myjob.service

Manage timers (cron replacement). List, run now, debug.

$ systemctl list-timers --all
$ sudo systemctl start <unit>.timer
$ journalctl -u <unit>.service -n 50

Run as the user, not root (per-user systemd instance).

$ systemctl --user status
$ systemctl --user enable --now <unit>
$ loginctl enable-linger $USER          # let user units run after logout

Files & paths#

Everything systemd reads, writes, or persists on disk, for when the live tools aren’t enough. The unit search path, runtime state, persistent journal, and DNS resolver state all live as files, so the operator can grep, diff, and back them up like any other configuration.

Unit files (where systemd reads them)#

systemd searches these directories in order; the first match wins:

/etc/systemd/system/, operator overrides; edit here.
/run/systemd/system/, runtime-only units (volatile).
/usr/lib/systemd/system/, distribution-shipped units.
/lib/systemd/system/, legacy path; symlink to /usr/lib/systemd/system/ on most distros.

User-scoped (systemctl --user):

~/.config/systemd/user/, per-user overrides.
~/.local/share/systemd/user/, user-installed units.
/etc/systemd/user/, system-wide for all users.
/usr/lib/systemd/user/, distribution-shipped user units.

Drop-ins (override pieces of a unit)#

/etc/systemd/system/<unit>.d/*.conf, override fragments.
/etc/systemd/system/<unit>.d/override.conf, the file systemctl edit <unit> creates.

Targets and aliases#

/etc/systemd/system/<target>.target.wants/, “what’s enabled for this target” (symlinks).
/etc/systemd/system/multi-user.target.wants/, the typical server boot target.
/etc/systemd/system/default.target, symlink to the boot target (multi-user.target or graphical.target).

Configuration#

/etc/systemd/system.conf, defaults for system-mode units (DefaultLimitNOFILE, DefaultTimeoutStopSec, …).
/etc/systemd/user.conf, defaults for user-mode units.
/etc/systemd/journald.conf, journal storage / retention.
/etc/systemd/logind.conf, login / session policy.
/etc/systemd/timesyncd.conf, NTP config.
/etc/systemd/resolved.conf, resolver config.

Runtime / state#

/run/systemd/, live systemd state.
/run/systemd/units/, runtime unit data.
/var/lib/systemd/, persistent systemd data (timers’ last-trigger marks, coredumps).

Logs (the journal)#

/var/log/journal/, persistent binary journal (when Storage=persistent in journald.conf).
/run/log/journal/, runtime-only journal (default if /var/log/journal doesn’t exist).
/var/log/syslog (Debian) / messages (RHEL), traditional logs (also written when ForwardToSyslog=yes).

Other init systems (legacy)#

/etc/init.d/, SysV-style init scripts.
/etc/rc<level>.d/, runlevel symlinks.
/etc/inittab, pre-systemd init configuration.
/etc/sv/, runit service directories (Void Linux, Alpine optional).
/etc/init/, Upstart job files.

References#

man systemd.unit, man systemd.service (standard unit reference).
man journalctl (complete journal query syntax).
Processes for the per-process view of what services run.
Boot for where targets fit into the boot sequence.