Processes

Processes#

A process is the operator’s smallest unit of observation on a live host. Every running program carries a PID, a parent, an owning user, an environment, and a set of open file descriptors, and the kernel exposes all of it under /proc/<pid>/ for anyone with the right to read it. Reading processes is how the operator answers “what is this box doing right now,” whether the goal is hunting an implant, tracing a misbehaving service, or surveying the attack surface before moving.

The day-to-day toolkit is short: ps, top / htop, pgrep, pstree, kill, strace, lsof. On-disk state files at the bottom of the page are what to reach for when the live tools run out.

Every process is a kernel-tracked record. Each row below is a slice of that record, and each one has a file under /proc/<pid>/ an operator can read directly.

Field

What it is

PID

Numeric process id; unique while the process is alive

PPID

Parent’s PID; PID 1 (systemd / init) is the root

UID / GID

Owning user / group; real, effective, saved triples

Command

argv[0] + args at /proc/<pid>/cmdline (NUL-separated)

Executable

On-disk binary at /proc/<pid>/exe (symlink)

CWD

Current working directory at /proc/<pid>/cwd

Environment

Inherited env vars at /proc/<pid>/environ (NUL-separated)

File descs.

Open files / sockets / pipes at /proc/<pid>/fd/

Memory

VSZ / RSS / mappings at /proc/<pid>/maps, status

State

R / S / D / Z / T (run, sleep, uninterruptible, zombie, stopped)

Threads

Per-thread tasks at /proc/<pid>/task/<tid>/

Namespaces

Isolation handles at /proc/<pid>/ns/{pid,net,mnt,...}

Cgroup

Resource limits at /proc/<pid>/cgroup

Capabilities

Fine-grained privileges at /proc/<pid>/status (Cap*)

Inspecting#

ps is the snapshot tool, top / htop are the live views, pgrep finds by name, and pstree shows the parent-child relationships. ps aux is the most-typed incantation in Linux admin; memorize it.

Command

Effect

ps aux

BSD-style snapshot of every process

ps -ef

SysV-style snapshot (more detail)

ps -o ... --sort=...

Custom columns + sort

pgrep -af PATTERN

PIDs matching a name + full command line

pidof NAME

PID(s) for an exact program name

top

Live, full-screen view (built-in)

htop

Friendlier top (colors, mouse, tree mode)

pstree -p

Process tree with PIDs

ps aux lists every process on the system with USER, PID, %CPU, %MEM, VSZ, RSS, TTY, STAT, START, TIME, COMMAND:

$ ps aux | head

USER       PID %CPU %MEM    VSZ   RSS TTY  STAT START   TIME COMMAND
root         1  0.0  0.1 168196 13312 ?    Ss   Apr28   0:14 /sbin/init
root       412  0.0  0.0  10000  4608 ?    Ss   Apr28   0:00 /lib/systemd/systemd-journald
root      1023  0.0  0.0  72420  6400 ?    Ss   Apr28   0:00 /usr/sbin/sshd -D
operator  3091  0.2  0.4 720180 71040 pts/0 Sl  09:14   0:03 /usr/bin/python3 worker.py

-o picks columns; --sort orders the output. Top memory hogs in one line:

$ ps -o pid,user,%mem,rss,cmd --sort=-%mem | head -5

PID USER     %MEM    RSS CMD
3091 operator  0.4  71040 /usr/bin/python3 worker.py
1023 root      0.0   6400 /usr/sbin/sshd -D

pgrep -a prints PID + full command line for everything matching a regex; -f matches against the full command, not just the name:

$ pgrep -af nginx

2811 nginx: master process /usr/sbin/nginx -g daemon on; master_process on;
2812 nginx: worker process

htop is a full-screen, scrollable, color view; F5 toggles the tree, F9 sends a signal:

$ htop

pstree -p shows parent-child links with PIDs in parentheses, useful when one binary forks many children:

$ pstree -p $(pgrep -o nginx)

nginx(2811)─┬─nginx(2812)
            ├─nginx(2813)
            └─nginx(2814)

Starting and Stopping#

A foreground command holds the terminal until it finishes; & backgrounds it; nohup and disown keep it alive past logout. kill sends a signal (SIGTERM by default), pkill / killall send by name.

Command

Effect

./prog

Run in the foreground

./prog &

Run in the background

nohup ./prog &

Background + survive logout (logs to nohup.out)

disown %1

Detach a backgrounded job from the shell

jobs

List the shell’s background jobs

fg %N / bg %N

Foreground / background job N

kill PID

Send SIGTERM (graceful)

kill -9 PID

Send SIGKILL (forced)

pkill -f PATTERN

Kill every process matching the regex

killall NAME

Kill by exact program name

Ctrl-Z suspends a foreground job; bg resumes it in the background; fg brings it back:

$ ./long-running-task
^Z
[1]+  Stopped   ./long-running-task
$ bg %1
$ jobs

[1]+  Running   ./long-running-task &

nohup redirects output to nohup.out so the process survives SIGHUP when the terminal closes; disown then removes the shell’s record so it isn’t killed at logout:

$ nohup ./worker.py > worker.log 2>&1 &
$ disown %1

kill PID defaults to SIGTERM (graceful and trappable). kill -9 is the nuclear option:

$ kill 3091
$ kill -9 3091

pkill matches by command line with -f; very useful when the binary is wrapped:

$ pkill -f 'python3 worker.py'

Signals#

Signals are the Unix IPC mechanism for “hey, do something.” The kernel delivers a number to a process, and the process either runs a handler, takes the default action (often exit or core dump), or ignores it. A few signals matter day-to-day; the rest are application-defined. Signal numbers and default actions are POSIX-defined (POSIX.1 <signal.h>), so the same names and behaviors hold across Linux, BSD, and macOS, with Linux extending the real-time range SIGRTMIN..``SIGRTMAX``.

Signal

Meaning

SIGTERM (15)

Graceful termination; default for kill

SIGKILL (9)

Forced termination; cannot be caught or ignored

SIGINT (2)

Interrupt (Ctrl-C)

SIGHUP (1)

Terminal hangup; many daemons reload config

SIGSTOP / SIGCONT

Pause / resume (SIGSTOP cannot be caught)

SIGTSTP (20)

Terminal stop from Ctrl-Z (catchable, unlike STOP)

SIGCHLD (17)

A child changed state; supervisors must wait()

SIGPIPE (13)

Wrote to a pipe with no reader (common in shell pipelines)

SIGQUIT (3)

Ctrl-\; like SIGINT but dumps core

SIGUSR1 / SIGUSR2

Application-defined (e.g. logrotate reload trigger)

SIGSEGV (11)

Segmentation fault; invalid memory access

SIGABRT (6)

abort(3) / assert(3); default action is core

SIGWINCH (28)

Terminal window-size change (tmux, less, readline)

SIGALRM (14)

alarm(2) / setitimer(2) timer expired

SIGBUS (7)

Bus error (misaligned access, mmap past EOF)

SIGFPE (8)

Arithmetic exception (divide by zero, etc.)

SIGILL (4)

Illegal instruction

SIGSYS (31)

Bad system call (where seccomp violations land)

SIGTTIN / SIGTTOU

Background job tried to read / write the controlling tty

SIGRTMIN to SIGRTMAX

Real-time signals (34 to 64); queued, app-defined

kill -HUP is the conventional “reload your config” signal for daemons (nginx, sshd, rsyslog, …):

$ kill -HUP $(pgrep -o nginx)

kill -l lists all signals the kernel knows:

$ kill -l | head -3

 1) SIGHUP    2) SIGINT       3) SIGQUIT      4) SIGILL       5) SIGTRAP
 6) SIGABRT   7) SIGBUS       8) SIGFPE       9) SIGKILL     10) SIGUSR1
11) SIGSEGV  12) SIGUSR2     13) SIGPIPE     14) SIGALRM     15) SIGTERM

Resource Limits#

Every process inherits rlimits, soft caps on file descriptors, address space, CPU time, core dumps, etc. Read with ulimit, set in /etc/security/limits.conf for login sessions or in a systemd unit’s LimitNOFILE= for a service.

Command / file

Effect

ulimit -a

Show this shell’s effective limits

ulimit -n 65535

Raise open-file limit (this shell)

ulimit -c unlimited

Allow core dumps

/etc/security/limits.conf

Persistent per-user / per-group limits

/etc/systemd/system.conf

Defaults for systemd-managed services

LimitNOFILE=65535

Per-unit override in a [Service] block

ulimit -a shows everything the current shell will pass to its children:

$ ulimit -a

open files                      (-n) 1024
max user processes              (-u) 31621
virtual memory          (kbytes, -v) unlimited
core file size          (blocks, -c) 0

A common production change: bump the per-process FD limit so a high-traffic daemon doesn’t EMFILE:

$ ulimit -n 65535
$ ulimit -n

65535

Tracing and Profiling#

Trace what a process is doing without source code or a debugger attached. strace shows syscalls, ltrace shows library calls, perf profiles CPU usage and hot paths, and lsof lists every file, socket, and pipe the process has open.

Command

Effect

strace -p PID

Attach and trace syscalls

strace -e openat ./prog

Filter to a single syscall

strace -f -o out.txt ./prog

Trace child processes; log to file

ltrace ./prog

Trace library calls

perf top

Live CPU profiler

perf record / perf report

Sample then analyze

lsof -p PID

Files opened by PID

lsof -i :PORT

Process listening on a TCP / UDP port

strace attached to a running process; Ctrl-C to detach:

$ sudo strace -p 2811

strace: Process 2811 attached
epoll_wait(8, [{events=EPOLLIN, data={u32=12, u64=12}}], 512, -1) = 1
accept4(12, {sa_family=AF_INET, sin_port=htons(54221), sin_addr=inet_addr("10.0.0.5")}, [16], SOCK_NONBLOCK) = 14

strace -e filters to one syscall family, great for finding which file a program tried to open:

$ strace -e openat ls / 2>&1 | tail -5

lsof -i :PORT is the “who’s holding port X” query:

$ sudo lsof -i :443

COMMAND  PID  USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
nginx   2811  root    7u  IPv4  18723      0t0  TCP *:https (LISTEN)
nginx   2812 nginx    7u  IPv4  18723      0t0  TCP *:https (LISTEN)

On-disk state#

Process state lives in the kernel’s virtual filesystems (/proc and /sys) and a small set of configuration files. The operator reads these directly when the live tools aren’t enough, usually to dig into limits, mappings, or scheduling details that ps and top flatten out.

Per-process state (/proc/<pid>/)#

  • /proc/<pid>/cmdline, launch command (NUL-separated).

  • /proc/<pid>/comm, short process name.

  • /proc/<pid>/status, human-readable summary (state, threads, uids, capabilities, memory).

  • /proc/<pid>/stat / statm, machine-readable variants.

  • /proc/<pid>/environ, environment variables (NUL-separated).

  • /proc/<pid>/exe, symlink to the executable.

  • /proc/<pid>/cwd, symlink to current working directory.

  • /proc/<pid>/root, symlink to its root directory (chroot-aware).

  • /proc/<pid>/fd/, one symlink per open file descriptor.

  • /proc/<pid>/fdinfo/, per-fd metadata (offset, flags).

  • /proc/<pid>/maps, memory map (binaries, shared libs, heap, stack).

  • /proc/<pid>/smaps, per-mapping memory stats (PSS, RSS).

  • /proc/<pid>/limits, effective rlimits.

  • /proc/<pid>/ns/, namespaces (mnt, pid, net, user, …).

  • /proc/<pid>/cgroup, cgroup memberships.

  • /proc/<pid>/oom_score / oom_adj, OOM-killer scoring.

  • /proc/<pid>/io, read / write byte counts.

  • /proc/<pid>/mountinfo, the process’s view of mounts.

  • /proc/<pid>/sched, scheduler statistics.

Self-references#

  • /proc/self/, “this process”; the kernel resolves it per-reader.

  • /proc/thread-self/, “this thread”.

Global process / kernel state#

  • /proc/cpuinfo, CPU topology and flags.

  • /proc/meminfo, memory statistics.

  • /proc/loadavg, 1 / 5 / 15-minute load average + last PID.

  • /proc/uptime, seconds since boot, idle time.

  • /proc/stat, CPU time, context switches, fork count.

  • /proc/swaps, active swap areas.

  • /proc/sysrq-trigger, magic SysRq.

Tunables#

  • /proc/sys/kernel/, kernel parameters (e.g. pid_max, threads-max, core_pattern).

  • /proc/sys/fs/, filesystem-related (file-max, inotify/max_user_watches).

  • /proc/sys/vm/, memory tuning (swappiness, overcommit_memory).

  • /etc/sysctl.conf / /etc/sysctl.d/*.conf, persisted tunings.

Cgroups (resource control)#

  • /sys/fs/cgroup/, cgroup v2 unified hierarchy (modern systems).

  • /sys/fs/cgroup/<slice>/<unit>/, per-service controllers.

  • /proc/cgroups, enabled controllers.

PID files and runtime#

  • /run/, runtime state, cleared at boot.

  • /run/<service>.pid, conventional PID file location.

  • /run/user/<uid>/, per-user runtime (XDG_RUNTIME_DIR).

Useful peeks#

$$ is the current shell’s PID; the next two commands show this shell’s status and the executable behind it:

$ cat /proc/$$/status | head -5
$ readlink /proc/$$/exe

Name:        bash
Umask:       0022
State:       S (sleeping)
Tgid:        3091
Ngid:        0
/usr/bin/bash

cmdline and environ use NUL separators, so pipe through tr to make them readable:

$ cat /proc/<pid>/cmdline | tr '\0' ' '
$ cat /proc/<pid>/environ | tr '\0' '\n'

pmap summarises /proc/<pid>/maps in one line per mapping:

$ pmap $(pgrep -o nginx) | tail

Common Tasks#

Inventory. Snapshot every running process and the user it runs as.

$ ps -eo pid,user,ppid,cmd --sort=user
$ ps auxf
$ pstree -p

Hunt anomalies. Find processes that should not be there.

$ ls -l /proc/*/exe 2>/dev/null | grep deleted
$ ls -l /proc/*/exe 2>/dev/null | grep -E '/tmp|/dev/shm'
$ ps -eo pid,user,cmd | awk '$2=="root"'

Trace a suspect. Watch what a single process is doing now.

$ strace -f -p <pid>
$ lsof -p <pid>
$ cat /proc/<pid>/environ | tr '\0' '\n'
$ cat /proc/<pid>/maps | head

Pivot via parent / child. A surprise parent is a finding.

$ cat /proc/<pid>/status | grep -E 'PPid|Uid|Gid'
$ pgrep -af <name> | xargs -I{} echo {}
$ ps -o pid,ppid,user,cmd --ppid <ppid>

Check resource limits. Raised limits are unusual.

$ cat /proc/<pid>/limits
$ prlimit --pid <pid>
$ ulimit -a

Control without killing. Pause for inspection, resume cleanly.

$ kill -STOP <pid>
$ kill -CONT <pid>
$ renice +10 -p <pid>
$ ionice -c 3 -p <pid>

Find the resource hog (CPU, RAM, I/O).

$ ps -eo pid,user,%cpu,%mem,cmd --sort=-%cpu | head
$ ps -eo pid,user,%mem,rss,cmd --sort=-%mem | head
$ top -b -n1 | head -20
$ sudo iotop -boqq -n 3 2>/dev/null

Find the process bound to a port or file. Who has it open.

$ sudo ss -tulnp | grep :443
$ sudo lsof -iTCP:443 -sTCP:LISTEN -P -n
$ sudo lsof /var/log/syslog
$ sudo fuser -v /var/log/syslog

Kill cleanly, then forcefully. Escalate signals.

$ kill <pid>            # SIGTERM, give it a chance
$ kill -INT <pid>       # like Ctrl-C
$ kill -HUP <pid>       # reload config
$ kill -KILL <pid>      # last resort
$ pkill -f 'pattern'; killall -9 nginx

Run a long job that survives logout. Background, detach, schedule.

$ nohup ./job.sh >run.log 2>&1 &
$ disown -h %1
$ setsid -f ./job.sh
$ at now + 10 minutes <<<'/usr/local/bin/job.sh'

Inspect open files and sockets. A process’s footprint.

$ ls -l /proc/<pid>/fd/ | head
$ lsof -p <pid>
$ ss -tnp | grep <pid>

References#

  • man 7 signal (complete signal list).

  • man 5 proc (standard reference for /proc).

  • Services for systemd as the supervisor of long-running processes.

  • Permissions for users, capabilities, sudo for what a process can do.

Contents