Memory

Memory#

Linux gives every process a private virtual address space and serves it from physical RAM through paging: virtual pages map to physical frames via per-process page tables, and the kernel evicts cold pages to swap when RAM runs short. The operator’s view covers the anatomy of a process’s memory, page faults, swap, the OOM killer, huge pages, and cgroup memory controls.

Memory has three distinct anatomies the operator must hold in mind at once. The hardware (DRAM chips, caches, MMU, TLB). The kernel (zones, allocators, page cache, slab) that owns and hands out physical frames. And the per-process virtual address space the program actually sees. The kernel sits in the middle, mediating, accounting, and presenting each process its own private map of the same physical RAM.

Hardware#

Physical memory on a modern x86_64 box is a hierarchy where each tier is faster, smaller, and closer to the CPU than the one below it. Latencies span seven orders of magnitude from a register to a spinning-disk swap fault.

Tier	Typical size	Latency (cycles)	Notes
Registers	<1 KB / core	0	Named (`rax`, `xmm0`, `ymm0`, …)
L1 cache	32-64 KB	4-5	Per-core; split I-cache + D-cache
L2 cache	256 KB-1 MB	12-15	Per-core
L3 cache	8-128 MB	40-75	Shared across cores in a socket
DRAM (local)	GB-TB	~300 (~80-100 ns)	Same NUMA node as the CPU
DRAM (remote)	GB-TB	~600 (~200 ns)	Other socket, over UPI / Infinity Fabric
Swap (NVMe)	GB-TB	~10⁵	10-100 µs per page in
Swap (HDD)	GB-TB	~10⁷	Why thrashing kills latency

The off-chip components.

DIMMs (UDIMM / RDIMM / LRDIMM) plug into channels driven by the CPU’s integrated memory controller (IMC). Each DIMM has one or more ranks (groups of chips operated together), divided into banks and rows. Modern desktops are dual- or quad-channel; servers reach 8 or 12 channels (DDR5 / EPYC, Xeon).
ECC (Error-Correcting Code) DIMMs add a parity chip to detect and correct single-bit flips; required on servers, optional on consumer parts. edac-util / rasdaemon report errors.
NUMA (Non-Uniform Memory Access): each socket has its own local DRAM. Cross-socket access is 2-3× slower. numactl --hardware shows the topology; /sys/devices/system/node/ exposes per-node stats.

The on-chip components the operator can’t see directly but whose behavior shapes everything.

Cache lines, the transfer unit between cache levels. 64 B on x86_64, 128 B on Apple M-series. Two threads writing different fields of the same line (“false sharing”) get serialised through cache coherency.
MMU (Memory Management Unit), translates virtual→physical on every load / store via the page tables.
TLB (Translation Lookaside Buffer), caches recent translations. A miss costs a page-table walk (4 or 5 levels on x86_64; see Pages and frames). Huge pages reduce TLB pressure.
IOMMU, the DMA equivalent of the MMU; sandboxes device access to RAM. dmesg | grep -i iommu.

Inspect the hardware.

$ sudo dmidecode -t memory                 # DIMM inventory
$ lscpu | grep -i 'cache\|numa'            # cache + NUMA summary
$ numactl --hardware                       # NUMA nodes + distance matrix
$ cat /proc/cpuinfo | grep -i 'cache size'

Kernel#

The kernel owns physical RAM. It splits it into zones based on hardware constraints, hands out pages via a buddy allocator, and caches small kernel objects with slab on top. Userspace only ever sees physical RAM through the kernel’s translation layer.

Component	Role
Zones	`ZONE_DMA` (legacy 16 MB), `ZONE_DMA32` (4 GB), `ZONE_NORMAL`, `ZONE_MOVABLE` partition physical pages by which devices can address them.
Buddy allocator	Manages free pages in power-of-two blocks (orders 0-10, i.e. 4 KB-4 MB). Coalesces freed pairs (“buddies”) to fight fragmentation. `/proc/buddyinfo`.
Slab / SLUB	Caches kernel objects (`task_struct`, `inode`, `dentry`, sockets…) on top of buddy pages. `/proc/slabinfo`, `slabtop`.
Page cache	File-backed pages held after a read; what `free` reports under `buff/cache`. Dropped on demand to satisfy allocations.
Anon LRU / File LRU	Two reclaim lists. Anonymous pages (heap, stack, anon mmap) go to swap when evicted; file-backed clean pages are simply dropped, dirty ones written back first.
vmalloc / kmap	Kernel-side virtual mappings for large allocations or non-contiguous physical pages. `/proc/vmallocinfo`.
Direct map	Linear kernel mapping of all physical RAM, how the kernel touches userspace pages without walking page tables.

The most common operator confusion. free shows almost no “free” RAM on a healthy system, because the kernel uses anything not held by a process as page cache. The honest answer is the available column, which counts reclaimable cache as available. “Free RAM is wasted RAM.”

$ free -h                                  # mind the `available` column
$ cat /proc/buddyinfo                      # free pages per order, per zone
$ slabtop -o | head -20                    # which kernel object is hot
$ sync && echo 3 | sudo tee /proc/sys/vm/drop_caches    # force-drop cache (test only)

Address space#

A process’s virtual address space is a sparse map of regions called VMAs (Virtual Memory Areas), each backed by either anonymous memory (heap, stack, anon mmap) or a file (mapped binary, library, mmap’d data). The kernel allocates physical frames lazily, on the first touch, not at mmap() time.

        block-beta
    columns 2
    hi["0xffff…"]:1 hi_lbl["high address"]:1
    space1["⋮"]:2
    k["kernel mappings"]:1 k_note["inaccessible from userspace"]:1
    space2["⋮"]:2
    s["stack ↓"]:1 s_note["anonymous, rw-p"]:1
    space3["⋮"]:2
    lib["mmap'd libraries"]:1 lib_note["file-backed: .so r-xp + rw-p"]:1
    mm["mmap region"]:1 mm_note["anonymous + file-backed"]:1
    space4["⋮"]:2
    h["heap ↑ (brk)"]:1 h_note["anonymous, lazily allocated"]:1
    bss["BSS"]:1 bss_note["zero-init data, anonymous, lazy"]:1
    d["data"]:1 d_note["init globals, file-backed"]:1
    t["text"]:1 t_note["machine code, file-backed, r-xp"]:1
    space5["⋮"]:2
    lo["0x000000"]:1 lo_lbl["low address"]:1

    classDef kernel  fill:#D13212,stroke:#a82710,color:#fff;
    classDef anon    fill:#8C4FFF,stroke:#6b3acc,color:#fff;
    classDef file    fill:#3F8624,stroke:#2b5e19,color:#fff;
    classDef mixed   fill:#3B48CC,stroke:#2a36a3,color:#fff;
    classDef code    fill:#FF9900,stroke:#cc7700,color:#fff;
    classDef bound   fill:none,stroke:none,color:#888;
    classDef gap     fill:none,stroke:none,color:#888;
    classDef note    fill:none,stroke:none,color:inherit;

    class k kernel;
    class s,h,bss anon;
    class lib,d file;
    class mm mixed;
    class t code;
    class hi,lo,hi_lbl,lo_lbl bound;
    class space1,space2,space3,space4,space5 gap;
    class k_note,s_note,lib_note,mm_note,h_note,bss_note,d_note,t_note note;

Two orthogonal properties apply to every VMA.

Anonymous vs file-backed. Anonymous pages have no on-disk origin and go to swap if evicted; file-backed pages can be re-read from the source file, so clean ones cost nothing to drop.
Private vs shared. A private mapping is copy-on-write; the process gets its own physical copy on the first write. A shared mapping propagates writes to other mappers, and (for file-backed shared) back to disk.

What you see in /proc/<pid>/maps.

Region	maps clue
Text	Path to the binary, `r-xp` permission
Data	Same path, `rw-p` permission
BSS / heap	`[heap]`, anonymous, `rw-p`
Stack (main thread)	`[stack]`, anonymous, `rw-p` (grows down)
Stack (other threads)	`[stack:<tid>]` or unnamed anonymous block
mmap’d library	Path to `.so`; multiple regions per perms
mmap’d file	Path, `r--p` or `rw-p`
Anonymous mmap	No path; `rw-p` (or copy-on-write)
vDSO / vvar	`[vdso]` / `[vvar]` (kernel-supplied helper pages)

Per-process accounting uses five numbers that all measure something different.

Metric	Meaning
VSZ	Virtual size; everything mapped, even pages never touched
RSS	Resident set size; pages actually in RAM right now
PSS	Proportional set size; shared pages divided across sharers
USS	Unique set size; pages only this process has (USS = PSS - shared)
Swap	Pages currently swapped out

VSZ is meaningless for “how much RAM does this use” (a process can mmap a terabyte file with no cost). RSS double-counts shared pages across processes; PSS is the honest cross-process number; USS is “how much RAM goes away if I kill this process.”

$ cat /proc/<pid>/smaps_rollup             # RSS, PSS, USS, Swap totals
$ smem -k -t -P chrome | head              # PSS-based ranking
$ pmap -X <pid>                            # per-VMA breakdown

Translation#

A page is a fixed chunk of virtual memory (getconf PAGESIZE; 4096 bytes on x86_64, 16 384 on Apple Silicon, configurable on ARM).
A frame is a same-sized chunk of physical RAM.
The page table maps virtual page → physical frame for one process, plus permission and presence bits.
The MMU translates on every memory access; the TLB caches the hottest translations. A TLB miss costs a page-table walk (4-5 levels on x86_64 with 5-level paging).

x86_64 page-table walk (5-level since 2018). Bit 0 sits on the right; the page offset indexes the byte inside the chosen frame, each 9-bit field indexes one level of the radix tree. 4-level paging (48-bit) drops P4D and stops at bit 47.

        ---
title: "57-bit virtual address (5-level paging; 48-bit drops P4D)"
---
packet-beta
    0-11: "page offset (12)"
    12-20: "PTE (9)"
    21-29: "PMD (9)"
    30-38: "PUD (9)"
    39-47: "P4D (9)"
    48-56: "PGD (9)"

Page faults#

When the CPU dereferences a virtual address whose page-table entry is missing (or whose access type is denied), the MMU raises a page fault trap. The kernel decides what to do.

Type	What happens
Minor fault	PTE missing but the data is in RAM (shared library already loaded; copy-on-write parent page). Kernel just installs the PTE. Cheap.
Major fault	PTE missing AND data must come from disk (swap or mmap’d file). Kernel reads, updates PTE, returns. ~10⁵× slower than a minor fault.
Soft fault	Synonym for minor.
Hard fault	Synonym for major.
Protection	Permission violation (write to read-only page); usually `SIGSEGV` unless caught (e.g. JIT pages).

Per-process counters.

$ ps -o pid,min_flt,maj_flt,cmd -p <pid>
$ pidstat -r 1                                  # min / maj faults per second
$ awk '{print $10, $12}' /proc/<pid>/stat       # cumulative min, maj

Mechanisms#

How the kernel handles memory pressure and lays out physical RAM, covering when pages must leave memory (swap), when allocation can no longer be served (OOM killer), how containers cap usage (cgroups), and how to cut TLB cost on large workloads (huge pages).

Swap#

Pages the kernel evicts from RAM go to swap (a swap partition, swap file, or in-RAM compressed pool). Reading them back is what makes a thrashing system feel slow.

Knob	Effect
`vm.swappiness`	0-200 (default 60); higher = swap anonymous pages sooner
`vm.overcommit_memory`	0 heuristic / 1 always allow / 2 strict (commit ≤ RAM + swap×ratio)
`vm.overcommit_ratio`	Used with `overcommit_memory=2`
`zswap.enabled`	Compressed in-RAM swap cache; defers actual disk swap
`zram`	Block device backed by compressed RAM; `swapon` it

$ swapon --show
$ cat /proc/swaps
$ sysctl vm.swappiness
$ sudo swapoff /swapfile && sudo swapon /swapfile     # cycle to drain

OOM killer#

When userspace can’t get a page, the kernel triggers the out-of-memory killer. It scores every process and SIGKILL-s the worst offender.

File / sysctl	Effect
`/proc/<pid>/oom_score`	Computed score; higher = more likely victim
`/proc/<pid>/oom_score_adj`	Operator override; -1000..+1000 (-1000 = never kill)
`vm.panic_on_oom`	0 invoke killer / 1 panic / 2 panic in cpuset
`vm.oom_kill_allocating_task`	Kill the task that triggered the OOM (less optimal but faster)
`MemoryMax` cgroup limit	Cgroup-local OOM fires before the global one

$ dmesg -T | grep -i 'killed process'
$ journalctl -kb | grep -i 'oom-killer'
$ echo -1000 | sudo tee /proc/<pid>/oom_score_adj    # protect a critical pid

cgroups#

cgroup v2 memory.* controllers cap how much RAM and swap a slice or container can use. The kernel applies the same OOM logic locally when the slice hits its limit, before global OOM kicks in.

Knob	Meaning
`memory.max`	Hard limit; tasks killed on overrun
`memory.high`	Soft limit; reclaim + throttle, don’t kill
`memory.low`	Reserve; protected from reclaim under contention
`memory.swap.max`	Cap how much swap the cgroup can use
`memory.current`	Current RAM use
`memory.events`	Counters: `low`, `high`, `max`, `oom`, `oom_kill`

$ cat /sys/fs/cgroup/system.slice/<service>.service/memory.max
$ cat /sys/fs/cgroup/<slice>/memory.events
$ systemd-run --scope -p MemoryMax=512M -p MemorySwapMax=0 ./hog

Huge pages#

Standard pages are 4 KB; huge pages are 2 MB or 1 GB and reduce TLB pressure on workloads with a large working set (databases, JVMs, DPDK).

Type	Behavior
HugeTLB	Operator-reserved pool; processes mmap with `MAP_HUGETLB`
Transparent (THP)	Kernel auto-promotes anonymous regions; `always` / `madvise` / `never`

$ cat /proc/meminfo | grep -i huge
$ cat /sys/kernel/mm/transparent_hugepage/enabled
$ echo madvise | sudo tee /sys/kernel/mm/transparent_hugepage/enabled

# reserve 1024 × 2 MB hugetlb pages
$ echo 1024 | sudo tee /proc/sys/vm/nr_hugepages

Hardening#

Linux ships several memory-related mitigations the operator should verify, not assume. Most are runtime-tunable via sysctl; a few are baked in at compile or boot time.

User-space mitigations.

Knob	Effect
`kernel.randomize_va_space`	`0` off / `1` libs+stack / `2` full ASLR (default)
`kernel.yama.ptrace_scope`	`0` classic / `1` parent-only (default) / `2` admin / `3` off
`vm.mmap_min_addr`	Lowest mmap addr; blocks NULL-deref → kernel exec
`fs.suid_dumpable`	`0` no core for setuid (default); `2` root-only readable
`RLIMIT_MEMLOCK`	Per-process cap on `mlock()`-able pages (`ulimit -l`)

Kernel-side leakage mitigations.

Knob	Effect
`kernel.kptr_restrict`	`0` show kernel pointers / `1` to root / `2` always hide
`kernel.dmesg_restrict`	`1` = root-only `dmesg` (default on hardened kernels)
`kernel.unprivileged_bpf_disabled`	`1` = block non-root eBPF (closes a kernel-attack class)
`kernel.lockdown`	`integrity` / `confidentiality` restrict root’s reach into kernel memory (`/dev/mem`, kexec, etc.)
KASLR (boot)	Default on; disable with `nokaslr` cmdline (debug only)

Per-binary mitigations are baked in at compile time. Check with checksec.

Property	Meaning
PIE	Position-independent (ASLR can randomise the binary itself)
RELRO	`full` makes the GOT read-only after relocation
Canary	Stack-smash detector value before the saved RIP
NX	Stack and heap marked non-executable (`GNU_STACK` `RW`)
FORTIFY	`_FORTIFY_SOURCE` adds bounds-checked libc wrappers

$ checksec --file=/usr/bin/sshd
$ sysctl -a 2>/dev/null | grep -E 'randomize_va|kptr_restrict|dmesg_restrict|ptrace_scope'
$ ulimit -l                                # current memlock limit
$ # protect a secret in your own program: mlockall(MCL_CURRENT|MCL_FUTURE)

Forensics#

Memory acquisition captures volatile state, credentials, keys, process trees, network connections, malware payloads, all gone after a reboot. Two scopes apply, per-process and whole-system.

Per-process.

Tool / path	Use
`gcore <pid>`	Snapshot a process to an ELF core (gdb wrapper); safest
`coredumpctl dump <pid>`	Pull from systemd-coredump; preserves auxv
`/proc/<pid>/mem`	Raw bytes; needs `CAP_SYS_PTRACE` and ptrace_scope ≤ 1
`/proc/<pid>/maps`	Region map; script `dd` per region for selective dump
`ptrace(PTRACE_ATTACH)`	Underlying primitive both `gdb` and `gcore` use

Whole-system (live RAM acquisition).

Tool	Notes
LiME	Loadable kernel module; `format=lime` for Volatility input
AVML	Microsoft, no kmod required (uses `/proc/kcore` + `/dev/mem`)
fmem	Older kmod; exposes `/dev/fmem` for `dd`
fastdump	Volatility’s helper for capture + symbol resolution

Caveats that bound what’s possible.

kernel.lockdown=confidentiality blocks /dev/mem and /proc/kcore; whole-system tools that rely on them fail.
yama.ptrace_scope >= 2 blocks gcore without CAP_SYS_PTRACE.
Acquisition itself perturbs RAM; record the tool’s own footprint (kmod insertion, allocations) for chain-of-custody.

Analysis (offline, from the captured image).

$ vol -f /tmp/mem.lime windows.pslist            # Volatility 3
$ vol -f /tmp/mem.lime linux.bash                # recover bash history
$ strings -a /tmp/mem.lime | grep -Ei 'BEGIN PRIVATE KEY|password='
$ bulk_extractor -o /tmp/be /tmp/mem.lime        # carve emails, URLs, ccns

Common Tasks#

Investigate#

Read memory and swap totals.

$ free -h
$ cat /proc/meminfo | head -20
$ vmstat -s | head -10

               total        used        free      shared  buff/cache   available
Mem:           15Gi       8.2Gi       1.4Gi       412Mi       6.0Gi       6.5Gi
Swap:         4.0Gi       1.2Gi       2.8Gi

Rank processes by memory consumption.

$ ps -eo pid,user,vsz,rss,pmem,cmd --sort=-rss | head
$ top -o %MEM
$ smem -k -t -P chrome | head           # PSS-based; better cross-process view

Diagnose a thrashing box.

$ vmstat 1                              # si / so > 0 means paging to / from disk
$ sar -B 1                              # pgpgin / pgpgout / pgfault / pgmajfault
$ pidstat -r 1                          # per-process minor / major faults

Read the OOM killer’s log.

$ dmesg -T | grep -i 'killed process'
$ journalctl -kb | grep -i 'oom-killer'
$ cat /proc/<pid>/oom_score             # higher = more likely victim
$ cat /proc/<pid>/oom_score_adj         # operator override

Map one process’s address space.

$ cat /proc/<pid>/maps
$ cat /proc/<pid>/smaps_rollup          # totals: RSS, PSS, Swap
$ pmap -X <pid>                         # human-readable map

Modify#

Add or resize swap online.

$ sudo fallocate -l 4G /swapfile
$ sudo chmod 600 /swapfile
$ sudo mkswap /swapfile
$ sudo swapon /swapfile

# disable a swap device
$ sudo swapoff /swapfile

Persist in /etc/fstab:

/swapfile none swap sw 0 0

Cap a process’s memory and let the kernel kill it on overrun.

$ systemd-run --scope -p MemoryMax=512M -p MemorySwapMax=0 ./mem-hog
$ systemd-run -t -p MemoryMax=2G ./worker

Tune swappiness.

$ sysctl vm.swappiness                  # 0-200 (default 60); higher = swap sooner
$ sudo sysctl -w vm.swappiness=10
$ echo 'vm.swappiness=10' | sudo tee /etc/sysctl.d/99-swappiness.conf

Reference#

Tool inventory, /proc + /sys paths, and pointers to deeper reading. Skim once; bookmark for ops.

Diagnostics#

Tool	Reports
`free -h`	System totals; `-w` separates buff vs cache
`vmstat 1`	Live: `si`, `so`, `bi`, `bo`, `us`, `sy`, `id`, `wa`
`sar -B 1`	Page-fault and paging rates
`pidstat -r 1`	Per-process minor / major faults
`smem`	PSS (proportional set size) totals
`ps -eo pid,vsz,rss,pmem,cmd`	Per-process VSZ / RSS / %MEM
`pmap -X <pid>`	Per-mapping breakdown (similar to smaps)
`top` / `htop`	Live; sort by `%MEM`
`slabtop`	Kernel slab allocator usage
`cat /proc/pressure/memory`	PSI: memory stall, `some` / `full` over 10s/60s/300s
`bpftrace` / `bcc`	eBPF; page-fault tracepoints, mm allocator events
`perf record -e mem-loads`	Sample memory-access events
`valgrind --tool=memcheck`	User-space leak detector (slow, exhaustive)
`heaptrack` / `massif`	Heap allocation profilers
`checksec`	Per-binary mitigation summary
`volatility3`	Offline analysis of acquired RAM images

Files & paths#

Path	Purpose
`/proc/meminfo`	System-wide memory totals
`/proc/vmstat`	System-wide VM event counters
`/proc/buddyinfo`	Free pages per order, per zone
`/proc/slabinfo`	Per-cache kernel slab usage
`/proc/swaps`	Active swap devices
`/proc/pressure/memory`	PSI memory pressure (kernel >= 4.20)
`/proc/sys/vm/*`	VM tunables (swappiness, overcommit, …)
`/proc/sys/kernel/randomize_va_space`	ASLR mode
`/proc/sys/kernel/yama/ptrace_scope`	ptrace gating
`/proc/<pid>/maps`	Per-process VMAs (one line per region)
`/proc/<pid>/smaps`	Same + RSS / PSS / Swap / shared / private clean / dirty
`/proc/<pid>/smaps_rollup`	Rollup totals from `smaps`
`/proc/<pid>/pagemap`	Per-page physical frame and flags (binary)
`/proc/<pid>/mem`	Raw process memory (ptrace required)
`/proc/<pid>/oom_score` / `oom_score_adj`	OOM scoring + override
`/proc/<pid>/status`	`VmSize`, `VmRSS`, `VmSwap`, `Threads`, `Cap*`
`/sys/kernel/mm/transparent_hugepage/`	THP knobs
`/sys/kernel/mm/hugepages/`	HugeTLB pools per page size
`/sys/devices/system/node/`	NUMA node stats
`/sys/fs/cgroup/.../memory.*`	cgroup memory controllers

References#

man 5 proc (/proc/[pid]/maps, smaps, status field reference).
man 8 vmstat, man 1 sar, man 8 swapon, man 7 cgroups.
Processes for per-process memory accounting and the task_struct view.
Services for MemoryMax / MemoryHigh / MemorySwapMax per systemd unit.
Hardening for broader hardening posture (sysctl, AppArmor / SELinux, lockdown).
Linux MM admin guide covers THP, hugetlbpage, KSM, NUMA, page-migration.
Volatility 3 docs, the de-facto offline RAM analysis framework.