Kernel

Kernel#

The kernel is the resident process at the bottom of every Linux box, the only thing on the host the operator cannot work around. Every syscall, every file open, every packet, every privilege check, every namespace boundary is the kernel’s call. Read what the kernel is doing, what version is loaded, what modules are live, and what the sysctls say, and the operator has read the constraints the rest of the host operates under.

The kernel is what the boot sequence ends at (see Boot). The userland binaries the operator runs (see The Terminal and Processes) are clients of the syscall surface. The filesystem (see Filesystem), the network stack (see Networking), the security model (see Hardening, Permissions), and the memory map (see Memory) are all kernel subsystems. This page is about the kernel as a unit: how to identify it, inspect it, configure it, and read what it is doing right now.

Concept#

The kernel is the privileged software that owns the hardware. On x86 it runs in CPU ring 0 with full access to physical memory, I/O ports, and control registers. Every other process on the machine (the shell, the init system, the operator’s tools, daemons, browsers) runs in ring 3, unprivileged, with no direct path to the hardware. The kernel mediates every request that crosses that boundary.

The mediation happens at one place. The system call. A userspace program asks the kernel to do something (open a file, send a packet, allocate memory, fork a process), the CPU transitions into kernel mode long enough to do it, then returns. The kernel is preemptive. A running process can be interrupted mid-syscall to run something else; the userspace process cannot return the favor.

Three properties follow that are worth committing to memory.

  • Single shared address space. All kernel code shares one virtual memory layout, separate from any userspace process. A kernel bug in one driver can corrupt another driver; a userspace bug cannot reach kernel memory without a vulnerability.

  • Cooperative within, preemptive without. Kernel threads run until they yield or block on I/O; userspace processes run only when the scheduler grants them a slice.

  • Stable userspace ABI. Linus enforces “we don’t break userspace”. A binary built against a 2.6 kernel still runs on a 2026 kernel without recompilation.

The kernel is not an application. It has no main() it can exit from. The bootloader loads it (see Boot), it takes ownership of the hardware, starts /sbin/init as PID 1, and then services syscalls until the machine powers off.

Components#

The kernel ships as a single binary (the vmlinuz image) plus a collection of loadable modules. Inside that binary, work is split across the subsystems below. Each has its own maintainers, mailing list, and merge tree, but they share the kernel address space and call one another freely.

        mindmap
  root((Linux<br/>kernel))
    Scheduler
      CFS (legacy)
      EEVDF (6.6+)
      SCHED_FIFO / RR
      SCHED_DEADLINE
    Memory
      Buddy allocator
      Slab / slub
      Swap + OOM killer
      THP / KSM
    VFS
      ext4 / XFS / Btrfs
      FUSE / NFS
      proc / sys / tmpfs
    Network
      Sockets + IP
      TCP / UDP / QUIC
      netfilter / nftables
      eBPF / XDP
    Security
      Capabilities
      Namespaces
      Cgroups
      LSM (SELinux, AppArmor)
      seccomp / IMA / Yama
    Drivers
      Block
      Network
      Input
      Char / V4L2 / ALSA
    IPC
      Pipes / FIFOs / signals
      SysV + POSIX IPC
      Futexes
      UNIX sockets
    Tracing
      ftrace / tracepoints
      perf / kprobes / uprobes
      eBPF (bcc, bpftrace)

Process Scheduler#

Picks which runnable task gets which CPU core at each moment. CFS (Completely Fair Scheduler) was the historical default; EEVDF (Earliest Eligible Virtual Deadline First) replaced it in 6.6. Real-time (SCHED_FIFO, SCHED_RR) and deadline (SCHED_DEADLINE) classes sit alongside for tasks that need bounded latency.

Memory Manager#

Owns every byte of physical RAM and the per-process virtual address spaces that map onto it. Every userspace malloc eventually lands here, every shared library mapping, every file the kernel caches, every page the operator sees in /proc/PID/maps.

The subsystem decomposes into seven cooperating pieces. The operator sees their effects in /proc/meminfo, vmstat, smem, slabtop, and dmesg when one of them complains.

Component

Role

Buddy allocator

The page-level back end. Hands out physical pages in power- of-two orders (4 KiB, 8 KiB, 16 KiB, … up to MAX_ORDER). Splits larger blocks on demand, coalesces on free, fights fragmentation. State in /proc/buddyinfo.

Slab / SLUB allocator

The object-level front end for the kernel itself. Carves buddy pages into typed caches (task_struct, inode, dentry, kmalloc-128, …) so frequent fixed-size allocations don’t churn the page allocator. State in /proc/slabinfo; live view with slabtop.

Virtual memory (per process)

The mm_struct describing a process’s address space: code, data, heap, stack, mmaps, shared libraries. Lives at /proc/PID/maps (text) and /proc/PID/smaps (with RSS, PSS, swap per region). Built by execve and torn down at exit.

Page cache

The kernel’s read-through cache for file-backed pages. Every read, write, and mmap of a regular file passes through it; eviction is LRU. Most “free” memory on a healthy box is page cache, not unused.

Swap

The pressure-relief valve. Anonymous pages (heap, stack, MAP_ANONYMOUS) that haven’t been touched lately are written out to a backing device (/swapfile or a swap partition) and faulted back in on demand. zram and zswap compress in RAM as a faster alternative.

OOM killer

The last-resort enforcer when physical RAM plus swap is exhausted. Picks a process by oom_score (roughly: RSS × oom_score_adj) and kills it. Decisions land in dmesg under Out of memory:.

Transparent Huge Pages (THP) + KSM

Two optional optimisations. THP transparently promotes 2 MiB / 1 GiB pages where the workload is friendly (large contiguous anonymous regions). KSM scans anonymous pages and merges identical ones across processes; valuable on hosts running many similar VMs or containers.

 
        flowchart LR
    APP["User process<br/>malloc / mmap / brk"] -->|page fault| MM["mm subsystem"]
    MM --> VMA["per-process VMAs<br/>(/proc/PID/maps)"]
    MM --> PC["page cache<br/>(file-backed)"]
    MM --> SLAB["slab / SLUB<br/>(kernel objects)"]
    SLAB --> BUDDY["buddy allocator<br/>(physical pages)"]
    PC --> BUDDY
    VMA --> BUDDY
    BUDDY --> RAM["Physical RAM<br/>(/proc/meminfo)"]
    BUDDY -. pressure .-> RECL["page reclaim<br/>(kswapd / direct)"]
    RECL --> SWAP[("swap<br/>+ zswap / zram")]
    RECL --> THP["THP / KSM<br/>(compact + dedup)"]
    SWAP -. exhausted .-> OOM["OOM killer<br/>(dmesg: Out of memory:)"]

For the operator-facing deep dive (cgroups memory accounting, NUMA, kmem leaks, kernel-pointer-deref triage), see Memory.

VFS and Filesystems#

The Virtual File System layer is the kernel’s “one API for every storage backend” abstraction. Userspace calls open, read, write, stat; the VFS dispatches to whichever filesystem driver owns the path, whether that’s ext4 on local SSD, NFS on the network, procfs synthesising data on the fly, or FUSE bouncing into a userland daemon. Every path the operator types eventually touches this layer.

The subsystem decomposes into five cooperating pieces.

Component

Role

VFS core

The dispatch layer. Defines struct file, struct inode, struct dentry, struct super_block and the file_operations / inode_operations / dentry_operations vtables every filesystem driver implements.

Dentry cache

Path-component cache. /usr/bin/ls resolves through 3 cached dentries (usr, bin, ls) instead of 3 disk reads. Live counts in /proc/sys/fs/dentry-state.

Inode cache

Object-attribute cache for files and directories (permissions, owner, size, timestamps). Backed by the slab allocator. /proc/sys/fs/inode-state.

Filesystem drivers

The backends. Local disk (ext4, XFS, Btrfs, F2FS, FAT, NTFS3); network (NFS, CIFS, 9P); special / pseudo (procfs, sysfs, cgroup, tmpfs, devtmpfs, debugfs, securityfs); userland (FUSE). Loaded modules listed by cat /proc/filesystems.

Block / I/O layer

Below local filesystems. Multi-queue block scheduler (mq-deadline, bfq, none), the page cache as read-through, the writeback flushers (kworker), and device-mapper / md-raid for stacked devices.

 
        flowchart TB
    APP["User process<br/>open / read / write / stat / mmap"] --> SYS["Syscall layer"]
    SYS --> VFS["VFS core<br/>(file / inode / dentry vtables)"]
    VFS --> DC["Dentry cache<br/>(path resolution)"]
    VFS --> IC["Inode cache<br/>(attributes)"]
    VFS --> FS{"Filesystem driver<br/>(by mount)"}
    FS -->|local disk| EXT["ext4 / XFS / Btrfs / F2FS"]
    FS -->|special| PROC["procfs / sysfs / cgroup<br/>tmpfs / devtmpfs / debugfs"]
    FS -->|network| NFS["NFS / CIFS / 9P"]
    FS -->|userland| FUSE["FUSE → userland daemon"]
    EXT --> PC["Page cache"]
    PC --> BLK["Block / I/O layer<br/>(mq scheduler, dm, md)"]
    BLK --> DRV["Block device driver"]
    DRV --> HW["Disk / SSD / NVMe"]

See Filesystem for the operator- facing deep dive on filesystem choice, repair, and forensics.

Network Stack#

One in-kernel pipeline from the NIC driver up to the BSD socket API. The operator’s ssh, curl, dig, nc, plus every listening daemon, every container’s overlay traffic, every WireGuard tunnel, all traverse the same code paths. The stack is per-CPU locked, RCU-heavy, and increasingly programmable through eBPF / XDP for high-throughput cases.

Eight layers compose it, top to bottom.

Component

Role

Socket API

socket, bind, connect, send, recv, epoll, io_uring. The BSD-derived interface every network program calls. Operator sees per-socket state in /proc/net/tcp, udp, unix; modern tool is ss.

Protocol handlers (L4)

TCP, UDP, SCTP, QUIC (kernel and userland variants), DCCP. Implement reliability, ordering, congestion control (cubic / bbr / reno; sysctl net.ipv4.tcp_congestion_control).

IP layer (L3)

IPv4 and IPv6. Routing (/proc/net/route, ip route), fragmentation, neighbour cache (ARP / NDP), ICMP. Multi-table policy routing via ip rule.

Netfilter / nftables

The kernel’s packet-filtering, NAT, and connection- tracking framework. Hooks at PREROUTING, INPUT, FORWARD, OUTPUT, POSTROUTING. nftables is the modern frontend; iptables rules now compile down to the same engine.

Traffic control (tc)

Egress and ingress queueing, shaping, policing, and classification. fq_codel, cake, HTB, BPF classifiers. Where bandwidth limits and DDoS scrubbing land.

eBPF / XDP

eXpress Data Path: BPF programs attached at the driver level, before skb allocation. Cilium, Katran, load-balancers, DDoS scrubbers run here. Sibling tc BPF programs attach further up the stack.

Link layer (L2)

net_device framework: bridges, VLANs, bonds, teams, VRFs, tun/tap, veth pairs (container plumbing). ip link for status; ethtool for NIC-level inspection.

NIC driver + ring buffers

Vendor driver (e1000e, mlx5, i40e, virtio_net) managing RX / TX descriptor rings, NAPI poll, hardware offloads (TSO, GRO, RSS, checksum). ethtool -i, ethtool -k, ethtool -S.

 
        flowchart TB
    APP["User process<br/>socket / send / recv / epoll / io_uring"] --> SOCK["Socket layer<br/>(/proc/net/tcp, ss)"]
    SOCK --> L4["TCP / UDP / QUIC / SCTP<br/>congestion control"]
    L4 --> NF1["netfilter OUTPUT / POSTROUTING<br/>conntrack, NAT"]
    NF1 --> L3["IP layer (v4 / v6)<br/>routing, neighbour"]
    L3 --> TCO["tc egress<br/>fq_codel / HTB / BPF"]
    TCO --> L2["L2: bridge / VLAN / bond / veth"]
    L2 --> DRV["NIC driver<br/>TX ring, offloads"]
    DRV ==> NIC["Hardware NIC"]
    NIC ==> DRV2["NIC driver<br/>RX ring, NAPI"]
    DRV2 --> XDP["XDP hook<br/>(pre-skb, fastest drop / fwd)"]
    XDP --> L2R["L2 ingress"]
    L2R --> TCI["tc ingress<br/>(BPF classifier)"]
    TCI --> NF2["netfilter PREROUTING / INPUT<br/>conntrack, NAT"]
    NF2 --> L3R["IP layer (v4 / v6)"]
    L3R --> L4R["TCP / UDP / QUIC / SCTP"]
    L4R --> SOCKR["Socket layer"]
    SOCKR --> APPR["User process"]

See Networking for the operator-facing deep dive on protocols, tools, and tradecraft.

Security and Isolation#

Capabilities split root into ~40 fine-grained permissions; namespaces give each process tree its own view of resources; cgroups cap and account resource use; the LSM framework (SELinux, AppArmor, Tomoyo, Smack) plugs in mandatory access control; seccomp filters syscalls; IMA/EVM measure file integrity; Yama gates ptrace. These compose to build containers and sandboxes.

Device Drivers#

Code that translates between the kernel’s uniform abstractions and the specific quirks of one piece of hardware. The kernel ships ~50,000 drivers; one kernel image runs on every laptop, server, phone, router, and embedded box because the right drivers load on demand for whatever the bus enumeration finds. Drivers are also the largest attack surface in the kernel; most kernel CVEs in any given year are in drivers.

Driver code fits into one of seven framework slots so userspace sees uniform interfaces over wildly different hardware.

Class

Userspace face

Examples and notes

Block

/dev/sd*, /dev/nvme*, /sys/block/

SATA, NVMe, SCSI, virtio-blk, MMC. Plugs into the block layer (mq scheduler, request queue) below the page cache.

Network

ip link, ethtool, /sys/class/net/

e1000e, mlx5, i40e, virtio_net, WireGuard. Implement net_device_ops; offer TSO/GRO/RSS offloads.

Char

/dev/<name> (no fixed size, byte-stream)

Serial (/dev/ttyS*), TPM, hwrng, /dev/null, /dev/urandom, GPIO chardev. The catch-all class.

Input

/dev/input/event*, evdev

Keyboard, mouse, touchpad, joystick, touchscreen. evdev protocol; libinput decodes for desktops.

USB / PCI / I2C / SPI buses

/sys/bus/<bus>/devices/

Bus core enumerates devices and matches them to drivers by vendor/device ID. lspci -nnk, lsusb -t.

V4L2 / DRM / ALSA

/dev/video*, /dev/dri/*, /dev/snd/*

Multimedia frameworks. V4L2 for capture, DRM/KMS for GPU and display, ALSA for audio.

MTD / virtio / IIO / others

varies

Raw flash, paravirtualised devices for guests, industrial I/O sensors, and the long tail.

 
        flowchart TB
    HW["Hardware<br/>NIC, NVMe, USB, GPU, sensor, ..."] --> BUS["Bus core<br/>PCI / USB / I2C / SPI / platform"]
    BUS -->|enumerate +<br/>match vendor:device ID| MATCH{"Driver match?"}
    MATCH -->|yes| DRV["Driver<br/>(built-in or loadable module)"]
    MATCH -->|no| WAIT["Wait for module<br/>(udev / modprobe)"]
    WAIT --> DRV
    DRV --> FW{"Framework slot"}
    FW -->|disk / SSD / NVMe| BLK["Block layer<br/>/dev/sd*, /dev/nvme*"]
    FW -->|NIC| NET["Network layer<br/>/sys/class/net/"]
    FW -->|keyboard, mouse| INPUT["evdev<br/>/dev/input/event*"]
    FW -->|TPM, serial, hwrng| CHAR["Char dev<br/>/dev/<name>"]
    FW -->|GPU / display| DRM["DRM/KMS<br/>/dev/dri/*"]
    FW -->|camera| V4L["V4L2<br/>/dev/video*"]
    FW -->|audio| ALSA["ALSA<br/>/dev/snd/*"]
    BLK --> APP["User process<br/>(uniform file API)"]
    NET --> APP
    INPUT --> APP
    CHAR --> APP
    DRM --> APP
    V4L --> APP
    ALSA --> APP

Most drivers ship as loadable modules under /lib/modules/$(uname -r)/kernel/drivers/; the dispatch table is auto-built from MODULE_DEVICE_TABLE declarations so udev can match a freshly-plugged USB ID to the right module.

IPC and Synchronisation#

The kernel-side primitives that let separate processes coordinate without raw shared memory. Every | in a shell pipeline, every Ctrl+C, every D-Bus message, every container’s docker.sock, every mutex inside a Python program, all bottom out in one of the mechanisms below. The operator who reads them in /proc/PID/fd/ and /proc/PID/status answers “what is this process actually talking to” without guesswork.

Eight primitives, grouped by communication pattern.

Primitive

Role

Pipes / FIFOs

Unidirectional byte stream between processes. Anonymous (pipe() between fork+exec’d children) or named (mkfifo on disk). ls /proc/PID/fd/ shows pipe:[inode] for each end.

Signals

Asynchronous notifications: SIGTERM, SIGKILL, SIGCHLD, SIGUSR1, etc. 31 standard + 32 real-time signals; kill -l lists them; signalfd lets a process read them as file events.

System V IPC

The classic trio. Message queues (msgget), semaphores (semget), shared memory (shmget). Persistent across process death; visible with ipcs, removable with ipcrm. Legacy; new code uses POSIX IPC.

POSIX IPC

Modernised replacements. mq_* for message queues, sem_* for semaphores, shm_* for shared memory. Backed by tmpfs (/dev/mqueue, /dev/shm).

UNIX-domain sockets

Local-only sockets, file-backed (/run/foo.sock) or abstract. The default for desktop and container IPC: D-Bus, Wayland, Docker, systemd-journald, Postgres, Redis. Pass file descriptors and credentials in-band.

eventfd / signalfd / timerfd

File-descriptor versions of events, signals, and timers so a single epoll / io_uring loop can wait on all of them uniformly.

Futexes

The kernel-side fast wait/wake primitive that userspace mutex, condvar, and semaphore libraries (pthread, glibc, parking_lot, Go runtime) build on top of. Mostly contention-free; only entered when a thread must block.

Memory-mapped + atomics

mmap(MAP_SHARED) plus atomic CPU instructions. Zero syscalls in the fast path. The substrate behind futexes; used directly by databases and JIT runtimes.

 
        flowchart LR
    P1["Process A"] --> PIPE["Pipe / FIFO<br/>byte stream"]
    PIPE --> P2["Process B"]
    P1 -.signal.-> P2
    P1 <--> UDS["UNIX-domain<br/>socket<br/>/run/foo.sock"]
    UDS <--> P2
    P1 --> SVIPC["SysV IPC<br/>(ipcs)<br/>msgq / sem / shm"]
    SVIPC --> P2
    P1 --> POSIX["POSIX IPC<br/>/dev/mqueue<br/>/dev/shm"]
    POSIX --> P2
    P1 --> FUTEX[("Futex<br/>(uaddr in shared page)")]
    FUTEX --> P2
    P1 <--> SHMEM[("mmap MAP_SHARED<br/>+ atomics<br/>(no syscalls fast path)")]
    SHMEM <--> P2
    EV["eventfd /<br/>signalfd /<br/>timerfd"] -.-> EP[("epoll / io_uring<br/>(uniform wait)")]
    P1 -.-> EP
    P2 -.-> EP

Tracing and Profiling#

The instrumentation surface the kernel exposes for “what is this box doing right now, in detail”. Five primitives plus the eBPF VM underneath compose every modern observability tool the operator reaches for: strace, perf, bpftrace, bcc, Falco, Tetragon, Cilium, Tracee. The mechanisms are also the operator’s offensive surface; the same hooks that audit a process can instrument cryptographic libraries or steal keystrokes.

Six layers, lowest to highest.

Component

Role

Tracepoints

Static, named instrumentation points compiled into the kernel. Stable contract. ls /sys/kernel/tracing/events/ lists every category (syscalls, sched, net, block, vfs, …).

kprobes / kretprobes

Dynamic probes that attach to any kernel function on demand (kprobe:vfs_read). No source-code support needed; works on stripped builds with symbols.

uprobes / uretprobes

Same idea, attached to userspace functions (uprobe:/usr/lib/libssl.so:SSL_read). Lets the operator instrument any binary on the box without modifying it.

perf events

Hardware PMU counters (cache misses, branch mispredictions, IPC) and software events (page faults, context switches). perf top, perf stat, perf record.

ftrace

Built-in function tracer. trace-cmd and the /sys/kernel/tracing/ interface. Function-level latency, call graphs, scheduler events. Lowest- overhead path to “what fired in what order”.

eBPF VM

The unifying runtime. Bytecode programs attached to tracepoints, kprobes, uprobes, perf events, LSM hooks, XDP, and tc. Verified for safety before load; data out via per-CPU maps and ring buffers. Powers bpftrace, bcc, Falco, Tetragon, Cilium, Tracee.

 
        flowchart LR
    subgraph SRC["Event sources"]
        TP["Tracepoints<br/>(static, /sys/kernel/tracing)"]
        KP["kprobes<br/>(dynamic, any kernel fn)"]
        UP["uprobes<br/>(dynamic, any user fn)"]
        PE["perf events<br/>(PMU + software)"]
        LSM["LSM hooks<br/>(BPF LSM)"]
        XDP["XDP / tc<br/>(packet path)"]
    end
    subgraph VM["eBPF VM<br/>verifier + JIT"]
        PROG["BPF program"]
        MAP["maps<br/>(hash / array /<br/>perf / ringbuf)"]
    end
    TP --> PROG
    KP --> PROG
    UP --> PROG
    PE --> PROG
    LSM --> PROG
    XDP --> PROG
    PROG --> MAP
    MAP --> US["User-space tools"]
    subgraph TOOLS["User-space tools"]
        BT["bpftrace"]
        BCC["bcc"]
        PERF["perf"]
        FT["ftrace / trace-cmd"]
        FALCO["Falco / Tetragon /<br/>Cilium / Tracee"]
    end

Interfaces#

The kernel sits between two worlds: userspace above, hardware below. It also has a third, less obvious boundary: between its own core image and the loadable modules that extend it. Each of these three boundaries is a separate contract, with a separate stability policy, a separate audience, and a separate failure mode when the operator gets it wrong.

Interface

Faces

Stability

Failure mode when broken

Syscall ABI

User space

Stable by policy (“do not break userspace”)

Userspace binary stops working; treated as a kernel bug.

Hardware interface

Hardware below

Per-arch + vendor; defined by buses, IRQs, DMA, ACPI / DT

Device fails to enumerate; dmesg shows probe failed.

Module ABI

Kernel modules above the core

Unstable by policy; can change every release

Module fails to load after kernel upgrade; modprobe reports symbol mismatch.

The diagram below shows the three together. Operator angle: syscalls are where instrumentation and exploitation hook; the hardware interface is where driver bugs and firmware quirks live; the module ABI is why a kernel upgrade can silently break NVIDIA, OpenZFS, or a vendor backport.

User-Space ABI (Syscalls)#

The kernel’s contract with everything above it, and the single most enforced stability promise in Linux. About 400 numbered system calls (read, write, mmap, execve, fork, ioctl, bpf, …) plus a per-architecture calling convention define the surface. A 1996 binary still runs on a 2026 kernel. Linus has been quoting “we do not break userspace” for twenty-plus years and means it.

Five pieces compose this interface.

Component

Role

Syscall numbers

One per call, per arch (__NR_read is 0 on x86_64, 63 on ARM64). Numbers are append-only; once assigned, they never move. ausyscall x86_64 --dump.

Calling convention

Which registers carry what. x86_64 SysV uses rax = number, rdi rsi rdx r10 r8 r9 = args 1-6, rcx + r11 clobbered by SYSCALL, return in rax. ARM64 uses x8 = number, x0-x5 = args, return in x0.

Trap instruction

syscall (x86_64), svc #0 (ARM64), ecall (RISC-V), scv 0 (POWER). Switches the CPU to kernel mode, vectors to the architecture’s syscall entry.

vDSO

“virtual dynamic shared object”, a tiny ELF mapped into every process. Serves gettimeofday, clock_gettime, getcpu, time, rt_sigreturn without a trap. Visible in cat /proc/PID/maps | grep vdso.

Audit / seccomp gates

Layered between trap and handler. auditd records; seccomp BPF filter can allow, errno, kill, or trace. The reason a sandboxed process gets EPERM on ptrace even as root.

 
        flowchart TB
    APP["User process<br/>(ring 3)"] --> CHOICE{"Which path?"}
    CHOICE -->|vDSO call<br/>gettimeofday, clock_gettime,<br/>getcpu, time, rt_sigreturn| VDSO["vDSO<br/>(user-mode read of<br/>kernel-maintained page)"]
    VDSO --> APP
    CHOICE -->|everything else| LIBC["libc / direct syscall<br/>load arg registers"]
    LIBC --> TRAP["trap instruction<br/>x86_64 SYSCALL · arm64 SVC<br/>riscv ECALL · power SCV"]
    TRAP --> ENTRY["arch syscall entry<br/>(entry_SYSCALL_64 / el0_svc)"]
    ENTRY --> NUMTBL["sys_call_table[rax]"]
    NUMTBL --> AUDIT["audit / seccomp / LSM<br/>(deny here = EPERM / kill)"]
    AUDIT --> HND["sys_<name> handler<br/>vfs_read, do_mmap, ..."]
    HND --> RET["SYSRET / ERET<br/>(return value in rax / x0)"]
    RET --> APP

Hardware Interface (Drivers)#

The kernel’s contract with the hardware below it. Buses (PCI, USB, I2C, SPI, MMIO) define how to talk to devices; the device model under /sys/devices tracks what is plugged in; IRQ lines deliver interrupts; DMA engines move data without CPU involvement; the device tree (ARM, RISC-V) or ACPI (x86) tells the kernel what hardware exists at boot. Drivers fit framework slots (the block layer, the network layer, the input subsystem, V4L2 for video, ALSA for audio) so userspace sees uniform abstractions over varied hardware.

In-Kernel Module ABI#

The kernel’s contract with itself, between the core image and loadable modules. It is deliberately unstable. Every kernel release can change in-kernel function signatures, structure layouts, and locking rules. Distros sign modules against a specific kernel build; out-of-tree modules (NVIDIA, OpenZFS, vendor drivers) recompile against /lib/modules/$(uname -r)/build headers. This is why module-load failures after a kernel upgrade are routine and why userspace breakage after a kernel upgrade is treated as a bug.

        flowchart TB
    subgraph US["User space &nbsp;&nbsp;(ring 3)"]
        direction TB
        APP["Applications<br/>shell, daemons, browsers, agents"]
        LIB["Libraries<br/>libc (glibc / musl / Bionic), libssl, ..."]
        VDSO["vDSO fast path<br/>gettimeofday, clock_gettime, getcpu<br/>no kernel trap"]
        APP --> LIB
        APP -.-> VDSO
    end

    ABI(["&nbsp;1.&nbsp; Syscall ABI &nbsp;&middot;&nbsp; STABLE<br/>open, read, mmap, execve, bpf, ioctl, ...<br/>'do not break userspace'"])

    subgraph KS["Kernel space &nbsp;&nbsp;(ring 0)"]
        direction TB
        ENTRY["syscall entry<br/>x86_64 SYSCALL &nbsp;&middot;&nbsp; arm64 SVC"]
        subgraph CORE["Core subsystems"]
            direction LR
            SCH["Scheduler<br/>EEVDF / RT"]
            MM["Memory<br/>pages, slab, OOM"]
            VFS["VFS<br/>ext4, XFS, Btrfs, proc, sys"]
            NET["Network<br/>sockets, IP, TCP/UDP, netfilter"]
            SEC["Security<br/>LSM, namespaces, cgroups, seccomp"]
        end
        MABI(["&nbsp;2.&nbsp; Module ABI &nbsp;&middot;&nbsp; UNSTABLE<br/>in-kernel function signatures<br/>recompile per release"])
        DRV["Device drivers<br/>built-in + loadable modules"]
        ENTRY --> CORE
        CORE --> MABI
        MABI --> DRV
    end

    HWABI(["&nbsp;3.&nbsp; Hardware interface<br/>PCI &middot; USB &middot; I2C &middot; SPI &middot; MMIO &middot; IRQ &middot; DMA<br/>per-arch, vendor-defined"])

    HW["Hardware<br/>CPU &middot; RAM &middot; disk &middot; NIC &middot; GPU"]

    LIB ==>|trap| ABI
    ABI ==> ENTRY
    DRV ==> HWABI
    HWABI ==> HW

    classDef stable fill:#0d3b22,color:#d6f7df,stroke:#3fb950
    classDef unstable fill:#3b1f1f,color:#fad7d7,stroke:#f85149
    class ABI stable
    class MABI unstable

Userland#

The kernel handles syscalls, scheduling, memory, and drivers. Everything the operator actually interacts with (the C library, init system, shell, coreutils, package manager, daemons, libraries, applications) runs in user space and is collectively the userland. Linus’ tree at kernel.org ships only the kernel; a usable OS is the kernel plus a userland someone else assembled.

What that userland is varies by OS. The kernel is one project, the userland is a separate project (or many), and any given OS picks-and-mixes.

OS

Kernel

Userland origin

Most Linux distros

Linux

GNU (glibc, bash, coreutils, util-linux) plus freedesktop.org (systemd, dbus). Hence “GNU/Linux”.

Alpine

Linux

musl libc, BusyBox coreutils, OpenRC init.

Android

Linux

AOSP: Bionic libc, Toybox utilities, Android init, ART runtime.

ChromeOS

Linux

Gentoo-derived GNU userland with Google’s session manager and UI.

FreeBSD, OpenBSD

BSD kernel

Their own BSD-licensed userland (libc, sh, ls, …) maintained in-tree.

macOS

XNU (Mach + BSD)

Apple’s Darwin userland (BSD heritage) plus Cocoa frameworks.

The practical consequence for the operator. “Linux” by itself is ambiguous. A Debian box and an Android phone share a kernel, but their userlands don’t share a single binary. Cross-compiled tooling, static binaries, and prebuilt wheels all break across libc boundaries (glibc vs musl vs Bionic), and exploit chains that lean on coreutils or shell behavior are tied to the userland, not the kernel.

Identification#

The first move on an unfamiliar box. Pin down which kernel is running, which command line it booted with, and how it identifies itself.

        flowchart LR
    SH["Shell on<br/>unfamiliar host"] --> UN["uname -srm<br/>release + arch"]
    UN --> PV["/proc/version<br/>build banner<br/>(compiler, date, distro)"]
    PV --> PCMD["/proc/cmdline<br/>boot args<br/>(root=, init=, lockdown=, ima=)"]
    PCMD --> OS["/etc/os-release<br/>distro + version"]
    OS --> BOOT["ls /boot/vmlinuz-*<br/>installed kernels"]
    BOOT --> CVE["Map distro string<br/>to CVE applicability"]
    
$ uname -a
$ uname -srm
$ cat /proc/version
$ cat /proc/cmdline
$ cat /etc/os-release         # distro, for context
$ ls /boot/vmlinuz-*

uname is the first read; /proc/version is the build banner (compiler, build host, build time); /proc/cmdline is the kernel command line passed by the bootloader (root device, init, security mode toggles, debug flags).

Useful flags:

Flag

Meaning

uname -s

kernel name (Linux)

uname -r

kernel release (6.5.0-15-generic)

uname -v

kernel build banner (date and config tag)

uname -m

machine hardware (x86_64, aarch64)

uname -p

processor (often unknown; prefer -m)

uname -o

operating system (GNU/Linux)

uname -a

all of the above

Other identification surfaces:

  • hostnamectl (systemd) prints kernel + distro + machine ID together.

  • /sys/kernel/version and /sys/kernel/osrelease mirror what uname reports.

  • /proc/sys/kernel/ostype, osrelease, version are the same data via the sysctl tree.

Modules#

Most of the kernel’s drivers, filesystems, and protocol handlers ship as loadable modules rather than being linked into the core image. Modules let one kernel binary support every NIC, every filesystem, every USB device the operator might encounter, while loading code only when it’s needed. A few subsystems are always built in (the scheduler, memory manager, VFS core), but the peripheral surface is module-driven.

Modules live under /lib/modules/$(uname -r)/ and load through udev triggers, systemd-modules-load, or explicit modprobe.

        flowchart LR
    HW["Hardware event<br/>(USB plug, PCI add)"] --> UDEV["udev<br/>(systemd-udevd)"]
    BOOT["Boot<br/>systemd-modules-load"] --> MOD["modprobe<br/>(kmod)"]
    CLI["Operator<br/>modprobe MODULE"] --> MOD
    UDEV --> MOD
    MOD --> RES["Resolve deps<br/>modules.dep"]
    RES --> CONF["Apply modprobe.d<br/>options, alias, blacklist"]
    CONF --> SIG["Verify signature<br/>(Secure Boot / lockdown)"]
    SIG --> LOAD["init_module()<br/>syscall"]
    LOAD --> RUN["Module live<br/>/sys/module/<name>"]
    LOAD --> LOG["printk -> dmesg"]
    
$ lsmod                                   # loaded modules
$ lsmod | grep -E 'nf_|nft_'              # netfilter / nftables modules
$ modinfo MODULE                          # parameters, license, author, deps
$ sudo modprobe MODULE                    # load
$ sudo modprobe -r MODULE                 # unload (rmmod is the older form)
$ sudo dmesg -T | tail                    # confirm load events

Listing what’s loaded:

Module                  Size  Used by
nf_conntrack_netlink    49152  0
nfnetlink              20480  3 nf_conntrack_netlink,...
xfrm_user              45056  1
...

Persistent control:

  • /etc/modules-load.d/*.conf: list module names to load at boot.

  • /etc/modprobe.d/*.conf: pass parameters, set aliases, blacklist.

  • blacklist MODULE in a modprobe.d file prevents auto-load.

Operator-relevant uses:

  • Hunt unsigned modules on a host with Secure Boot or kernel lockdown. cat /proc/keys shows trusted signing keys; tainted modules show (O) or (P) flags in lsmod.

  • Find LKM rootkits. Compare lsmod against the persistent config; check for hidden modules via cat /proc/modules vs ls /sys/module/ divergence; check dmesg for load events during the suspect window.

  • Disable risky kernel features (e.g. usb-storage, firewire-core, bluetooth) on a hardened workstation.

Module signing and lockdown live under /sys/kernel/security/lockdown and /proc/sys/kernel/modules_disabled.

Syscalls#

Every interaction the operator has with the kernel goes through a system call: open, read, write, mmap, fork, execve, ptrace, bpf, ioctl, and a few hundred others. The syscall ABI is the contract between userspace and the kernel; profiling, debugging, and tradecraft all anchor here.

        sequenceDiagram
    autonumber
    participant APP as User process<br/>(ring 3)
    participant LIBC as libc<br/>(glibc / musl)
    participant K as Kernel entry<br/>(ring 0)
    participant CHK as Pre-handler checks
    participant SUB as Subsystem<br/>(VFS / net / mm)

    APP->>LIBC: read(fd, buf, n)
    Note right of LIBC: load registers<br/>rax = 0 (NR_read)<br/>rdi = fd<br/>rsi = buf<br/>rdx = n
    LIBC->>K: SYSCALL (trap, ring 3 → 0)
    Note over K: entry_SYSCALL_64<br/>save user regs, switch GS,<br/>load kernel stack
    K->>CHK: dispatch via sys_call_table[0]
    CHK->>CHK: seccomp filter
    CHK->>CHK: audit_filter_syscall
    CHK->>CHK: LSM hook (SELinux / AppArmor)
    CHK->>CHK: tracepoint sys_enter (kprobe / bpf attach)
    CHK->>SUB: sys_read → vfs_read → file_op->read_iter
    SUB-->>CHK: bytes returned, or -errno
    CHK->>CHK: tracepoint sys_exit
    CHK-->>K: rax = bytes or -errno
    K-->>LIBC: SYSRET (restore ring 3)
    LIBC-->>APP: ssize_t return value

Watch what a process is doing:

$ sudo strace -f -p PID                   # follow forks; live
$ sudo strace -e openat,connect,execve -p PID
$ sudo strace -c CMD                       # summary count + time per syscall
$ ltrace CMD                               # library calls (one level above syscalls)
$ sudo perf trace -p PID                   # lower-overhead alternative
$ sudo bpftrace -e 'tracepoint:syscalls:sys_enter_execve { printf("%s\n", str(args->filename)); }'

Inspect kernel-side surface:

$ ausyscall x86_64 --dump | head -40       # syscall numbers (audit-userspace)
$ ls /sys/kernel/debug/tracing/events/syscalls/ | head
$ cat /proc/PID/syscall                    # current syscall + args
$ cat /proc/PID/status | grep -i 'cap\|seccomp\|nohash'

The vDSO (virtual dynamic shared object, mapped per process) implements a few cheap calls (gettimeofday, clock_gettime, getcpu) without crossing into the kernel; cat /proc/PID/maps | grep vdso shows the mapping.

/proc and /sys#

The kernel’s two read-write windows into itself.

``/proc`` is the older interface, started as per-process state (/proc/PID/...) and grew to hold kernel-wide knobs and read-only views.

``/sys`` is the newer, typed interface for the device model (buses, devices, drivers, kernel objects).

        flowchart LR
    subgraph PROC["/proc (procfs)"]
        direction TB
        PPID["/proc/PID/<br/>per-process state"]
        PKERN["/proc/version<br/>cmdline / meminfo / loadavg"]
        PSYSCTL["/proc/sys/<br/>sysctl tree"]
    end
    subgraph SYS["/sys (sysfs)"]
        direction TB
        SBUS["/sys/class/<br/>net, block, input"]
        SDEV["/sys/devices/<br/>bus topology"]
        SKERN["/sys/kernel/<br/>security, tracing, debug"]
        SMOD["/sys/module/<br/>loaded modules"]
    end
    PS["ps, top"] --> PPID
    FREE["free, vmstat"] --> PKERN
    SYSCTL["sysctl"] --> PSYSCTL
    IP["ip, ethtool"] --> SBUS
    LSBLK["lsblk, smartctl"] --> SBUS
    LSMOD["lsmod, modinfo"] --> SMOD
    SECTL["aa-status, sestatus"] --> SKERN

Together they’re how every well-behaved Linux tool reads the host. ps reads /proc/PID, ip reads /sys/class/net, lsblk walks /sys/block, free reads /proc/meminfo.

Selected entries every operator should know cold:

Path

What it shows

/proc/version

kernel build banner

/proc/cmdline

boot command line

/proc/cpuinfo

CPU model, flags (vmx, svm, sgx, aes, …)

/proc/meminfo

memory totals, slab, swap

/proc/loadavg

1 / 5 / 15-minute load

/proc/uptime

uptime + idle time

/proc/mounts

mounted filesystems (kernel view)

/proc/modules

loaded modules (low-level form of lsmod)

/proc/kallsyms

kernel symbol table (often restricted)

/proc/keys

keyring contents

/proc/PID/maps

the process address space

/proc/PID/status

one-screen process summary

/proc/PID/limits

resource limits

/proc/PID/ns/

per-process namespaces (mnt, pid, net, user, ipc, uts, cgroup, time)

/proc/sys/...

the sysctl tree

/sys/class/net/

network interfaces

/sys/block/

block devices

/sys/firmware/efi/

EFI variables (when booted UEFI)

/sys/kernel/security/

LSM and lockdown state

/sys/kernel/tracing/

ftrace control plane

/sys/kernel/debug/

debugfs (root-only on hardened systems)

sysctl#

Knobs the operator turns to change kernel behavior at runtime. The sysctl tree is exposed under /proc/sys/ (read or write a file) and via the sysctl command (which writes the same files with nicer syntax).

        mindmap
  root((sysctl))
    kernel.
      kptr_restrict
      dmesg_restrict
      unprivileged_bpf_disabled
      yama.ptrace_scope
      core_pattern
      randomize_va_space
    net.
      ipv4.ip_forward
      ipv4.tcp_syncookies
      ipv4.conf.all.rp_filter
      ipv4.conf.all.accept_redirects
      ipv6.conf.all.accept_ra
      core.somaxconn
    vm.
      swappiness
      overcommit_memory
      max_map_count
      drop_caches
      min_free_kbytes
    fs.
      file-max
      inotify.max_user_watches
      protected_symlinks
      protected_hardlinks
      suid_dumpable
    user.
      max_user_namespaces
      max_pid_namespaces
    dev.
      tty.legacy_tiocsti
    
$ sysctl -a 2>/dev/null | wc -l            # how many knobs there are
$ sysctl net.ipv4.ip_forward
$ sudo sysctl -w net.ipv4.ip_forward=1     # live; lost on reboot
$ sysctl --system                          # reload from /etc/sysctl.d/

Persistence:

  • /etc/sysctl.conf: legacy, still works.

  • /etc/sysctl.d/*.conf: drop-in directory; preferred.

  • Distros ship defaults under /usr/lib/sysctl.d/.

Hardening defaults the operator typically flips on a fresh box (see Hardening):

kernel.kptr_restrict=2
kernel.dmesg_restrict=1
kernel.unprivileged_bpf_disabled=1
kernel.yama.ptrace_scope=1
net.ipv4.tcp_syncookies=1
net.ipv4.conf.all.rp_filter=1
net.ipv4.conf.all.accept_redirects=0
net.ipv6.conf.all.accept_ra=0
fs.protected_symlinks=1
fs.protected_hardlinks=1

Logging#

The kernel writes to a single ring buffer that userland reads with dmesg (and journalctl -k on systemd boxes, which copies the ring buffer into the journal).

        flowchart LR
    SRC["Sources<br/>module load/unload, HW events,<br/>OOM, panic, netfilter LOG,<br/>SELinux / AppArmor denials"] --> PR["printk()"]
    PR --> RING[(Ring buffer<br/>kmsg / /dev/kmsg)]
    RING --> DM["dmesg<br/>(direct read)"]
    RING --> KMSGD["systemd-journald<br/>(reads /dev/kmsg)"]
    KMSGD --> JOUR[(journal<br/>/var/log/journal/)]
    JOUR --> JCTL["journalctl -k"]
    KMSGD --> SYSLOG["rsyslog / syslog-ng<br/>(optional forward)"]
    SYSLOG --> REMOTE["Remote syslog<br/>/ SIEM"]
    
$ dmesg -T                                 # human-readable timestamps
$ dmesg -T --level=err,warn                # filter
$ dmesg -w                                 # follow live
$ journalctl -k --since '1 hour ago'       # systemd journal view
$ journalctl -k -p err --since today

What lands in the buffer:

  • Module load and unload events.

  • Hardware events (USB plug, NIC up, disk error).

  • OOM killer activity.

  • Panic / oops messages.

  • netfilter / iptables LOG target output.

  • Security events (SELinux denials, AppArmor denials, audit if routed via printk).

The ring buffer is finite (/proc/sys/kernel/printk controls verbosity; /sys/kernel/debug/dynamic_debug/control enables selective debug prints). Persistent kernel logs are journald’s job.

Versions and Channels#

The kernel project ships in three streams the operator should distinguish.

        flowchart LR
    L["Linus tree<br/>(mainline)"] -->|~9 wk tag| M["Mainline release<br/>(6.x)"]
    M -->|forked| S["Stable series<br/>(6.x.y, ~2 mo)"]
    M -.->|selected picks| LTS["Long-term (LTS)<br/>(2-6 yr)"]
    S --> RHEL["RHEL / Debian stable<br/>frozen + backports"]
    LTS --> UBHWE["Ubuntu LTS / HWE<br/>rolling stable"]
    LTS --> ARCH["Arch / Gentoo / Void<br/>mainline-tracking"]
    LTS --> FEDORA["Fedora<br/>current stable line"]

Upstream streams (what kernel.org tags).

Stream

Cadence

Support window

Notes

Mainline

new release every ~9 weeks

until the next mainline tag

The version Linus tags. No backported fixes; you take the whole new tree or wait.

Stable

point releases on the current mainline

~2 months, until the next mainline

Each mainline tag becomes a stable series with targeted security and bug fixes.

Long-term (LTS)

chosen mainline series, point releases

2-6 years (recent picks: 2 years)

The kernel.org LTS page is the authoritative list of which versions get long-term care.

Downstream patterns (how distros consume upstream).

Pattern

Distros

Behaviour

Frozen with backports

RHEL, Debian stable, SLES

Major version stays put for the life of the release. Security and selected feature backports come from the distro maintainer’s CVE team. CVE applicability requires checking the distro DB, not upstream.

HWE / rolling stable

Ubuntu LTS (Hardware Enablement stack), Fedora, openSUSE Tumbleweed

The kernel rolls forward with the upstream stable or LTS line every few months. Newer hardware support; same userland.

Mainline-tracking

Arch, Gentoo, Void, NixOS unstable

Distro kernel package follows mainline or LTS directly. Operator gets the newest features and the newest regressions on roughly upstream cadence.

Why this matters for the operator. CVE applicability depends on the distro kernel (with backports), not the headline upstream version. uname -r gives the distro string (6.5.0-15-generic, 5.14.0-427.42.1.el9_4.x86_64); the distro’s CVE database is the real source of truth on whether a fix is in.

Compartmentalization#

The three primitives the kernel exposes for compartmentalisation. Containers are built on top of them; understanding what each one isolates (and what it does not) is a defender’s, attacker’s, and forensic operator’s standing question.

        flowchart TB
    subgraph CONT["A container (or sandbox)"]
        direction LR
        subgraph NS["Namespaces<br/>(what it sees)"]
            MNT["mnt"]
            PID["pid"]
            NET["net"]
            USER["user"]
            IPC["ipc"]
            UTS["uts"]
            CGRP["cgroup"]
            TIME["time"]
        end
        subgraph CAPS["Capabilities<br/>(what it can do as root)"]
            CAP1["CAP_NET_ADMIN"]
            CAP2["CAP_SYS_ADMIN"]
            CAP3["CAP_DAC_OVERRIDE"]
            CAPN["..."]
        end
        subgraph CG["Cgroups<br/>(how much it gets)"]
            CPU["cpu"]
            MEM["memory"]
            IO["io"]
            PIDS["pids"]
            NETIO["net_cls / net_prio"]
        end
    end
    SECCOMP["seccomp filter<br/>(which syscalls it can call)"] --> CONT
    LSM["LSM policy<br/>SELinux / AppArmor"] --> CONT

Read together: namespaces decide what the process sees, capabilities decide what it can do if it is root inside, cgroups decide what resources it gets, and seccomp plus the LSM narrow which syscalls and operations it can issue at all. Docker, Podman, systemd-nspawn, Kubernetes, Bubblewrap, firejail, all compose the same primitives differently.

Capabilities split the historical “root or not” privilege check into ~40 individual capabilities (CAP_NET_ADMIN, CAP_SYS_ADMIN, CAP_DAC_OVERRIDE, CAP_SETUID, …). A binary can run as root but with a restricted capability set; an unprivileged user can be granted a single capability for a single binary.

$ getpcaps PID                             # what a process holds
$ cat /proc/PID/status | grep ^Cap         # the four sets, hex
$ getcap /usr/bin/ping                     # file capabilities
$ sudo setcap cap_net_raw+ep /usr/local/bin/myping
$ capsh --print                            # decode current shell's caps

Namespaces isolate a process’s view of a kernel resource. Eight in current kernels:

Namespace

Isolates

mnt

mount table

pid

process IDs (init becomes PID 1 inside)

net

network interfaces, routing, sockets

user

UID / GID mappings, capabilities

ipc

System V IPC, POSIX message queues

uts

hostname and domainname

cgroup

cgroup root view

time

boottime / monotonic clocks (newer kernels)

 
$ ls -la /proc/PID/ns/
$ readlink /proc/PID/ns/net
$ sudo nsenter -t PID -n -p ip a           # enter PID's net + pid ns
$ unshare --user --map-root-user --net bash  # roll a quick sandbox

Cgroups control resource use (CPU, memory, IO, PIDs) for a process tree. cgroup v1 is hierarchical per controller; cgroup v2 unifies into a single hierarchy. Modern systemd boxes use v2.

$ mount | grep cgroup
$ cat /sys/fs/cgroup/memory.max               # v2 root memory cap
$ systemd-cgls                                # tree of cgroups
$ systemd-cgtop                               # live resource use per group
$ cat /proc/PID/cgroup                        # which group a process is in

A container runtime (Docker, containerd, podman, runc, crun) combines all three: a new mnt + pid + net + user + ipc + uts namespace, a per-container cgroup, and a reduced capability set, with an OCI spec describing the lot.

Security#

The kernel exposes a layered set of security mechanisms; an operator can read which ones are active by inspection without touching the distro’s posture.

        flowchart TB
    subgraph US["User space"]
        APP["Process"]
    end
    SECCOMP["seccomp BPF filter<br/>(per-process syscall allowlist)"]
    APP --> SECCOMP
    SECCOMP --> SYS["Syscall ABI"]
    SYS --> LSM["LSM hooks<br/>SELinux | AppArmor | Tomoyo | Smack"]
    LSM --> CAPS["Capabilities check<br/>(40+ cap bits)"]
    CAPS --> YAMA["Yama (ptrace)"]
    YAMA --> CORE["Kernel subsystem<br/>(VFS, network, mm, ...)"]
    CORE --> IMA["IMA / EVM<br/>(file hash + signature)"]
    CORE --> LOCK["Lockdown<br/>none / integrity / confidentiality"]
    CORE --> EBPF["eBPF programs<br/>(LSM, XDP, kprobe, tracepoint)<br/>Falco, Tetragon, Cilium, Tracee"]
    EBPF -.observe.-> CORE

LSMs plug into kernel security hooks. Major implementations:

  • SELinux (Red Hat, Fedora): label-based mandatory access control. getenforce (Enforcing / Permissive / Disabled); ausearch -m AVC for denials.

  • AppArmor (Debian, Ubuntu, SUSE): path-based MAC. aa-status; /sys/kernel/security/apparmor/profiles.

  • Tomoyo, Smack: rarely seen in production.

Multiple LSMs can be stacked (lsm= boot parameter). Read which are active:

$ cat /sys/kernel/security/lsm
$ getenforce 2>/dev/null
$ aa-status 2>/dev/null

Seccomp filters syscalls per process. A profile expressed as BPF (cBPF historically, eBPF for seccomp_bpf) decides allow, errno, kill, or trace per syscall. Container runtimes ship default profiles; docker inspect shows the seccomp profile applied.

$ cat /proc/PID/status | grep Seccomp        # 0=disabled 1=strict 2=filter
$ scmp_sys_resolver bind                     # syscall name lookup

eBPF lets the kernel run sandboxed userspace-supplied programs at attach points (XDP, tc, kprobe, uprobe, tracepoint, LSM). Modern observability and runtime security (Falco, Tracee, Cilium, Tetragon, Aya-based agents) all anchor here.

$ sudo bpftool prog show
$ sudo bpftool map show
$ sudo bpftool perf show

Lockdown (since 5.4) gates kernel introspection and modification when the system was booted with Secure Boot. Three states:

$ cat /sys/kernel/security/lockdown
#  none / integrity / confidentiality

In confidentiality mode, /dev/mem, /dev/kmem, unsigned modules, kexec, hibernation, and most introspection surfaces are restricted; many kernel-debugging tradecraft moves fail.

IMA / EVM: Integrity Measurement Architecture and Extended Verification Module hash and (optionally) sign files at access; used in regulated environments to detect file tampering. /sys/kernel/ security/ima/ holds the policy and runtime measurements.

Yama: a tiny LSM that gates ptrace to children only by default. /proc/sys/kernel/yama/ptrace_scope.

Common Tasks#

The most common kernel-related questions an operator answers under time pressure on an unfamiliar host. Each starts with what’s running, not what’s documented.

Pin down the kernel for CVE applicability and exploit selection.

$ uname -srm
$ cat /proc/version
$ cat /proc/cmdline
$ ls -la /boot/vmlinuz-*

Inventory the security posture to know which moves are off the table.

$ cat /sys/kernel/security/lsm
$ getenforce 2>/dev/null
$ aa-status 2>/dev/null
$ cat /sys/kernel/security/lockdown
$ cat /proc/sys/kernel/kptr_restrict
$ cat /proc/sys/kernel/dmesg_restrict
$ cat /proc/sys/kernel/yama/ptrace_scope
$ cat /proc/sys/kernel/unprivileged_bpf_disabled

Audit loaded modules for unsigned modules, oddly-named modules, or known rootkit artifacts.

$ lsmod
$ awk '{print $1}' /proc/modules | sort > /tmp/proc_modules
$ ls /sys/module/ | sort > /tmp/sys_module
$ diff /tmp/proc_modules /tmp/sys_module     # divergence is suspicious
$ for m in $(awk '{print $1}' /proc/modules); do modinfo "$m" 2>/dev/null | head -3; done

Check namespace and capability surface of every running process, which is how you spot containers, sandboxes, and privileged services without ps cooperating.

$ for pid in $(ls /proc | grep -E '^[0-9]+$'); do
    ns=$(readlink /proc/$pid/ns/pid 2>/dev/null)
    echo "$pid $ns"
  done | sort -k2 -u
$ for pid in $(ls /proc | grep -E '^[0-9]+$'); do
    grep -H ^CapEff /proc/$pid/status 2>/dev/null
  done | grep -v 0000000000000000      # processes with non-empty effective caps

Read the live ring buffer for boot-time messages, hardware events, OOM kills, and security denials.

$ dmesg -T --level=err,warn
$ dmesg -T | grep -iE 'oom|killed|denied|tainted|secure boot'
$ journalctl -k -p err --since today

Tune a sysctl now and persist it if the engagement allows configuration changes.

$ sudo sysctl -w net.ipv4.ip_forward=1
$ sudo install -m 0644 /dev/stdin /etc/sysctl.d/99-fwd.conf <<'EOF'
net.ipv4.ip_forward=1
EOF
$ sudo sysctl --system

Trace what a process is asking the kernel for when its behavior is opaque from the outside.

$ sudo strace -f -e openat,connect,execve -p PID
$ sudo perf trace -p PID
$ sudo bpftrace -e 'tracepoint:syscalls:sys_enter_execve / pid == TARGET / { printf("%s\n", str(args->filename)); }'

Read kernel logs for tamper indicators during an investigation.

$ sudo journalctl -k --since '24 hours ago' | grep -iE 'taint|secure|module|bpf'
$ sudo dmesg -T | grep -iE 'oom|panic|oops|stack'
$ sudo ausearch -m AVC --start today           # SELinux denials

References#

  • kernel.org: the Linux kernel project homepage; releases, mailing list archives, the news.

  • Kernel Newbies: per-version changelogs in human-readable form (LinuxChanges).

  • Linux Documentation: the kernel’s own docs, including admin guide, driver model, ftrace, eBPF, tracing, security, and the sysctl reference.

  • LWN.net: kernel-development reporting and analysis; the standard secondary source.

  • Boot: the path that ends with the kernel running.

  • Memory, Processes: kernel-managed primitives.

  • Hardening: kernel hardening from the operator’s seat.

  • Devices: /dev, udev, the device model.

  • eBPF: the eBPF subsystem in depth.

  • man 7 capabilities, man 7 namespaces, man 7 cgroups, man 7 sysctl.conf, man 8 modprobe, man 5 modprobe.d.

Contents