Regex

Regex#

The operator’s bread-and-butter pattern language. Every triage of a log haul, every indicator extraction from a dump, every quick filter across a packet capture or a social feed touches a regex. Regular expressions are a small declarative language for matching patterns in text. Underneath them is the theory of regular languages and finite automata; in practice, regex is the second-most-used DSL in software (after SQL).

Every modern language ships a regex engine. They differ enough in feature set and semantics that “regex” is really a family of dialects, and the operator who works across grep, ripgrep, Python, and JavaScript will hit those differences in the middle of a job.

Theory in One Paragraph#

A regular expression denotes a regular language, a set of strings recognizable by a finite automaton. Pure regular expressions can be compiled to a deterministic finite automaton (DFA) and matched in O(n) time. Most production engines extend the language with backreferences, lookarounds, and recursion, which makes them strictly more powerful but also enables exponential-time worst cases.

Core Syntax#

The vocabulary that 90% of regexes draw from. Literals, character classes, quantifiers, alternation, grouping, and anchors; everything else is layered on top of these. The table below is the operator’s quick reference for the patterns that show up in nearly every dialect.

Pattern

Matches

a

literal a

.

any character (often except newline)

\d

digit

\w

word character ([A-Za-z0-9_] typically)

\s

whitespace

[abc]

one of a, b, c

[^abc]

anything except those

[a-z]

range

a*

zero or more of a

a+

one or more

a?

zero or one

a{n}

exactly n

a{n,m}

between n and m

ab|cd

ab or cd

(...)

capturing group

(?:...)

non-capturing group

^ / $

start / end of line

\b

word boundary

Greedy vs. Lazy#

By default, + and * are greedy; they match as much as possible. Append ? to make them lazy:

pattern    input "<a><b>"        match
<.+>       greedy                <a><b>
<.+?>      lazy                  <a>

Anchors#

Zero-width markers that match a position rather than characters. They constrain where a pattern may begin or end – the difference between matching cat anywhere and matching it as a whole word at the start of a line. Multiline mode shifts which anchors mean line edges versus input edges.

  • ^, start of input (or start of line in multiline mode).

  • $, end of input (or end of line).

  • \A, \z, absolute start / end of input regardless of mode.

  • \b, \B, word boundary, non-word-boundary.

Tricky on multiline input. Most engines have MULTILINE / DOTALL / IGNORECASE flags.

Capturing#

Groups serve two purposes: scoping a quantifier to multiple characters, and pulling matched text out for substitution or inspection. Most regexes use both. Naming captures keeps long patterns readable; non-capturing groups keep memory usage and group-number arithmetic predictable.

  • (...), capturing group; available as $1 / \1 / group(1).

  • (?:...), non-capturing; saves memory and avoids confusion.

  • (?<name>...) (or (?P<name>...) in Python), named capture.

  • \1, backreference; matches what group 1 captured.

Backreferences make a pattern non-regular in the formal sense. They’re also the primary source of catastrophic backtracking.

Lookarounds#

Assertions that test what surrounds a position without consuming the surrounding characters. They turn into particularly heavy weapons in two cases (“match X only when followed by Y” and “match X only when not preceded by Y”) that would otherwise need two passes over the input.

Zero-width assertions, match without consuming.

  • (?=...), positive lookahead.

  • (?!...), negative lookahead.

  • (?<=...), positive lookbehind.

  • (?<!...), negative lookbehind.

\d+(?= dollars)       # digits followed by " dollars" (without consuming)
(?<!not )good         # "good" not preceded by "not "

Useful, but most regexes that “need” a lookaround would be cleaner as two passes over the data.

Engines and Dialects#

Regex engines split into two families with different performance profiles and feature sets. Backtracking engines support the full PCRE feature set including backreferences and lookarounds; DFA / NFA engines drop those features in exchange for guaranteed linear-time matching.

Two main implementation styles.

Backtracking (PCRE, Python re, Perl, Java Pattern, JavaScript RegExp, .NET, Ruby’s default):

  • Flexible, supports backreferences and lookarounds.

  • Susceptible to catastrophic backtracking on adversarial input.

  • Variable per-engine syntax.

NFA / DFA based (Google RE2, Rust regex, Hyperscan, Go regexp):

  • O(n) per match, guaranteed.

  • No backreferences (and no recursion).

  • Immune to ReDoS.

For untrusted input, prefer DFA-style engines.

Common Dialects#

  • POSIX BRE / ERE, in grep, sed. Different escaping rules from the rest of the world.

  • PCRE (Perl-Compatible), the de-facto standard for backtracking engines.

  • ECMAScript, JavaScript’s; subtly different lookbehind support history.

  • RE2 / re2 syntax, a subset of PCRE without features that break linear time.

  • Vim regex, has its own conventions.

Translating between dialects is a frequent source of bugs.

Catastrophic Backtracking#

The single most common security and operational pitfall with regex. A pattern that nests quantifiers ((a+)+, (a*)*) can take exponential time on adversarial input, turning a regex into a denial-of-service vector. Mitigations range from rewriting the pattern to swapping the engine.

The classic ReDoS pattern.

(a+)+$

On input aaaaaaaaaaaaaaaaaaaaaaa!, a backtracking engine tries exponentially many groupings before failing. A user-controlled regex running on attacker input is a denial-of-service vector.

Mitigations.

  • Avoid nested quantifiers ((a+)+, (a*)*).

  • Use possessive quantifiers (a++, a*+) where supported.

  • Use atomic groups (?>...).

  • Best: use a DFA-style engine (RE2 / Rust regex / Go regexp) for untrusted input.

Practical Patterns#

A small library of regexes that operators reach for in scripts and one-off extractions. Each is “good enough” for casual work; a strict format like RFC-822 email addresses requires a real parser. The patterns below are tuned for readability over maximal precision.

# email-ish (not strictly RFC)
^[\w.+-]+@[\w-]+(\.[\w-]+)+$

# URL-ish
https?://[\w.-]+(:\d+)?(/[\w./?&=#%+-]*)?

# IPv4 (loose)
\b\d{1,3}(\.\d{1,3}){3}\b

# ISO-8601 date
\d{4}-\d{2}-\d{2}

# quoted string (no escapes)
"([^"]*)"

# whole-word match
\bword\b

Real-world email / URL / IP patterns get long. When precision matters, use a parser (or a vetted library) rather than a regex.

Substitution#

Find-and-replace driven by captured groups. The substitution side has its own escape rules, varying per engine ($1 versus \1 versus $& for “the whole match”), and is a frequent source of papercuts when porting a substitution between languages.

Most engines support replacement using captured groups.

sed -E 's/^([0-9]+) (.*)$/\2 \1/'        # swap two fields
"  ".replace(/\s+/g, ' ')                # collapse whitespace (JS)
re.sub(r'(\d+)', r'[\1]', text)          # wrap numbers (Python)

Substitution often has its own escape rules ($1 vs. \1 vs. $&).

Flags#

Per-pattern modifiers that toggle case sensitivity, multiline behavior, dotall, extended whitespace, Unicode awareness, and global matching. Most engines support both pattern-level flag arguments and inline flags via (?i) / (?m) / (?s) inside the pattern.

Flag

Effect

i

case-insensitive

m

multiline (^ / $ per line)

s

dotall (. matches newlines)

x

extended (whitespace and comments allowed)

u

Unicode-aware (varies; on by default in many)

g

global (find all matches; JS-specific syntax)

Inline flags: (?i)abc enables case-insensitive for abc.

Unicode#

Engines vary widely in how seriously they treat Unicode. Some match \d against any Unicode digit; others limit it to ASCII unless a flag is set. Property classes (\p{L}, \p{N}) give precise control where the engine supports them, including specific scripts and categories.

  • \p{L}, any letter (Unicode property).

  • \p{N}, any number.

  • \p{Nd}, \p{Lu}, \p{Greek}, specific subcategories.

Available in PCRE, .NET, Java (\p{...}), Rust regex (with the unicode feature), Python regex package (not stdlib re).

In \d, \w, etc., behavior varies by engine, ASCII-only or Unicode-aware depending on flag.

Tools#

The interactive testers and grep variants that operators use when authoring or debugging a regex. regex101 and regexr are the lab; debuggex is the visualization aid for nested patterns; the grep family on the command line is where the finished pattern earns its keep.

  • regex101.com, interactive tester with multiple flavors and step-by-step explanation.

  • regexr.com, alternative tester.

  • debuggex, visualizes regex as railroad diagrams.

  • pcregrep, rg (ripgrep), ugrep, ag, regex grep variants.

When to Use#

The kinds of problem where regex is the right reach. Quick extraction, lexer-shaped tokens, simple validation, and editor-driven search-and-replace all fit the format’s strengths, pattern matching against linear text where precision can be relaxed for brevity.

  • Quick string extraction or validation.

  • Scripts and one-offs.

  • Lexer-shaped tokens in a parser.

  • Search-and-replace in editors.

When not to use.

  • Parsing real grammars (HTML, JSON, source code), use a parser.

  • Untrusted regex on untrusted input, ReDoS risk.

  • Anything that needs to evolve much over time, regex is fragile to change.

  • Performance-critical hot loops on large inputs, compile once, or use a different algorithm.

The Cardinal Rules#

  1. Compile the regex once; reuse the compiled object.

  2. Anchor your regex unless you really mean “anywhere”.

  3. Use non-capturing groups when you don’t need the capture.

  4. For untrusted input, use a DFA engine.

  5. When the pattern feels hard, switch to a parser.

Contents