Regex

Regex#

Regular expressions: the operator’s general-purpose pattern language. Reach for it any time the goal is to find, validate, extract, or rewrite text by structure rather than by exact string. The same syntax works (with documented quirks) across grep / rg / sed / awk on the shell, the re / regex modules in Python, RegExp in JavaScript, Pattern in Java, Regex in .NET, regexp in Go / Rust, YARA rules in malware tooling, and most editors.

This page is structured for lookup: read top-down to learn, or jump to one section by header. Engine-specific notes are called out inline; the engine-comparison section maps which features port across implementations.

Anchors#

Anchor

Description

Example

Valid

Invalid

^

Start of string or line.

^foam

foam

bath foam

\A

Start of string in any match mode.

\Afoam

foam

bath foam

$

End of string or line.

finish$

finish

finnish

\Z

End of string, or char before last new line, in any match mode.

finish\Z

finish

finnish

\z

End of string in any match mode.

\G

End of previous match, or start of string for the first match.

^(get|set)|\G\w+$

setValue

seValue

\b

Word boundary, between \w and \W.

\bis\b

This island is beautiful

This island isn't beautiful

\B

Not a word boundary.

\Bland

island

peninsula

Assertions#

Assertion

Description

Example

Valid

Invalid

(?=...)

Positive lookahead.

question(?=s)

questions

question

(?!...)

Negative lookahead.

answer(?!s)

answer

answers

(?<=...)

Positive lookbehind.

(?<=appl)e

apple

application

(?<!...)

Negative lookbehind.

(?<!goo)d

mood

good

Character class#

Class

Description

Example

Valid

Invalid

[ ]

Class definition.

[axf]

a, x, f

b

[ - ]

Class definition range.

[a-c]

a, b, c

d

[ \ ]

Escape inside class.

[a-f.]

a, f, .

g

[^ ]

Not in class.

[^abc]

d, e

a

[:class:]

POSIX class.

[:alpha:]

string

0101

.

Any char except newline.

b.ttle

battle, bottle

bttle

\s

White space, [\n\r\f\t ].

good\smorning

good morning

good.morning

\S

Non-white space, [^\n\r\f\t ].

good\Smorning

goodmorning

good morning

\d

Digit.

\d{2}

23

1a

\D

Non-digit.

\D{3}

foo, bar

fo1

\w

Word, [a-zA-Z0-9_].

\w{4}

v411

v4.1

\W

Non-word.

.$%?

.$%?

.ab?

Sequence#

Sequence

Description

Example

Valid

Invalid

|

Alternation.

apple|orange

apple, orange

melon

( )

Subpattern.

foot(er|ball)

footer or football

footpath

(?P<name>...)

Capture submatch into name.

(?P<greeting>hello)

hello

hallo

(?:...)

Subpattern, no capture.

(?:hello)

hello

hallo

Quantifiers#

Quantifier

Description

Example

Valid

Invalid

+

One or more.

ye+ah

yeah, yeeeah

yah

*

Zero or more.

ye*ah

yeeah, yeeeah, yah

yeh

?

Zero or one.

yes?

yes, ye

yess

??

Zero or one, lazy.

yea??h

yeah

yeaah

+?

One or more, lazy.

/<.+?>/g

<P>foo</P> matches only <P> and </P>

*?

Zero or more, lazy.

/<.*?>/g

<html>

{n}

n times exactly.

fo{2}

foo

fooo

{n,m}

From n to m times.

go{2,3}d

good, good

gooood

{n,}

At least n times.

go{2,}

goo, gooo

go

(?(condition)...)

If-then pattern.

(<)?[p](?(1)>)

<p>, p

<p

(?(condition)...|...)

If-then-else pattern.

`^(?(?=q)que

ans`

question, answer

Escapes#

Char

Description

\

General escape.

\n

New line.

\r

Carriage return.

\t

Tab.

\v

Vertical tab.

\f

Form feed.

\a

Alarm.

[\b]

Backspace.

\e

Escape.

\cchar

Ctrl + char (\cc is Ctrl+c).

\ooo

Three-digit octal (\123).

\xhh

One or two-digit hexadecimal (\x10).

\x{hex}

Any hexadecimal code (\x{1234}).

\p{xx}

Char with Unicode property (\p{Arabic}).

\P{xx}

Char without Unicode property.

Unicode Properties#

\p{...} selects characters by Unicode property; \P{...} is the negation. The most-used categories are below. Supported by PCRE / PCRE2, Python re, JavaScript with the u flag (ES2018+), Java, .NET, Ruby, Rust regex. Not supported by Go regexp, POSIX BRE/ERE, or stock grep -P on some GNU builds.

General categories#

Property

Matches

\p{L}

any letter

\p{Ll}

lowercase letter

\p{Lu}

uppercase letter

\p{Lt}

titlecase letter

\p{Lo}

other letter (no case, e.g. CJK)

\p{M}

any combining mark

\p{Mn}

non-spacing combining mark (accent)

\p{N}

any numeric

\p{Nd}

decimal digit

\p{Nl}

letter-like number (Roman numerals)

\p{No}

other numeric (fractions, superscript)

\p{P}

any punctuation

\p{Pd}

dash punctuation

\p{Ps} / \p{Pe}

opening / closing punctuation

\p{Pi} / \p{Pf}

initial / final quote punctuation

\p{S}

any symbol

\p{Sc}

currency symbol

\p{Sm}

math symbol

\p{Z}

any separator

\p{Zs}

space separator

\p{Zl} / \p{Zp}

line / paragraph separator

\p{C}

any “other” (control, surrogate, private use)

\p{Cc}

control character

\p{Cf}

format character (zero-width joiner, RTL marks)

Scripts (common)#

Property

Matches

\p{Latin}

Latin script

\p{Greek}

Greek script

\p{Cyrillic}

Cyrillic script

\p{Arabic}

Arabic script

\p{Hebrew}

Hebrew script

\p{Han}

Han / CJK ideographs

\p{Hiragana} / \p{Katakana}

Japanese kana

\p{Hangul}

Korean Hangul

\p{Devanagari}

Devanagari

\p{Thai}

Thai script

Operator picks#

Goal

Pattern

any letter, locale-independent

\p{L}+

alphanumeric (any script)

[\p{L}\p{N}]+

strip emoji / pictographs

\p{So}

detect bidi text injection

[\p{Cf}\p{Bidi_Control}]

punctuation-only line

^\p{P}+$

non-ASCII whitespace

[\p{Z}&&[^ ]] (Java / Ruby)

Modifiers#

Modifier

Description

g

Global match.

i

Case-insensitive, match both upper and lower.

m

Multiple lines.

s

Single line (by default).

x

Ignore whitespace, allow comments.

A

Anchored, pattern is forced to ^.

D

Dollar end only, $ matches only at the end.

S

Extra analysis performed, useful for non-anchored patterns.

U

Ungreedy, greedy patterns become lazy by default.

X

Additional functionality of PCRE (PCRE extra).

J

Allow duplicate names for subpatterns.

u

Unicode, pattern and subject are treated as UTF-8.

Backreferences#

Reuse a captured group in the same pattern (or in the replacement text). PCRE / Python / Ruby / Perl use \1 … \9 and named-group syntax; ECMAScript and Go’s regexp differ slightly.

Syntax

Description

Example

\1 … \9

Backreference to numbered group.

(\w+)\s+\1 matches a doubled word.

\g<1>

PCRE explicit numbered backref (avoids digit-collision).

(\w+)\g<1>

(?P<name>...)

Python / PCRE named capture.

(?P<host>\S+)

(?<name>...)

ECMAScript / .NET / Ruby named capture.

(?<host>\S+)

(?P=name)

Python backref by name.

(?P<q>['\"]).+(?P=q) matches same quote on both sides.

\k<name>

PCRE / .NET / Ruby backref by name.

(?<q>['\"]).+\k<q>

Substitution#

The replacement-side syntax used by sed, awk, perl, rg --replace, and many editors.

Token

Description

Example

$1 / \1

Backref to capture group N (engine-dependent).

s/(\w+) (\w+)/\2 \1/ swaps two words.

$& / \0

The entire match.

s/\d+/[&]/g brackets every number.

$' / $ ``

Text after / before the match (Perl / JS).

rare; mostly Perl one-liners.

\u / \l

Uppercase / lowercase next char (Perl, vim).

s/(\w+)/\u\1/ capitalizes.

\U / \L

Uppercase / lowercase to \E.

s/(\w+)/\U\1\E/ uppercases the word.

\E

End \U / \L / \Q block.

paired with \U / \L.

Substitution by Engine#

Engine

Replacement backref syntax

PCRE / Perl

$1 / $& / ${name}

Python

\1 / \g<name>

sed (BRE)

\1

sed -E (ERE)

\1

JavaScript

$1 / $<name>

.NET

$1 / ${name}

Java

$1

Engines#

The major regex implementations the operator meets. Two families: backtracking (PCRE-flavored, feature-rich, ReDoS-prone) and finite-state (RE2-style, linear-time, no backrefs / lookarounds).

Engine

Type

Notes

PCRE

backtracking

De-facto reference for “advanced” regex. Rich features; ReDoS-prone.

PCRE2

backtracking + JIT

Successor; includes JIT. Original PCRE deprecated.

Perl

backtracking

The original “modern” regex.

Python re

backtracking

Stdlib; Unicode-conservative; no possessive / atomic by default.

Python regex

backtracking

Third-party; closer to PCRE; named groups, possessive, Unicode properties.

JavaScript RegExp

backtracking

ES2018+: lookbehind, named groups, s flag, u flag, \p{...}.

Java Pattern

backtracking

Possessive + atomic; Unicode properties; lookbehind length constraint.

.NET Regex

backtracking

Rich features; balancing groups; possessive.

Ruby Regexp

Onigmo backtracking

Similar to Java + Perl extensions.

Rust regex

finite-state (NFA / DFA)

RE2-style; no backreferences, no lookarounds; guaranteed linear time.

Go regexp

RE2-derived

Like Rust; same trade-offs.

RE2

NFA / DFA

Google; safety-first; the model for Go / Rust.

Hyperscan

NFA / streaming

Intel; high-throughput packet inspection.

Feature Portability#

Which features port across engines. Y = supported, N = not supported, ~ = supported with a flag or alternative syntax.

Feature

PCRE / Perl

Python re

ECMAScript

POSIX BRE

Go regexp

\d \w \s

Y

Y

Y

N

Y

Lookahead (?=...) / (?!...)

Y

Y

Y

N

N

Lookbehind (?<=...) / (?<!...)

Y

Y (3.7+)

Y (ES2018+)

N

N

Named captures

Y

Y

Y (ES2018+)

N

Y

Backreferences \1

Y

Y

Y

~ (BRE: \1)

N (RE2)

Possessive quantifiers a++

Y

N

N

N

N

Atomic groups (?>...)

Y

Y (3.11+)

N

N

N

Inline flags (?i) … (?-i)

Y

Y

~ (ES2025 modifiers)

N

Y

Unicode properties \p{L}

Y

Y

Y (u flag)

N

N

Linear-time guarantee

N

N

N

N

Y (RE2)

POSIX Classes#

Portable name-based character classes. Use inside a bracket expression: [[:digit:][:alpha:]_].

Class

Matches

[:alpha:]

[A-Za-z]

[:alnum:]

[A-Za-z0-9]

[:digit:]

[0-9]

[:xdigit:]

[0-9A-Fa-f]

[:lower:]

[a-z]

[:upper:]

[A-Z]

[:space:]

whitespace (\s)

[:blank:]

[ \t]

[:punct:]

punctuation

[:cntrl:]

control characters

[:print:]

printable ([:graph:] + space)

[:graph:]

visible ([:alnum:][:punct:])

[:ascii:]

[\x00-\x7F]

Common Tasks#

The operator’s daily-driver invocations.

Goal

Command

extended regex with grep

grep -E 'pat1|pat2' file

Perl-compatible regex with grep

grep -P '\d{3}-\d{4}' file

ripgrep (PCRE2 with -P)

rg -P '(?<=user=)\w+' /var/log

case-insensitive search

grep -i pat file / rg -i pat

only the matched part

grep -oE pat file / rg -oP pat

count matches per file

grep -c pat file / rg -c pat

invert match

grep -v pat file / rg -v pat

recursive search

grep -RIn pat dir/ / rg pat dir/

search history-respecting (.gitignore)

rg pat (default)

sed substitution (BRE)

sed 's/old/new/g' file

sed substitution (ERE)

sed -E 's/(\w+) (\w+)/\2 \1/' file

sed substitution (PCRE)

sed -E -e 's/.../...' file

in-place edit

sed -i.bak 's/old/new/g' file

awk match

awk '/pat/ {print $2}' file

awk capture (gawk)

gawk 'match($0, /(\w+)/, a) {print a[1]}' file

perl one-liner (capture)

perl -ne 'print "$1\n" if /(\d{4})/' file

find by regex name

find . -regex '.*\.\(c\|h\)$'

rename by regex

rename 's/\.txt$/.md/' *.txt

URL-extract from text

rg -oP 'https?://\S+' file

delete every blank line

sed -i '/^[[:space:]]*$/d' file

validate regex compiles

echo '' | grep -E -- "<pat>" 2>&1 || echo BAD

Common Patterns#

Frequently-needed building blocks. For richer IOC patterns see Indicators of Compromise.

Match

Pattern

integer

-?\d+

signed float

-?\d+(?:\.\d+)?

hex literal

0[xX][0-9A-Fa-f]+

leading / trailing whitespace

^[ \t]+|[ \t]+$

blank line

^[[:space:]]*$

quoted string

"[^"\\]*(?:\\.[^"\\]*)*"

email (pragmatic)

[A-Za-z0-9._%+-]+@(?:[A-Za-z0-9-]+\.)+[A-Za-z]{2,}

URL

https?://[^\s<>"']+

IPv4

\b(?:(?:25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)\.){3}(?:25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)\b

MAC address

(?:[0-9A-Fa-f]{2}[:-]){5}[0-9A-Fa-f]{2}

UUID v4

[0-9a-fA-F]{8}-(?:[0-9a-fA-F]{4}-){3}[0-9a-fA-F]{12}

SHA-256

\b[A-Fa-f0-9]{64}\b

ISO 8601 date

\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}(?:Z|[+-]\d{2}:?\d{2})

CVE id

CVE-\d{4}-\d{4,7}

IPv6 (uncompressed)

\b(?:[A-Fa-f0-9]{1,4}:){7}[A-Fa-f0-9]{1,4}\b

Bitcoin address

\b(?:bc1[ac-hj-np-z02-9]{8,87}|[13][a-km-zA-HJ-NP-Z1-9]{25,34})\b

Ethereum address

\b0x[a-fA-F0-9]{40}\b

JWT

[A-Za-z0-9_-]+\.[A-Za-z0-9_-]+\.[A-Za-z0-9_-]+

Phone (NANP, US / Canada)

\(?(\d{3})\)?[-. ]?(\d{3})[-. ]?(\d{4})

US ZIP

\d{5}(?:-\d{4})?

BIP-39 mnemonic (12 / 24 words)

\b(?:[a-z]{3,8}\s+){11,23}[a-z]{3,8}\b

ReDoS Minefield (avoid)#

Patterns that explode on attacker-controlled input. Each runs in exponential time on a long sequence of the inner character.

Anti-pattern

Why

(a+)+$

Nested + on the same character; exponential backtracking on long a-strings.

(a*)*

Same structure with *.

(.+)+$

Nested + on .; catastrophic on any long line of user input.

([a-zA-Z]+)*$

Nested * over a class; same explosion class.

(a|aa)+$

Overlap-based ambiguity: two ways to consume each a.

\w+\d+\w+

Ambiguous boundary; quadratic on long inputs.

Mitigations: possessive quantifiers (a++), atomic groups ((?>a+)), tighter character classes ([^x] instead of .), or pick a DFA / RE2 engine (Go regexp, Rust regex).

Pitfalls#

Pitfall

Notes

Greedy by default

.+ swallows as much as it can. Use .+? (lazy) or a more specific class like [^"]+ when matching between delimiters.

^ / $ and multiline

In most engines, ^ / $ match string boundaries by default. Pass the m flag to make them match line boundaries.

. and newlines

. skips newlines by default. s flag (PCRE / Python / Go) or [\s\S] (JS) crosses them.

Catastrophic backtracking

Nested quantifiers like (a+)+ over a long input explode. Rewrite as a+ or use atomic groups (?>a+) / possessive a++.

Word boundary surprises

\b is between \w ([A-Za-z0-9_]) and \W. Punctuation-adjacent matches behave oddly; use explicit lookarounds for non-ASCII.

Anchored vs unanchored

grep PATTERN is unanchored (substring). ^PATTERN$ forces full-line match.

Locale and Unicode

[a-z] is ASCII; for accented letters use \p{Lower} (PCRE) or set LC_ALL=en_US.UTF-8 for POSIX engines.

Engine flavors

grep (BRE) vs grep -E (ERE) vs grep -P (PCRE). Some metacharacters require a backslash in BRE (e.g. \(group\), \{2,3\}).

Linear-time vs backtracking

Go regexp and RE2 are O(n) but lack backreferences / lookarounds. PCRE supports them but can be exponential.

HTML / XML with regex

Don’t. Use a parser (xmllint, htmlq, BeautifulSoup). Regex over balanced tags is a known anti-pattern.

References#

Contents