Unicode

Unicode#

Reference of Unicode encodings, normalization forms, and operator-relevant blocks (control characters, lookalike / homoglyph attack risks, emoji, RTL marks). Pairs with ASCII for the 7-bit subset and Languages for script identification.

Encodings#

Encoding

Notes

UTF-8

variable 1-4 bytes; ASCII-compatible; the dominant web + Linux + macOS encoding.

UTF-16 LE / BE

variable 2 / 4 bytes (surrogate pairs); Windows native.

UTF-32

fixed 4 bytes; rarely used on disk.

UCS-2

legacy 2-byte (BMP only); deprecated.

US-ASCII

7-bit; the universal subset of UTF-8.

ISO-8859-1 (Latin-1

) 8-bit Western European; bridge during transition.

ISO-8859-15

Latin-9; adds €.

Windows-1252

MS extension to Latin-1.

Shift_JIS, EUC-JP

legacy Japanese; still some web use.

GB18030

Chinese; mandatory in PRC software.

Big5

Traditional Chinese.

KS X 1001 / EUC-KR

Korean.

UTF-8 byte ranges#

Byte 1

Encodes

0xxxxxxx (0x00-7F)

ASCII (1 byte).

110xxxxx (DF)

2-byte sequence.

1110xxxx (EF)

3-byte sequence.

11110xxx (F4)

4-byte sequence.

10xxxxxx (BF)

continuation byte.

Normalization forms#

Form

Notes

NFC

canonical composition; combine “e” + combining acute -> “é” (U+00E9). The web default.

NFD

canonical decomposition; “é” -> “e” + combining acute.

NFKC

compatibility composition; collapses fancy variants (e.g. ① -> 1).

NFKD

compatibility decomposition.

Equivalence

NFC vs NFD strings can be visually identical but bytewise different; always normalize before compare.

Surrogate pairs (UTF-16)#

Range

Use

U+D800-DBFF

high surrogate.

U+DC00-DFFF

low surrogate.

U+10000-10FFFF

supplementary planes; encoded as a surrogate pair in UTF-16; 4 bytes in UTF-8.

Important code points#

Code point

Use

U+0000 NULL U+0009 TAB

string terminator in C; parsers reject in many contexts.

U+000A LF

Unix line ending.

U+000D CR U+0020 SPACE

Mac Classic line ending; CRLF on Windows.

U+00A0 NBSP U+1680 OGHAM SPACE

non-breaking space; visually a space.

U+200B ZWSP

zero-width space; common spoofing vector.

U+200C ZWNJ

zero-width non-joiner.

U+200D ZWJ

zero-width joiner; underlies emoji combinations.

U+200E LRM

left-to-right mark (BiDi).

U+200F RLM U+2028 LINE SEP U+2029 PARA SEP

right-to-left mark.

U+202A LRE

left-to-right embedding (BiDi).

U+202B RLE

right-to-left embedding.

U+202C PDF

pop directional formatting.

U+202D LRO

left-to-right override.

U+202E RLO

right-to-left override; spoofing risk (Trojan Source).

U+2060 WJ

word joiner.

U+2066 LRI

left-to-right isolate (BiDi).

U+2067 RLI

right-to-left isolate.

U+2068 FSI

first-strong isolate.

U+2069 PDI

pop directional isolate.

U+FEFF BOM

byte-order mark; also ZWNBSP at start of file.

U+FFFC OBJ

object replacement character.

U+FFFD

replacement character (encoding error).

U+FFFE / U+FFFF

non-character.

U+E0000-E007F

tag characters; “watermark” trick (invisible, embed ASCII).

Homoglyphs / lookalike attacks#

Glyph

Code point Confusable

Latin

a / e / o / p / c Cyrillic а / е / о / р / с (U+0430, U+0435, U+043E, U+0440, U+0441).

Latin

A / B / C / E / H / K / M / O / P / T / X / Y / 3 Greek / Cyrillic / Cherokee equivalents.

Greek small

ο Cyrillic о and Latin o.

Latin

i Greek ι, Cyrillic і, Cherokee Ꭵ.

Latin

rn (rn) looks like m.

Hyphen

- en/em dashes, minus sign, fullwidth hyphen, Armenian hyphen.

Period

. Arabic small high dot, fullwidth full stop.

Slash

/ division slash, fraction slash, fullwidth solidus.

Quote

' / " curly quotes, prime, modifier letter.

Punycode

xn-- ASCII-encoded IDN; what the web displays as Unicode.

RTL / BiDi#

The Bidirectional Algorithm (UAX #9) lays text out by character-direction class (L, R, AL, AN, EN, …) and embedding-level overrides. Operationally relevant:

  • Trojan Source (CVE-2021-42574): RLO / LRE / PDF abuse to visually hide source-code differences.

  • iMessage / Telegram spoofing has used RLO to hide attachment file extensions.

  • Filename forensics, RLO in filename (“filenameSCD-1NCH3X.exe” visually shows “filename<reversed>X.exe”); modern OS strip on display.

Emoji#

Concept

Notes

Unicode 17.0 (2026)

~3700 standard emoji; Emoji 18 specs.

ZWJ sequences

👨‍👩‍👧‍👧 = man + ZWJ + woman + ZWJ + girl + ZWJ + girl; per-platform rendering varies wildly.

Skin-tone modifiers

Fitzpatrick 1-2 / 3 / 4 / 5 / 6 (U+1F3FB - U+1F3FF).

Variation selectors

U+FE0F (emoji) / U+FE0E (text) for default-text glyphs.

Regional indicator

pairs of A-Z indicators -> flag emoji.

Tag sequences

black-flag plus tag chars for sub-national flags (Scotland, Wales, England).

Scripts in Unicode#

Each script has a 4-letter ISO 15924 code (see Languages). Full coverage of nearly every living language; many historical (Egyptian Hieroglyphs, Cuneiform, Linear B). Ongoing additions each Unicode version.

Tooling#

Tool

Notes

hexdump / xxd

raw byte inspection.

unicode

CLI character lookup.

python -c "import unicodedata; print(unicodedata.name('x'))"

per-char metadata.

iconv

encoding conversion.

uconv (ICU)

full normalization + script analysis.

utf8.app

web tool.

unicode-table.com

web reference.

confusables (UTS #39)

Unicode confusable detection.

idna libraries

Python idna, JS punycode, etc. for IDN.

References#

Contents