Unicode General Categories Explained

Every character in Unicode is assigned a General Category — a two-letter code that classifies the character as a letter, number, symbol, punctuation mark, separator, mark, or "other." This classification is the single most important property for any code that processes text, because it tells you what a character is before you know anything else about it. Regex engines, word-break algorithms, identifier validation, and security scanners all rely on General Category as their first line of decision-making.

This guide walks through the full General Category hierarchy, explains every two-letter value, shows you how to query categories in Python, JavaScript, Java, and regular expressions, and highlights the practical scenarios where categories matter most.

The Two-Level Hierarchy

Unicode organizes General Categories into 7 major classes, each subdivided into specific two-letter values. The first letter of the code identifies the major class; the second letter narrows it down.

When the Unicode Standard or a regex engine refers to Lu, the L means "Letter" and the u means "uppercase." When it refers to just L, it means all Letter subcategories combined.

Complete List of 30 General Categories

Letters (L) — 6 values

The Lo category dwarfs all others because it contains CJK ideographs, Hangul syllables, and characters from scripts that do not distinguish case. Lt is the smallest: it exists only for a handful of digraph characters in Latin Extended-B where only the first component is capitalized (e.g., U+01C5 LATIN CAPITAL LETTER D WITH SMALL LETTER Z WITH CARON: Dž).

Marks (M) — 3 values

Marks are essential for scripts like Devanagari, Thai, Arabic, and Hebrew where vowels or tone indicators are written as combining characters above, below, or beside the base letter.

Numbers (N) — 3 values

Nd includes digits from every script that has a positional numeral system — Arabic-Indic, Devanagari, Bengali, Thai, and dozens more. All Nd digits have a Numeric_Value property from 0 to 9, which makes them usable in numeric parsing.

Punctuation (P) — 7 values

The distinction between Pi (initial quote) and Pf (final quote) matters for typographic algorithms that need to pair opening and closing quotation marks correctly.

Symbols (S) — 4 values

So (Other Symbol) is the catch-all for arrows, box-drawing characters, dingbats, emoji (when not modified by variation selectors), and many more.

Separators (Z) — 3 values

Zl and Zp each contain exactly one character. In practice, most software uses \n (U+000A, category Cc) as a line separator instead of U+2028.

Other (C) — 5 values

Cs code points never appear as isolated characters — they exist only as part of UTF-16 surrogate pairs. Cn is the largest category by code point count, covering the vast unassigned space of Unicode's 17 planes.

Querying General Category in Code

Python

Python's unicodedata module provides the category() function:

import unicodedata

unicodedata.category("A")   # 'Lu' — Uppercase Letter
unicodedata.category("3")   # 'Nd' — Decimal Digit
unicodedata.category("$")   # 'Sc' — Currency Symbol
unicodedata.category("\u200D")  # 'Cf' — Format (Zero Width Joiner)

To test for a major class, check the first character of the result:

def is_letter(char: str) -> bool:
    return unicodedata.category(char).startswith("L")

def is_symbol(char: str) -> bool:
    return unicodedata.category(char).startswith("S")

JavaScript

JavaScript exposes General Category through Unicode property escapes in regular expressions (ES2018+):

// Test for a specific category
/^\p{Lu}$/u.test("A");      // true — Uppercase Letter
/^\p{Nd}$/u.test("٣");      // true — Arabic-Indic digit three

// Test for a major class
/^\p{Letter}$/u.test("あ");  // true
/^\p{Number}$/u.test("½");   // true
/^\p{Symbol}$/u.test("€");   // true

The long names (Letter, Number, Symbol) match the major class; the short names (Lu, Nd, Sc) match the specific two-letter category.

Java

int cp = "A".codePointAt(0);
int type = Character.getType(cp);
// type == Character.UPPERCASE_LETTER (1)

// Or use regex with Unicode categories
"A".matches("\\p{Lu}");     // true
"3".matches("\\p{Nd}");     // true

Regular Expressions (PCRE, Python re)

Most modern regex flavors support \p{Category}:

\p{L}     — Any letter (Lu + Ll + Lt + Lm + Lo)
\p{Lu}    — Uppercase letter only
\p{N}     — Any number (Nd + Nl + No)
\p{Nd}    — Decimal digit only
\p{P}     — Any punctuation
\p{S}     — Any symbol
\p{Z}     — Any separator
\p{C}     — Any "other" (control, format, surrogate, private use, unassigned)

In Python, you need the regex package (not the built-in re) to use \p{} syntax, or you can use unicodedata.category() directly.

Why General Categories Matter

1. Identifier Validation

Programming languages define valid identifier characters using General Categories. For example, Python 3's identifier rules follow Unicode Standard Annex #31 (UAX #31):

• Start characters: Lu, Ll, Lt, Lm, Lo, Nl, plus underscore
• Continue characters: all start characters plus Mn, Mc, Nd, Pc

This means café = 42 is valid Python, because é is Ll (lowercase letter), and price₹ = 100 is valid because ₹ is not — wait, ₹ is Sc (currency symbol), so it is not a valid identifier character. General Category is the gatekeeper.

2. Word Segmentation

The Unicode word-break algorithm (UAX #29) uses General Category to decide where words begin and end. Letters (L) and combining marks (M) stick together; spaces (Zs) and punctuation (P) create breaks. This is how Ctrl+Shift+Left selects a whole word in your editor.

3. Security — Confusable Detection

Unicode Technical Report #36 (Unicode Security Mechanisms) relies on General Category to flag suspicious text. A string that mixes Lo characters from different scripts (e.g., Latin "a" and Cyrillic "а") is a confusable candidate. Category checks are the first filter in any mixed-script detection algorithm.

4. Text Rendering

Font engines use General Category to decide rendering behavior. Mn (non-spacing mark) characters must be rendered on top of the preceding base character. Zs characters produce whitespace. Cf characters are invisible but affect shaping (like ZWJ in emoji sequences).

5. Data Cleaning and Validation

When cleaning user input, General Category tells you what to keep and what to strip:

import unicodedata

def clean_text(text: str) -> str:
    """Remove control characters and unassigned code points."""
    return "".join(
        c for c in text
        if unicodedata.category(c) not in ("Cc", "Cf", "Cs", "Co", "Cn")
        or c in ("\n", "\t")  # Keep common whitespace controls
    )

Edge Cases and Surprises

Emoji are So (Other Symbol) — despite being rendered as colorful images, base emoji characters have the General Category So. Emoji modifiers and ZWJ sequences involve Cf and Sk characters as well.

Digits from other scripts are all Nd — Bengali ৩ (U+09E9), Thai ๓ (U+0E53), and Devanagari ३ (U+0969) are all Nd, just like ASCII 3. They all have Numeric_Value=3. However, most parsers only accept ASCII digits for numbers.

Underscore is Pc (Connector Punctuation) — not a letter, not a symbol. This is why it requires special handling in identifier rules: Pc is explicitly added to the "allowed" set for continue characters.

Surrogates (Cs) are ghosts — they exist in the code point space but should never appear as characters in well-formed text. They are artifacts of UTF-16 encoding.

The category of an unassigned code point is Cn — and it can change when a future Unicode version assigns a character to that code point. Code that hard-codes behavior based on Cn may break when Unicode is updated.

Quick Reference Card

Summary

• Unicode defines 30 General Categories organized into 7 major classes (L, M, N, P, S, Z, C).
• Every code point has exactly one General Category — it is a required, non-optional property.
• Categories drive identifier rules, word segmentation, security analysis, rendering, and data cleaning.
• Query categories with unicodedata.category() in Python, \p{Lu} in regex, or Character.getType() in Java.
• The Lo category contains the majority of assigned characters (CJK, Hangul, scripts without case), while Cn (unassigned) covers most of the total code point space.
• General Category is stable for assigned characters — once a code point is assigned, its category never changes in future Unicode versions.