التطبيع

Why the Same Text Can Look Identical But Be Different

Consider the letter é. You can encode it two ways: as a single precomposed character é (U+00E9, LATIN SMALL LETTER E WITH ACUTE) or as the sequence e (U+0065) followed by the combining acute accent ´ (U+0301). Both render identically, both are valid Unicode — but they are different byte sequences and will not compare as equal with a naive string comparison.

This is the core problem Unicode Normalization solves. Without normalization, the same word typed on macOS (which prefers decomposed forms) can fail to match the same word stored on a Linux system (which may prefer composed forms). Searching, sorting, deduplication, and hashing all break when you have silent encoding differences.

The Four Normal Forms

Unicode defines four normalization forms, each serving different needs:

The "K" variants additionally fold compatibility characters — characters that are semantically equivalent but visually or historically distinct, such as fi (fi ligature, U+FB01) → fi, or ² (superscript 2, U+00B2) → 2.

Using Normalization in Python

Python's unicodedata module provides normalization through a single function:

import unicodedata

text = "caf\u00e9"          # café with precomposed é (NFC)
nfd = unicodedata.normalize("NFD", text)
print(len(text))             # 4
print(len(nfd))              # 5 (e + combining acute)

# Roundtrip
assert unicodedata.normalize("NFC", nfd) == text

# Checking which form a string is already in
print(unicodedata.is_normalized("NFC", text))   # True
print(unicodedata.is_normalized("NFD", text))   # False

A safe comparison pattern for user-facing text:

def normalize_for_comparison(s: str) -> str:
    return unicodedata.normalize("NFC", s.casefold())

Quick Facts