Canonical Equivalence

What Is Canonical Equivalence?

Two Unicode strings are canonically equivalent if they represent the same abstract character sequence and should be treated as identical in all Unicode-conforming operations. They look the same, are pronounced the same, and have the same semantic value—the only difference is how the code points are arranged.

The canonical equivalence. The most common example is a precomposed character versus a base letter followed by a combining diacritic:

• U+00F1 LATIN SMALL LETTER N WITH TILDE (ñ) — a single code point
• U+006E LATIN SMALL LETTER N + U+0303 COMBINING TILDE — two code points

These two sequences are canonically equivalent. They must render identically and compare as equal after normalization.

Canonical Normalization Forms

Unicode defines two canonical normalization forms:

import unicodedata

# Two ways to write Spanish "ñ"
precomposed = "\u00F1"          # ñ as single code point
decomposed  = "\u006E\u0303"    # n + combining tilde

# They look the same:
print(precomposed, decomposed)
# ñ ñ

# But they are NOT equal as raw Python strings:
print(precomposed == decomposed)
# False
print(len(precomposed), len(decomposed))
# 1 2

# After NFC normalization they are equal:
nfc_pre = unicodedata.normalize("NFC", precomposed)
nfc_dec = unicodedata.normalize("NFC", decomposed)
print(nfc_pre == nfc_dec)
# True

# After NFD normalization they are also equal:
nfd_pre = unicodedata.normalize("NFD", precomposed)
nfd_dec = unicodedata.normalize("NFD", decomposed)
print(nfd_pre == nfd_dec)
# True
print(len(nfd_pre), len(nfd_dec))
# 2 2  (both are now decomposed)

Why This Matters

String comparison: Any application that compares user input against stored data must normalize both sides to the same form. Passwords, usernames, and search queries can silently differ due to canonical equivalence. The Python unicodedata.normalize("NFC", s) call is the standard fix.

Database storage: PostgreSQL uses NFC internally for text; MySQL's behavior depends on collation. Storing NFD strings in a NFC-collating database can cause subtle lookup failures.

File systems: macOS HFS+ normalizes filenames to NFD; Windows NTFS and Linux ext4 are normalization-agnostic. A file named ñ.txt may be stored differently on different systems, causing sync tools to create duplicates.

Quick Facts