String Comparison

What is Unicode String Comparison?

Unicode string comparison is the process of determining whether two Unicode strings are equal, or which comes first in a sorted order. It sounds simple, but Unicode's encoding flexibility makes naive byte-by-byte comparison unreliable. The same text can be encoded in multiple valid ways, and language-specific sorting rules vary enormously across the world's scripts and locales.

Binary Comparison Pitfalls

The most obvious approach — comparing strings byte-by-byte or code-point-by-code-point — fails silently in many real-world situations. Consider the character é. It can be encoded as:

• U+00E9 — a single precomposed code point (NFC form)
• U+0065 U+0301 — the base letter e followed by a combining acute accent (NFD form)

These two sequences are canonically equivalent under the Unicode Standard but are binary-unequal. A naive comparison would declare them different strings, even though they represent identical text. This causes bugs in search, deduplication, password checks, and username lookups.

NFC Normalization Before Comparing

The standard defense is to normalize both strings to the same Unicode normalization form before comparing. NFC (Canonical Decomposition followed by Canonical Composition) is the recommended form for most applications because it produces compact, precomposed forms that work well with legacy systems.

import unicodedata

a = "e\u0301"          # e + combining acute (NFD-style)
b = "\u00e9"           # precomposed é (NFC-style)

a == b                                        # False — binary comparison
unicodedata.normalize("NFC", a) == unicodedata.normalize("NFC", b)  # True

Always normalize to NFC before storing usernames, email addresses, or any text that will be compared for equality.

Locale-Aware Collation with ICU

Sorting order is a separate problem from equality. In English, ä typically sorts near a, but in Swedish, ä sorts after z. In French, accents are compared from right-to-left as a tiebreaker. These rules are formalized in the Unicode Collation Algorithm (UCA) and implemented by the ICU library (International Components for Unicode).

In Python, the pyuca package or the locale module provide UCA-based sorting. In JavaScript, Intl.Collator wraps ICU directly.

// JavaScript locale-aware sort
const words = ["ä", "z", "a"];
words.sort(new Intl.Collator("sv").compare); // Swedish: ["a", "z", "ä"]
words.sort(new Intl.Collator("de").compare); // German: ["a", "ä", "z"]

Case-Insensitive Comparison via Case Folding

Converting both strings to lowercase before comparing does not work correctly across all scripts. Unicode defines case folding (documented in CaseFolding.txt) as a locale-independent way to erase case distinctions for comparison purposes. Case folding handles edge cases like the German ß, which folds to ss, and the Greek capital letter sigma Σ, which folds to σ.

# Python case folding
"Straße".casefold() == "STRASSE".casefold()  # True

Quick Facts