Сегментация текста

Boundaries in Unicode Text

Text is not a flat sequence of code points — it has structure. Users think in terms of characters, words, and sentences. But Unicode code points do not map cleanly to these concepts. A single visible character (a grapheme cluster) can span multiple code points. A "word" means different things in English, Japanese, and Arabic. A sentence boundary after a period is ambiguous when periods also appear in abbreviations and numbers.

Unicode Text Segmentation (UAX #29) defines algorithms for finding grapheme cluster boundaries, word boundaries, and sentence boundaries. These algorithms are the foundation for correct cursor movement, text selection, word counting, and spell checking in any Unicode-aware application.

The Grapheme Cluster Problem

Python's len() function counts code points, not user-perceived characters:

# Emoji with ZWJ sequence: 1 visible character, 7 code points
family = "\U0001F468\u200D\U0001F469\u200D\U0001F467\u200D\U0001F466"
print(len(family))        # 7 (code points)
# User sees: 👨‍👩‍👧‍👦 (one family emoji)

# Combining characters
cafe = "cafe\u0301"       # e + combining acute = é
print(len(cafe))           # 5 (code points)
print(len("café"))         # 4 (precomposed NFC)
# Both render as "café" — 4 user-perceived characters

# Flag emoji: 2 regional indicator symbols = 1 flag
flag = "\U0001F1FA\U0001F1F8"  # 🇺🇸
print(len(flag))           # 2 (code points)
# User sees: 🇺🇸 (1 flag)

A grapheme cluster is the minimal unit a user thinks of as a single character. UAX #29 defines grapheme cluster boundary rules that handle: - Base + combining marks - Hangul syllable sequences (jamo combining rules) - Regional indicator pairs (flags) - Zero Width Joiner (ZWJ) sequences (family/profession emoji) - Extend characters (tags, emoji modifiers)

Using UAX #29 in Python

The grapheme package provides UAX #29-compliant grapheme cluster segmentation:

# pip install grapheme
import grapheme

family = "\U0001F468\u200D\U0001F469\u200D\U0001F467\u200D\U0001F466"
print(grapheme.length(family))           # 1
print(list(grapheme.graphemes(family)))  # ['👨‍👩‍👧‍👦']

text = "Hello, 世界! 🌍"
print(grapheme.length(text))             # 11 (user-perceived chars)

# Safe string slicing (by grapheme, not code point)
print(grapheme.slice(text, 0, 5))        # 'Hello'

For industrial-strength segmentation including word and sentence boundaries, use ICU via PyICU:

from icu import BreakIterator, Locale

text = "Don't stop. Dr. Smith arrived at 3.14 PM."
bi = BreakIterator.createSentenceInstance(Locale("en_US"))
bi.setText(text)
start = 0
for end in bi:
    print(repr(text[start:end]))
    start = end
# "Don't stop. " | "Dr. Smith arrived at 3.14 PM."

Quick Facts