Big5

What is Big5?

Big5 is a Traditional Chinese character encoding standard developed in Taiwan in 1984 by a consortium of five companies (hence "Big Five"). It uses a double-byte scheme where each Chinese character is represented by 2 bytes, while ASCII characters remain as single bytes. Big5 is the dominant encoding for Traditional Chinese text used in Taiwan, Hong Kong, and Macau.

Understanding Big5 matters for anyone working with legacy Chinese text data, web scraping Chinese websites, processing historical documents from Taiwan or Hong Kong, or maintaining systems that predate the widespread adoption of Unicode.

How Big5 Works

Big5 is a double-byte character encoding (DBCS):

• ASCII characters (0x00–0x7F): Encoded as single bytes, identical to ASCII.
• Chinese characters: Encoded as 2-byte sequences where the first byte (high byte) is in the range 0x81–0xFE and the second byte (low byte) is in the range 0x40–0x7E or 0xA1–0xFE.

The original Big5 standard covered 13,053 characters: - Level 1: 5,401 frequently used characters (common vocabulary) - Level 2: 7,652 less-common characters

Note the unusual gap in the low-byte range: 0x7F and 0x80 are excluded from the low-byte range to avoid conflicts with ASCII DEL (0x7F) and to allow certain software to use 0x80 as a sentinel value.

Extensions and Variants

Because the original Big5 had gaps and limitations, numerous extensions were created:

The HKSCS extension is particularly important for Cantonese text because Cantonese has many characters specific to the spoken dialect that do not appear in Mandarin-oriented Big5.

Code Examples

# Python: Big5 encoding/decoding
text = '你好世界'  # "Hello World" in Chinese

# Encode to Big5
encoded = text.encode('big5')
print(encoded)       # b'\xa7A\xa6n\xa5@\xac\xc9'
print(len(encoded))  # 8 bytes for 4 Chinese characters

# Decode from Big5
decoded = b'\xa7A\xa6n'.decode('big5')
print(decoded)  # '你好'

# Python supports multiple Big5 variants
'你'.encode('big5')        # Standard Big5
'你'.encode('big5hkscs')   # Hong Kong Supplementary Character Set
'你'.encode('cp950')       # Microsoft variant

# Check for characters not in Big5
try:
    '𠀀'.encode('big5')    # Rare CJK extension B character
except UnicodeEncodeError as e:
    print(f"Cannot encode: {e}")

Encoding Conflicts with ASCII

The most significant technical hazard of Big5 is that the second byte of a two-byte sequence can overlap with printable ASCII values (0x40–0x7E includes @, A–Z, a–z, etc.). This means a naive scanner cannot reliably find ASCII characters within a Big5 stream by scanning byte-by-byte — it must track whether it is inside a two-byte sequence.

The byte 0x5C is the backslash in ASCII. In Big5, it can appear as the second byte of a two-byte sequence. Early versions of software designed for ASCII would misinterpret this second byte as a backslash, causing path-separator bugs, SQL injection vulnerabilities, and other security issues.

Quick Facts

Common Pitfalls

Confusing Big5 with GB2312/GBK. Big5 encodes Traditional Chinese characters used in Taiwan; GB2312/GBK encodes Simplified Chinese used in mainland China. These are entirely different encodings with non-overlapping byte ranges for Chinese characters. A Big5 file decoded as GBK produces complete garbage.

The backslash vulnerability. As noted above, Big5 second bytes can be 0x5C (backslash). In security-sensitive applications, never scan Big5 text byte-by-byte looking for ASCII delimiters without full two-byte sequence awareness. This class of vulnerability has been exploited in SQL injection attacks against older web applications.

Incomplete Big5 support in libraries. Not all libraries support Big5-HKSCS or Big5+. Characters unique to Hong Kong Cantonese may fail to encode in software that only implements the original Big5 standard.