U+ UnicodeFYI

The Developer's Unicode Handbook · Chapter 2

The Encoding Minefield

File I/O, HTTP headers, database collation, BOM detection — encoding issues lurk everywhere. This chapter provides a systematic approach to choosing and using encodings correctly across your entire stack.

~4,500 words · ~18 min read · · Updated

Unicode solves the problem of representing all human writing in a single character set. Encoding solves the problem of turning those characters into bytes. The gap between these two layers — the encoding layer — is where most real-world Unicode bugs live. This chapter is a field guide to the mines you'll step on and how to defuse them.

The UTF-8 BOM Trap

The Byte Order Mark (BOM) is a zero-width no-break space character (U+FEFF) placed at the start of a file to signal byte order and encoding. UTF-16 actually needs it to distinguish big-endian from little-endian. UTF-8 does not — byte order is irrelevant for single-byte code units. But some programs (particularly Microsoft products) emit UTF-8 BOMs anyway. The result: a mysterious \\xEF\\xBB\\xBF at the start of every file that silently corrupts CSV parsing, JSON decoding, and HTTP responses.

# Reading a UTF-8 BOM file the wrong way
with open("file.csv", "r", encoding="utf-8") as f:
    first_line = f.readline()
    # first_line might start with '\\ufeff' or '\\xef\\xbb\\xbf'
    # CSV parser chokes; JSON parser raises an error

# The right way: use utf-8-sig which strips BOM automatically
with open("file.csv", "r", encoding="utf-8-sig") as f:
    first_line = f.readline()
    # BOM is stripped, content is clean

# Detecting BOM in bytes
with open("file.csv", "rb") as f:
    raw = f.read(4)
    has_bom = raw.startswith(b"\\xef\\xbb\\xbf")  # UTF-8 BOM
    has_utf16_le_bom = raw.startswith(b"\\xff\\xfe")
    has_utf16_be_bom = raw.startswith(b"\\xfe\\xff")

When writing files that will be consumed by Windows tools (Excel, Notepad), emitting a UTF-8 BOM can be helpful. For web APIs and inter-service communication, never emit a BOM.

Double-Encoding: The Classic Mojibake Source

Double-encoding happens when already-encoded text gets encoded again. The tell-tale signs: é for é, “ for ", £ for £. This is called mojibake — Japanese for "character transformation."

# How double-encoding happens
original = "café"
encoded_once = original.encode("utf-8")   # b'caf\\xc3\\xa9'
# Somewhere, this bytes object gets decoded as latin-1 instead of UTF-8:
mangled = encoded_once.decode("latin-1")  # 'café'
# Now mangled gets encoded as UTF-8 and stored in a database:
double_encoded = mangled.encode("utf-8")  # b'caf\\xc3\\x83\\xc2\\xa9'

# To reverse it: decode as UTF-8, encode as latin-1, decode as UTF-8
def fix_double_encoding(s: str) -> str:
    try:
        return s.encode("latin-1").decode("utf-8")
    except (UnicodeEncodeError, UnicodeDecodeError):
        return s  # Not double-encoded, return as-is

print(fix_double_encoding("café"))  # café

The root cause is almost always a pipeline where bytes are treated as latin-1 text instead of UTF-8. Classic culprits: MySQL with latin1 charset, PHP mb_string misconfiguration, legacy Perl scripts, Windows-1252 files read without specifying encoding.

Encoding Detection with chardet

When you receive bytes without a declared encoding, you have to guess. The chardet library uses statistical analysis to identify the most likely encoding:

import chardet

# Reading an unknown file
with open("mystery.txt", "rb") as f:
    raw_bytes = f.read()

result = chardet.detect(raw_bytes)
print(result)
# {'encoding': 'Windows-1252', 'confidence': 0.73, 'language': 'English'}

# Safe decoding with fallback
detected = chardet.detect(raw_bytes)
encoding = detected.get("encoding") or "utf-8"
confidence = detected.get("confidence", 0)

if confidence > 0.8:
    text = raw_bytes.decode(encoding, errors="replace")
else:
    # Low confidence — try UTF-8, then fall back to latin-1
    try:
        text = raw_bytes.decode("utf-8")
    except UnicodeDecodeError:
        text = raw_bytes.decode("latin-1")  # latin-1 never raises, decodes everything

For HTTP responses, always check the Content-Type header before guessing:

import requests

response = requests.get("https://example.com/page")
# requests uses chardet internally and sets .encoding
# but HTTP charset declaration takes priority
print(response.encoding)        # 'utf-8' (from Content-Type header)
print(response.apparent_encoding)  # chardet's guess

# Use apparent_encoding only when header is absent or wrong
text = response.content.decode(response.encoding or response.apparent_encoding)

Python's bytes/str Split

Python 3 made a strict separation between bytes (raw binary data) and str (Unicode text). This prevents the implicit coercion that caused Python 2's notorious encoding bugs, but it requires you to be explicit at every I/O boundary.

# Every I/O boundary requires explicit encoding/decoding
# File I/O
with open("file.txt", "r", encoding="utf-8") as f:  # text mode: str
    text: str = f.read()

with open("file.txt", "rb") as f:  # binary mode: bytes
    data: bytes = f.read()
    text = data.decode("utf-8")

# Network I/O
import socket
sock = socket.socket()
sock.connect(("example.com", 80))
sock.send(b"GET / HTTP/1.0\\r\\n\\r\\n")  # bytes only
response_bytes: bytes = sock.recv(4096)
response_text: str = response_bytes.decode("utf-8", errors="replace")

# subprocess
import subprocess
result = subprocess.run(["ls", "-la"], capture_output=True)
output_bytes: bytes = result.stdout
output_text: str = output_bytes.decode("utf-8")  # or use text=True parameter

# subprocess with text=True handles encoding automatically
result = subprocess.run(["ls", "-la"], capture_output=True, text=True, encoding="utf-8")
output_text: str = result.stdout  # already decoded

A common anti-pattern is using errors="ignore" to silence encoding errors. This silently drops data and hides bugs. Use errors="replace" for display purposes or errors="backslashreplace" for debugging — but fix the root cause.

Node.js: Buffer vs String

Node.js has the same bytes/string duality as Python, but the APIs are less disciplined about enforcing it:

const fs = require("fs");

// Reading as Buffer (bytes)
const buffer = fs.readFileSync("file.txt");
console.log(buffer instanceof Buffer);  // true
console.log(typeof buffer);             // 'object'

// Reading as string (decodes as UTF-8 by default)
const text = fs.readFileSync("file.txt", "utf8");
console.log(typeof text);  // 'string'

// Converting between Buffer and string
const buf = Buffer.from("café", "utf8");
console.log(buf);                       // <Buffer 63 61 66 c3 a9>
console.log(buf.toString("utf8"));      // 'café'
console.log(buf.toString("latin1"));    // 'café' ← wrong encoding!

// HTTP: always set charset
const http = require("http");
const server = http.createServer((req, res) => {
    res.setHeader("Content-Type", "text/html; charset=utf-8");
    res.end("café");  // string, automatically encoded to UTF-8
});

Database Charset: The Hidden Time Bomb

Database encoding bugs are particularly nasty because they corrupt data silently and the corruption isn't visible until you query the data back. MySQL's infamous utf8 charset only supports 3-byte UTF-8 (no characters above U+FFFF, including emoji). Use utf8mb4 for real UTF-8 support.

-- MySQL: Always use utf8mb4
CREATE TABLE users (
    name VARCHAR(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci
);

-- Or set at database/table level:
ALTER DATABASE mydb CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;

-- Check current charset:
SHOW CREATE TABLE users\\G
SHOW VARIABLES LIKE 'character_set%';

# Django: ensure connection charset
DATABASES = {
    "default": {
        "ENGINE": "django.db.backends.mysql",
        "OPTIONS": {
            "charset": "utf8mb4",
        },
    }
}

# PostgreSQL: UTF-8 by default, but verify
# psql: SHOW server_encoding;
# Python psycopg2 handles encoding automatically

File I/O Encoding Traps

Python's open() uses the system's default encoding when no encoding argument is provided. On Windows, this is often cp1252. On Linux/Mac it's typically utf-8. Code that works on your Mac will break on a Windows server.

import locale

# Never rely on the default — always specify explicitly
# Bad: platform-dependent
with open("file.txt", "r") as f:
    text = f.read()

# Good: explicit
with open("file.txt", "r", encoding="utf-8") as f:
    text = f.read()

# Check what the default would be:
print(locale.getpreferredencoding())  # 'UTF-8' on Mac, 'cp1252' on some Windows

# For portable code, set PYTHONIOENCODING in your environment:
# export PYTHONIOENCODING=utf-8
# or in Python 3.7+: use -X utf8 flag

The pathlib module has the same issue:

from pathlib import Path

# Also requires explicit encoding
text = Path("file.txt").read_text(encoding="utf-8")
Path("output.txt").write_text(content, encoding="utf-8")

Quick Encoding Diagnosis Checklist

When you encounter garbled text, walk through this checklist:

  1. What are the actual bytes? Use repr() or a hex editor to see raw bytes.
  2. What encoding does the sender claim? Check HTTP headers, file headers, database charset.
  3. Is it double-encoded? Look for à followed by a character — classic UTF-8-as-latin-1.
  4. Is there a BOM? Check for \\xef\\xbb\\xbf (UTF-8 BOM) or \\xff\\xfe (UTF-16 LE BOM).
  5. Is it Windows-1252 misread as UTF-8? The range \\x80–\\x9f is undefined in latin-1 but used in Windows-1252 for smart quotes, em-dashes, etc.
  6. Can you reproduce it? Find the exact bytes that produce the garbled output, then work backward.

Encoding bugs compound over time. A single latin-1-in-utf-8 conversion doesn't just corrupt one database column — it corrupts every downstream system that processes that column. Fix encoding issues at the source, not with errors="ignore" patches throughout your codebase.

← Previous

String Length Is a Lie

Next →

Comparison and Sorting