エンコーディング / デコーディング

What Is Encoding and Decoding?

Encoding converts a Unicode string (sequence of abstract code points) into bytes. Decoding converts bytes back into a Unicode string. The encoding specifies the mapping: how each code point is represented as a sequence of bytes.

The distinction matters because computers transmit and store bytes, not abstract characters. The string "Hello, 世界" must be encoded before saving to a file, sending over a network, or inserting into a database. When reading it back, you must use the same encoding to decode correctly.

The Most Important Encodings

Python Encoding and Decoding

# str → bytes (encoding)
text = "Hello, 世界"
utf8_bytes  = text.encode("utf-8")     # b"Hello, \xe4\xb8\x96\xe7\x95\x8c"
utf16_bytes = text.encode("utf-16")    # includes BOM
utf32_bytes = text.encode("utf-32")

# bytes → str (decoding)
utf8_bytes.decode("utf-8")             # "Hello, 世界"
utf8_bytes.decode("latin-1")           # garbled — wrong codec!

# Error handling
bad_bytes = b"\xff\xfe"
bad_bytes.decode("utf-8", errors="strict")   # UnicodeDecodeError
bad_bytes.decode("utf-8", errors="replace")  # "\\ufffd\\ufffd" (replacement chars)
bad_bytes.decode("utf-8", errors="ignore")   # "" (silently drops bad bytes)
bad_bytes.decode("utf-8", errors="backslashreplace")  # "\\xff\\xfe"

# File I/O
with open("file.txt", "w", encoding="utf-8") as f:
    f.write("Hello, 世界")

with open("file.txt", "r", encoding="utf-8") as f:
    text = f.read()

# Detect encoding (unreliable but useful)
import chardet
result = chardet.detect(unknown_bytes)
encoding = result["encoding"]  # e.g., "UTF-8", "ISO-8859-1"

JavaScript Encoding and Decoding

// TextEncoder / TextDecoder (modern, available in browsers and Node.js)
const encoder = new TextEncoder();  // Always UTF-8
const bytes = encoder.encode("Hello, 世界");
// Uint8Array [72, 101, 108, 108, 111, 44, 32, 228, 184, 150, ...]

const decoder = new TextDecoder("utf-8");
decoder.decode(bytes);  // "Hello, 世界"

const latin1decoder = new TextDecoder("iso-8859-1");
latin1decoder.decode(bytes);  // garbled — wrong codec

// Buffer in Node.js
Buffer.from("Hello, 世界", "utf8");
Buffer.from("Hello, 世界", "utf8").toString("utf8");

The Encoding Mismatch Problem

The single most common Unicode bug is encoding/decoding with the wrong codec:

# File saved as UTF-8, opened as Latin-1
with open("utf8_file.txt", "r", encoding="latin-1") as f:
    content = f.read()
# "Hello, 世界" — mojibake: UTF-8 bytes misinterpreted as Latin-1

UTF-8 bytes for CJK characters start with 0xE4–0xEF. In Latin-1, these are accented characters, producing the garbled 世界 output.

Encoding Best Practices

1. Always use UTF-8 for new files, APIs, and databases.
2. Declare encoding explicitly — never rely on system defaults.
3. Handle errors — use errors="replace" for resilience, errors="strict" for correctness.
4. Validate input: Decode user-provided bytes early; work with strings internally; encode at output boundaries.

Quick Facts