Unicode, Bytes, and What Text Actually Is

Select examples to compare their grapheme, code-point, and byte counts.

What does a model actually see when you type "Hello"?

A text language model receives token IDs from a tokenizer. To understand how those IDs represent text, start with its byte encoding: in UTF-8, Hello is 0x48 0x65 0x6C 0x6C 0x6F.

For Hello, five visible letters occupy five bytes. In café, the é is one code point encoded in two bytes. Each code point in 日本語 takes three bytes. The family emoji 👨‍👩‍👧‍👦 combines seven code points and twenty-five bytes into one displayed symbol.

Sequence-to-sequence models and Bahdanau attention took input tokens as given. This arc examines how text becomes those tokens.

Prerequisites: binary and hexadecimal notation, also used in the floating-point post.

From ASCII to Unicode, the short version

ASCII represents text using seven bits per character: 128 values covering Latin letters, digits, punctuation, and control characters such as newline and tab. It does not include accented letters or non-Latin scripts.

The problem showed up the moment anyone tried to use a computer in French, or German, or Japanese. ASCII had no room for accented characters, let alone thousands of CJK ideographs. The response was code pages: region-specific extensions that reused the same byte values (128-255) to mean completely different characters depending on what country you were in. A file encoded as Windows-1252 (Western European) and opened as Windows-1251 (Cyrillic) would render as garbage. If you've ever seen a web page with é splattered everywhere that should say é, that's UTF-8 bytes being misread as Latin-1. Same bug family.

Unicode provides a shared code space across writing systems, assigning characters integers called code points. The Latin capital letter A is U+0041. The Japanese character 日 is U+65E5. The pile-of-poo emoji is U+1F4A9. These are abstract identifiers. A code point isn't a byte, and it isn't a glyph. Unicode assigns these identifiers independently of how a font draws them.

UTF-8: the encoding that won

A code point is just a number. U+0041 is the integer 65. U+65E5 is the integer 26,085. U+1F4A9 is the integer 128,169. To store these in a file or send them over a network, you need an encoding: a rule for turning code points into bytes.

Three common Unicode encodings make different storage tradeoffs:

UTF-32 uses four bytes per Unicode scalar value. In big-endian order, U+0041 becomes 00 00 00 41. You can locate the nth scalar value at byte offset 4n, but a grapheme may still span several values. ASCII-heavy text uses four times as much space as in UTF-8.

UTF-16 uses 2 bytes for code points in the Basic Multilingual Plane (excluding the surrogate range U+D800 to U+DFFF) and 4 bytes for everything above that, via a scheme called surrogate pairs. This was a reasonable bet in the early 1990s, when the Unicode Consortium believed 65,536 code points would be enough. It wasn't. UTF-16 is not byte-compatible with ASCII. The byte 0x00 shows up constantly inside normal text, which breaks C string functions and anything that treats null bytes as terminators.

UTF-8 is variable-width: 1 byte for ASCII (U+0000 to U+007F), 2 bytes for Latin extensions and most European scripts (U+0080 to U+07FF), 3 bytes for the rest of the Basic Multilingual Plane including CJK (U+0800 to U+FFFF), and 4 bytes for everything else (U+10000 to U+10FFFF). The encoding scheme uses a leading-bit pattern that makes it self-synchronizing:

The surrogate range U+D800 to U+DFFF is excluded: those values are not valid standalone Unicode scalar values and cannot be encoded in UTF-8.

The x bits carry the actual code point value. The leading bits tell you how many bytes this character occupies. Every continuation byte starts with 10, so if you land in the middle of a multi-byte sequence, you can scan backwards until you find a byte that does not start with 10, and that is the start of the current character.

And because single-byte UTF-8 characters have a leading 0, every valid ASCII file is automatically a valid UTF-8 file. That backward compatibility is what made adoption possible. You didn't have to re-encode the entire internet, and UTF-8 became the standard encoding for the web.

A quick math detour: the bit budget

For a 1-byte character, you get 7 payload bits (the leading 0 is overhead). For 2 bytes, you pay 5 bits of overhead (110 + 10) and get 11 payload bits, which covers $2^{11} = 2048$ code points. For 3 bytes, you pay 8 bits of overhead and get 16 payload bits ($2^{16} = 65536$). For 4 bytes, 11 bits of overhead and 21 payload bits ($2^{21} = 2097152$), far more than the 1,114,112 code points Unicode actually allows.

The prefix bits buy self-synchronization and ASCII compatibility. An ASCII document is byte-for-byte identical in UTF-8. BMP Chinese characters take three bytes each rather than UTF-16's two; characters above the BMP take four bytes in either encoding.

Code points vs. grapheme clusters

Consider the letter é. There are two ways to represent it in Unicode:

1. Precomposed: U+00E9, a single code point, LATIN SMALL LETTER E WITH ACUTE. In UTF-8: 0xC3 0xA9 (2 bytes).
2. Decomposed: U+0065 (LATIN SMALL LETTER E) followed by U+0301 (COMBINING ACUTE ACCENT). In UTF-8: 0x65 0xCC 0x81 (3 bytes).

Both forms represent the same letter, but one uses 1 code point and 2 bytes while the other uses 2 code points and 3 bytes. Python's len(s) counts the code points, so it returns different values despite the same displayed letter.

The Unicode Consortium defines the concept of a grapheme cluster to handle this: an approximation to a user-perceived "character." UAX #29 (Unicode Text Segmentation) specifies the segmentation rules. They have to account for combining marks, emoji modifiers, skin tone selectors, regional indicator pairs (flags), and ZWJ sequences.

So we have three different ways to measure the "length" of a string:

• Bytes: the raw storage cost. What len(s.encode('utf-8')) gives you in Python.
• Code points: the number of Unicode abstract integers. What len(s) gives you in Python (because Python 3 strings are sequences of code points).
• Grapheme clusters: what a human would count. What you need a regex library or ICU to compute.

Worked example

The family emoji and a Vietnamese letter show two different ways to combine code points.

The family emoji

The string: 👨‍👩‍👧‍👦

Each emoji person is a code point above U+10000, so they take 4 bytes each in UTF-8. Each ZWJ (U+200D) lives in the BMP and takes 3 bytes. Total: $4 \times 4 + 3 \times 3 = 25$ bytes.

In Python 3, len("👨‍👩‍👧‍👦") returns 7 because Python counts code points. In JavaScript, "👨‍👩‍👧‍👦".length returns 11 because JavaScript strings are UTF-16 and each emoji above U+FFFF is a surrogate pair (2 code units each). You get 4 surrogate pairs + 3 single code units = 11. Two languages, two different answers for the "length" of the same visual symbol, and neither is the "1" that a human would give.

Vietnamese with combining diacritics

The Vietnamese word Việt ("Vietnamese") in NFD (fully decomposed) form:

V + i + e + combining dot below + combining circumflex + t

In NFC form, the same word is V + i + ệ (precomposed e with circumflex and dot below) + t, which is 4 code points and 6 bytes. Same word, same visual rendering, different internal representation.

This is what Unicode normalization is for. NFC (Canonical Decomposition, followed by Canonical Composition) collapses combining sequences into precomposed code points where one exists. NFD (Canonical Decomposition) expands precomposed characters into their base + combining mark sequences. If you want string comparison to work reliably, you normalize first. Whether a tokenizer normalizes text is part of its configuration.

Implementation sketch: UTF-8 encoding from scratch

The encoder and decoder below expose the shifts and masks in UTF-8. For application code, use Python's .encode('utf-8') and .decode('utf-8'), which also handle validation.

def encode_codepoint(cp: int) -> bytes:
    """Encode a single Unicode code point to UTF-8 bytes."""
    if cp < 0 or cp > 0x10FFFF or 0xD800 <= cp <= 0xDFFF:
        raise ValueError(f"Invalid code point: {cp:#x}")
 
    if cp <= 0x7F:
        # 1 byte: 0xxxxxxx
        return bytes([cp])
 
    elif cp <= 0x7FF:
        # 2 bytes: 110xxxxx 10xxxxxx
        return bytes([
            0b11000000 | (cp >> 6),
            0b10000000 | (cp & 0b00111111),
        ])
 
    elif cp <= 0xFFFF:
        # 3 bytes: 1110xxxx 10xxxxxx 10xxxxxx
        return bytes([
            0b11100000 | (cp >> 12),
            0b10000000 | ((cp >> 6) & 0b00111111),
            0b10000000 | (cp & 0b00111111),
        ])
 
    else:
        # 4 bytes: 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx
        return bytes([
            0b11110000 | (cp >> 18),
            0b10000000 | ((cp >> 12) & 0b00111111),
            0b10000000 | ((cp >> 6) & 0b00111111),
            0b10000000 | (cp & 0b00111111),
        ])
 
def encode_string(s: str) -> bytes:
    """Encode a full string to UTF-8."""
    result = bytearray()
    for char in s:
        result.extend(encode_codepoint(ord(char)))
    return bytes(result)
 
# Quick test
text = "café 🤖"
our_bytes = encode_string(text)
std_bytes = text.encode("utf-8")
assert our_bytes == std_bytes, "Mismatch!"
print(f"Text:     {text}")
print(f"Bytes:    {list(our_bytes)}")
print(f"Hex:      {' '.join(f'0x{b:02x}' for b in our_bytes)}")
print(f"Length:   {len(text)} code points, {len(our_bytes)} bytes")
# Text:     café 🤖
# Bytes:    [99, 97, 102, 195, 169, 32, 240, 159, 164, 150]
# Hex:      0x63 0x61 0x66 0xc3 0xa9 0x20 0xf0 0x9f 0xa4 0x96
# Length:   6 code points, 10 bytes

The bit operations map directly to the table above. For a 2-byte encoding, we shift the code point right by 6 to get the high bits, OR them with the 110 prefix, then mask the low 6 bits and OR with the 10 prefix. Each additional byte adds one more layer of shift-and-mask.

def decode_utf8(data: bytes) -> list[int]:
    """Decode strictly: reject invalid, truncated, and overlong encodings."""
    codepoints = []
    i = 0
    while i < len(data):
        lead = data[i]
        if lead < 0x80:
            width, cp, minimum = 1, lead, 0
        elif 0xC2 <= lead <= 0xDF:
            width, cp, minimum = 2, lead & 0x1F, 0x80
        elif 0xE0 <= lead <= 0xEF:
            width, cp, minimum = 3, lead & 0x0F, 0x800
        elif 0xF0 <= lead <= 0xF4:
            width, cp, minimum = 4, lead & 0x07, 0x10000
        else:
            raise ValueError(f"Invalid leading byte at {i}")
        if i + width > len(data):
            raise ValueError(f"Truncated sequence at {i}")
        for byte in data[i + 1:i + width]:
            if byte & 0xC0 != 0x80:
                raise ValueError(f"Invalid continuation byte after {i}")
            cp = (cp << 6) | (byte & 0x3F)
        if cp < minimum or cp > 0x10FFFF or 0xD800 <= cp <= 0xDFFF:
            raise ValueError(f"Invalid scalar value or overlong encoding at {i}")
        codepoints.append(cp)
        i += width
    return codepoints
 
# Round-trip test
raw = "日本語 👨‍👩‍👧‍👦"
encoded = encode_string(raw)
decoded_cps = decode_utf8(encoded)
reconstructed = "".join(chr(cp) for cp in decoded_cps)
assert reconstructed == raw
print(f"Original:     {raw}")
print(f"Code points:  {[f'U+{cp:04X}' for cp in decoded_cps]}")
print(f"Byte count:   {len(encoded)}")
# Original:     日本語 👨‍👩‍👧‍👦
# Code points:  ['U+65E5', 'U+672C', 'U+8A9E', 'U+0020', 'U+1F468',
#                'U+200D', 'U+1F469', 'U+200D', 'U+1F467', 'U+200D',
#                'U+1F466']
# Byte count:   35

The decoder is the mirror image. You inspect the leading bits of the first byte to decide how many bytes to consume, then mask away the prefix bits and reassemble the payload. The 10xxxxxx check on continuation bytes is how you detect corruption: if a continuation byte doesn't start with 10, the stream is malformed.

Normalization: when identical text has different bytes

Two strings can render identically while containing different code-point and byte sequences:

import unicodedata
 
# The "same" character, two representations
nfc = "\u00e9"           # U+00E9: precomposed é
nfd = "e\u0301"          # U+0065 + U+0301: e + combining acute
 
print(f"NFC: {nfc!r}  bytes={nfc.encode('utf-8').hex(' ')}")
print(f"NFD: {nfd!r}  bytes={nfd.encode('utf-8').hex(' ')}")
print(f"Look the same?  '{nfc}' vs '{nfd}'")
print(f"Are equal?       {nfc == nfd}")
print()
 
# Normalize to NFC
nfd_normalized = unicodedata.normalize("NFC", nfd)
print(f"After NFC:  {nfd_normalized!r}  equal? {nfc == nfd_normalized}")
 
# NFC: 'é'  bytes=c3 a9
# NFD: 'é'  bytes=65 cc 81
# Look the same?  'é' vs 'é'
# Are equal?       False
#
# After NFC:  'é'  equal? True

Copy-pasted text and files from different sources can contain mixed normalization. A tokenizer that preserves the original bytes can encode visually identical strings differently. SentencePiece defaults to a form of NFKC normalization; byte-level BPE tokenizers such as GPT-2's preserve these distinctions. Libraries offer several normalizers, so inspect the tokenizer configuration rather than assuming one is applied. Compatibility normalization can also erase distinctions you meant to keep.

Why this matters for language models

A language model doesn't see characters, code points, or grapheme clusters. It sees tokens: integers from a fixed vocabulary. The tokenizer is the bridge between raw text and those integers, and it has to make a fundamental choice about its input representation.

The early NMT adaptation of BPE operated on Unicode characters. You start with a vocabulary of individual characters and iteratively merge the most frequent pairs. This mostly works, but it has a problem: the initial vocabulary has to include every Unicode character that might appear in the training data. For multilingual models, that initial vocabulary can be enormous, and rare characters (from scripts with low representation in the training data) consume vocabulary slots that could be used for more useful subword units.

GPT-2 made a different choice. Radford et al. ran BPE on raw UTF-8 bytes. The initial vocabulary is just 256 entries: one per possible byte value. Every piece of text in every language can be represented as a sequence of these 256 bytes. The tokenizer is completely language-agnostic at the byte level. It doesn't need special handling for Chinese, Arabic, or Devanagari, because they're all just byte sequences.

Before any merges, "Hello" is 5 bytes, 日本語 is 9, and the family emoji is 25. BPE can merge several bytes into one token, so these byte counts are not final token counts. The resulting sequence length depends on which pieces the vocabulary learned. Languages poorly represented in the tokenizer's training data often need more tokens for comparable content.

A tokenizer trades vocabulary size against sequence length and language coverage. The next post on Byte-Pair Encoding implements the merges that combine small units into longer tokens.

Misconceptions

"Unicode is an encoding." Unicode is a character set: a mapping from abstract characters to code point integers. UTF-8, UTF-16, and UTF-32 are encodings of that character set. The code point identifies the character; the encoding determines its bytes.

"Python len() tells me how many characters are in a string." It tells you how many code points are in a string. For text with combining characters, emoji sequences, or mixed normalization, the number of code points can be very different from the number of things a reader would call characters. If you need human-perceived length, you need grapheme cluster segmentation.

"Emoji are 4 bytes." A single emoji code point above U+FFFF is 4 bytes in UTF-8. But many emoji are multi-code-point sequences (flags, skin tones, ZWJ families) that can be 8, 15, or 25+ bytes. The byte count depends on which emoji.

What's next

Next: Byte-Pair Encoding from Scratch.

Additional reading (and watching)

• Yergeau, F. RFC 3629: UTF-8. Defines valid byte sequences, including the exclusions for overlong encodings and surrogate code points.

• Unicode Consortium. "UAX #29: Unicode Text Segmentation." Defines grapheme cluster boundaries, word boundaries, and sentence boundaries. unicode.org/reports/tr29/

• Kudo, T. & Richardson, J. (2018). SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing. EMNLP 2018.

• Radford, A., et al. (2019). "Language Models are Unsupervised Multitask Learners." OpenAI. Used byte-level BPE to avoid gaps in character coverage; its pre-tokenization still uses Unicode character categories.

• Petrov, A., et al. (2023). Language Model Tokenizers Introduce Unfairness Between Languages. Compares tokenization costs across languages.

• Pagnoni, A., et al. (2024). Byte Latent Transformer: Patches Scale Better Than Tokens. Meta FAIR. Introduces dynamic byte patching as an alternative to fixed tokenization.

• Google. SentencePiece documentation. Documents normalization, whitespace handling, and byte fallback.