Tokenization: The Core of LLM Workload

Recommended resources

The freely available resources this lesson will direct you to.

Hello and welcome to the next module in your preparation for a software engineering role at Anthropic.

In our previous module, we concluded by building a robust, fault-tolerant web crawler capable of persisting its state. We transformed a simple script into a system that could recover from crashes, a crucial feature for any production data-processing pipeline. This work focused on the acquisition of data at scale.

Now, we pivot to the systems that consume and serve intelligence derived from such data. This module, "System Design: LLM Inference Service," takes us into the very core of what a company like Anthropic builds. We'll start with the most fundamental concept of all.

This lesson addresses the learning outcome: Explain how tokenization (e.g., BPE) works and why token count is the fundamental unit of LLM workload. Understanding this is non-negotiable for designing, building, or even discussing any system involving Large Language Models. It is the bedrock upon which all performance, cost, and architectural decisions are built.

1. What are Tokens? The Currency of LLMs

At a fundamental level, LLMs do not understand words or characters. They are mathematical functions that operate on numbers. Tokenization is the process of converting a piece of text into a sequence of integers, called tokens. Decoding is the reverse process of turning a sequence of tokens back into text.

This might seem like a simple encoding step, but the specific way text is converted to tokens has profound implications. Let's start with a short video that highlights a key mystery: why does the same text result in a different number of tokens on different platforms?

Most devs don't understand how LLM tokens work

The video 'Most devs don't understand how LLM tokens work' by Matt Pocock provides an excellent introduction to what tokens are and why they are central to using LLMs. Pay attention to how he frames tokens as the 'currency' of LLMs.

Watch the segment from 00:00:35 to 00:04:21. Focus on: How input and output tokens are the basis for billing. The demonstration that 'Hello World' yields different token counts for Anthropic and Google models. The core idea that an LLM has a vocabulary of tokens (words, subwords, characters), and the model 'thinks' in terms of these token IDs, not text.

As the video demonstrates, an LLM's workload is not measured in characters or words, but in the number of tokens it processes. This simple fact is the key to understanding LLM system performance and economics.

2. Why Token Count is the Fundamental Unit of Workload

Your experience building Rudi at AWS has given you deep insights into measuring and managing system load, using metrics like requests-per-second (TPS) and enforcing Quality of Service (QoS). In the world of LLMs, the primary unit for these metrics is the token.

Let's explore the three main reasons why.

Token optimization: The backbone of effective prompt engineering

The article 'Token optimization: The backbone of effective prompt engineering' from IBM Developer clearly articulates why managing token count is critical. It directly links tokens to the practical concerns of any software engineer building on LLMs.

Please read the sections 'Understanding tokens in prompt engineering', 'Token optimization: a key driving factor for prompt engineering', and 'Why token optimization matters'. As you read, focus on how tokens define the boundaries and costs of the system.

Based on the article, we can summarize the impact of tokens on three critical system design constraints:

A. Cost
API providers, including Anthropic, price their services based on the number of tokens processed, often with different rates for input (prompt) tokens and output (completion) tokens. For any application, cost = (input_tokens * price_per_input_token) + (output_tokens * price_per_output_token). Minimizing token count is therefore a direct and primary lever for managing operational expenditure.

B. Performance & Throughput
The computational work done by a GPU during inference is directly proportional to the number of tokens being processed.

Latency: The time to generate a response (especially time-to-first-token) depends on the length of the input prompt in tokens.
Throughput: The overall capacity of an inference server is measured in tokens per second. This is the key metric for capacity planning and autoscaling, much like TPS is for a traditional microservice.

This chart clearly shows how hardware performance is benchmarked in tokens/second.

LLaMA2 Throughput Comparison Across Different GPUs — This chart illustrates how system throughput for various GPU models (L40, A100, MI210) is measured in tokens per second when running LLM inference. This is the standard performance metric for LLM serving systems.

C. Context Window Limits
Every model has a maximum context window, which is the total number of tokens (input + output) it can handle in a single request. For example, some models have context windows of 4k, 32k, or even 200k tokens. This is the model's effective "working memory." Any input that exceeds this limit will be rejected or truncated. As a system designer, you must operate within this hard constraint.

3. How Tokenization Works: Byte-Pair Encoding (BPE)

Now that we know why tokens are so important, let's look at how they are created. The most common algorithm used by modern LLMs like those from OpenAI and Anthropic is Byte-Pair Encoding (BPE).

At its heart, BPE is a data compression algorithm. It starts with a base vocabulary and iteratively creates new tokens by merging the most frequently occurring adjacent pairs.

TOKENIZATION: How AI models turn text into numbers | Byte-Pair Encoding

For a clear, step-by-step explanation of the BPE algorithm, watch this video from Annie Sexton. It builds the concept from the ground up.

Watch the segment from 00:00:00 to 00:05:36. Focus on: The limitations of simple word-based or character-based tokenization. The core BPE loop: count pairs, merge the most frequent pair to create a new token, and repeat. The crucial shift from character-level to byte-level BPE. This allows the tokenizer to handle any text, including all languages, emojis, and even typos, with a base vocabulary of just 256 tokens (one for each byte value).

As you just saw, byte-level BPE is elegant and powerful. It guarantees that any string of text can be tokenized without ever encountering an "unknown" token, because at the lowest level, everything is just a sequence of bytes.

A Deeper Look: The BPE Algorithm and Pre-tokenization

Let's ground this with a more formal description and a key implementation detail that is crucial for creating a good tokenizer.

Building a Fast BPE Tokenizer from Scratch

The article 'Building a Fast BPE Tokenizer from Scratch' provides a technical deep-dive into the algorithm. We'll focus on the sections that describe the core algorithm and a critical pre-processing step.

Please read the sections 'Algorithm & Naive Implementation' and the part of the 'Naive Implementation' that explains the GPT-2 pretokenization regex. Don't worry about the Python code itself, but focus on: The four steps: Pretokenization, Counting pairs, Merging, and Iteration. The explanation of byte-level BPE. The role of the pre-tokenization regex: why it's necessary to split text into chunks before applying BPE to prevent nonsensical merges across word boundaries (e.g., merging the 'e' from 'the' and the 't' from 'time').

The concept of pre-tokenization is an important practical detail. Without a smart initial split, the BPE algorithm might create tokens that don't align with meaningful linguistic units, such as merging the end of one word with the beginning of another. The regex from GPT-2 is a sophisticated pattern designed to segment text along logical boundaries (words, numbers, punctuation) before the merging process begins. This ensures the resulting vocabulary is more semantically meaningful.

4. Practical Trade-offs in Tokenization

Building a tokenizer is not a one-size-fits-all process. The choices made during its training create a set of trade-offs that impact the model's performance and efficiency. As a system engineer, understanding these trade-offs helps you reason about why a model might behave differently on various inputs.

Tokenization Trade-offs in LLM Development — This diagram illustrates the key trade-offs in tokenization. For instance, increasing the vocabulary size leads to better compression (fewer tokens per document) but a larger model embedding layer. Training on domain-specific text improves compression on that domain but may reduce it elsewhere.

The core trade-off, as highlighted in the diagram and the Matt Pocock video, is between vocabulary size and compression ratio:

Small Vocabulary: A smaller vocabulary (e.g., 32,000 tokens) results in a smaller model embedding matrix, saving memory. However, it means text is often broken into smaller pieces, leading to longer token sequences. This increases the computational load during inference.
Large Vocabulary: A larger vocabulary (e.g., 200,000 tokens) can represent common phrases and words as single tokens, resulting in shorter sequences (better compression). This reduces the computational load but increases the model's memory footprint.

This also explains why text in less common languages or with unusual words (like "frabjous") gets broken into many more tokens: those sequences of bytes were not frequent enough in the training data to be merged into a single token.

Conclusion

In this lesson, we've unpacked the concept of tokenization, the essential first step in any LLM workflow. You now have the foundation to reason about LLM performance, cost, and system architecture.

Key Takeaways:

LLMs operate on tokens (integers), not raw text. Tokenization is the process of converting text to tokens.
Token count is the fundamental unit of workload for an LLM. It dictates cost (pricing is per-token), performance (throughput is measured in tokens/sec), and capability (context windows are defined in tokens).
Byte-Pair Encoding (BPE) is the standard algorithm used to train tokenizers. It's a compression algorithm that starts with individual bytes and iteratively merges the most frequent adjacent pairs to build a vocabulary.
The design of a tokenizer involves trade-offs, primarily between vocabulary size (affecting model memory) and compression ratio (affecting computational load).

Next Lesson Preview

Now that we understand how text is converted into tokens for the model to process, our next step is to look inside the model itself. How does an LLM generate a response, token by token? And what is the single most important optimization for making this process fast and memory-efficient?

In the next lesson, we will explore the answer: the KV Cache. You will learn its role in accelerating autoregressive inference and the significant memory implications it has for designing a high-throughput LLM serving system.