Tokenization: How it Works, Impacts Model Behavior, and Affects Cost

From Text to Tokens: The Language of LLMs

Large Language Models do not process raw text—they don't see words, letters, or sentences in the way we do. Instead, they operate on tokens. A token is a chunk of text that the model has been trained to see as a single unit. This could be a whole word, a common part of a word (a subword), a single character, or a piece of punctuation.

The process of converting a string of text into a sequence of tokens is called tokenization. These tokens are then mapped to integer IDs, and it is these numbers that are fed into the model for processing.

To see this process in action, let's watch a short, practical demonstration.

How Tokenizers Are Made: The Subword Compromise

So, how does a tokenizer decide how to split up text? There are a few ways to do it, each with trade-offs.

• Word-level tokenization: Every unique word in the training data becomes a token. This leads to an enormous vocabulary and struggles with words it has never seen before (out-of-vocabulary words), as well as variations like "run," "running," and "ran."
• Character-level tokenization: Every character is a token. This results in a very small vocabulary but creates extremely long token sequences. Since the computational cost of the transformer's attention mechanism scales quadratically with sequence length, this is very inefficient.
• Subword tokenization: This is the happy medium and the standard for modern LLMs. It breaks down rare words into smaller, more frequent subword units while keeping common words as single tokens. For example, "tokenization" might become ["token", "ization"].

The most common algorithm for creating a subword tokenizer is Byte Pair Encoding (BPE). Given your background in CS, you can think of it as a greedy data compression algorithm. It starts with a vocabulary of all individual characters and iteratively merges the most frequently occurring adjacent pair of tokens into a new, single token, adding it to the vocabulary. This process repeats until the vocabulary reaches a desired size (e.g., 50,000 or 200,000 tokens).

This article from Redis provides a concise overview of these three methods.

Why Tokenization Matters: Cost, Performance, and Behavior

This seemingly minor preprocessing step has a massive impact on everything an LLM does. Understanding these effects is critical for both building with and teaching about AI.

Let's break down the key impacts:

1. Direct Impact on Cost and Speed

API providers bill you per token, for both your input (prompt) and the model's output (completion). As the diagram above shows, an inefficient tokenizer can drastically increase the number of tokens for the same text, directly increasing your costs.

Furthermore, longer token sequences require more computation, increasing latency. This is because the self-attention mechanism, which we discussed in a previous lesson, has a computational complexity that scales with the square of the sequence length ($O(N^2)$). Fewer tokens mean faster responses.

2. The "Token Budget" and Context Window

A model's context window—the maximum amount of information it can consider at once—is measured in tokens. A model with a 16k context window can process 16,384 tokens, not 16,384 words. Efficient tokenization allows you to "stretch" this budget, fitting more meaningful information into the same window.

This article introduces the helpful mental model of a "token budget" that is spent on the system prompt, user input, retrieved context (in RAG), and the model's output.

3. Quirks in Model Behavior

Many of an LLM's strange "blind spots" are rooted in tokenization.

• Spelling and Arithmetic: Since a word like "misspelled" might be a single token, the model doesn't "see" its individual letters. This is why LLMs can struggle with tasks like reversing a word or counting its characters. Similarly, the number 12345 might be tokenized as ["12", "345"], making arithmetic unreliable.
• Sensitivity to Spacing: As the Mark Hennings video (LINK) showed, a simple trailing space can change which tokens are generated, sometimes drastically altering the model's output because " cat" and "cat" can be two different tokens with different learned associations.
• Unusual Words & Code: A made-up word, a rare name, or code in a less common programming language will be broken down into many small, sometimes single-character, tokens. This makes it more "expensive" for the model to process and can sometimes lead to strange outputs.

4. The Multilingual Divide

Perhaps the most dramatic effect of tokenization is seen across different languages. Early tokenizers, trained primarily on English text, were notoriously inefficient for other languages. A single character in Tamil or Gujarati might be split into three or four tokens.

However, this has improved dramatically. Modern tokenizers have much larger vocabularies and are trained on more diverse multilingual data. This has a direct and massive impact on cost and performance for non-English users.

This next article provides a fantastic deep-dive, including a powerful example from your first language, Gujarati.

The 86% token reduction for Gujarati text is a perfect, real-world example to use in your own course to make the concept of tokenization efficiency tangible and impactful for a diverse audience.

Your Project: "The Token Economist" Challenge

Let's design a game-based activity that makes the economic impact of tokenization real for your students.

Challenge: "The Token Economist"
Objective: To achieve a specific goal using an LLM while staying under a strict "token budget."

Your Task:
Blueprint a competitive classroom challenge with the following structure:

1. The Setup:

   • Divide learners into teams.
   • Give each team a starting budget of, for example, 1,000 tokens.
   • Introduce a task, such as: "Get the AI to explain the concept of photosynthesis in exactly three sentences, suitable for a 10-year-old."

2. The Gameplay Loop (Multiple Rounds):

   • Round 1: Teams write their first prompt. You (the instructor) run these prompts through a tokenizer tool (like OpenAI's Tokenizer) and show them the input token cost.
   • You then run the prompt through an actual LLM and show the output. You calculate the total cost (input + output tokens) and deduct it from their budget.
   • Refinement: Teams analyze their results. Was the output correct? Was the cost high? They then refine their prompt for the next round, trying to be more concise or clever to reduce token usage while improving the output. For example, using "Explain photosynthesis for a 10-year-old in 3 sentences" is more token-efficient than "Could you please provide me with an explanation of the biological process known as photosynthesis, but can you make it simple enough for a fifth-grader to understand and keep it to just three sentences?"

3. Advanced Twist (The Multilingual Round):

   • Introduce a new task: "Translate the phrase 'Where is the nearest library?' into Spanish."
   • First, use an older tokenizer model (e.g., GPT-3's r50k_base if the tool allows) to show a higher token count.
   • Then, switch to a modern tokenizer (o200k_base) to show the dramatic reduction in tokens for the same Spanish phrase. This viscerally demonstrates the efficiency gains discussed in the article.

Your Deliverable:
Write a brief outline for this "Token Economist" challenge. Describe the narrative wrapper (e.g., "You are a team of AI engineers at a startup with a limited budget..."), the rules for winning (e.g., the first team to achieve the goal with tokens left, or the team with the most tokens remaining after completing the task), and how you would debrief the activity to connect it back to real-world API costs and prompt engineering.

Conclusion

Today, we've pulled back the curtain on tokenization, the invisible yet crucial process that governs how LLMs interact with text.

Key Takeaways:

• LLMs operate on tokens (subword units mapped to integers), not raw text.
• Subword tokenization (like BPE) is the standard, balancing vocabulary size against sequence length.
• Tokenization has a direct and significant impact on cost, speed, and the effective context window of a model.
• Model quirks, like difficulty with spelling or sensitivity to spacing, are often artifacts of tokenization.
• Modern tokenizers are vastly more efficient for non-English languages, reducing costs and improving performance for global users.

We have now examined the model's architecture, its training process, and the fundamental data units it consumes. In our next lesson, we will broaden our perspective to discuss common LLM limitations, including hallucinations, context window constraints, and the biases inherited from training data. Understanding tokens provides the foundation for grasping some of these limitations, which we will now explore in greater depth.