Welcome to your second lesson! In our previous session, we established that Large Language Models (LLMs) are fundamentally next-token predictors. Today, we'll dive into the practical and crucial implications of that design. We'll move from the abstract concept of a "token" to the concrete engineering challenges it creates.
This lesson focuses on tokens, context windows, and the limits they impose. Understanding these concepts is not just theoretical; it's the foundation for writing efficient, cost-effective, and reliable prompts for the AI agents we'll be building. By the end of this hour, you will be able to describe how text is tokenized, what a context window is, and how its finite size dictates our approach to designing prompts and handling responses.
What Exactly Is a Token?
In the last lesson, we defined a token as the basic unit of text an LLM processes. But what does that mean in practice? How does "Hello world" become a set of tokens, and why should you, as a developer, care about the details?
The answers relate directly to performance and, critically, cost. As you'll be building production-grade systems, understanding the "currency" of LLMs is paramount. This video by Matt Pocock, a developer-focused educator, provides an excellent explanation.
Most devs don't understand how LLM tokens work
This video, "Most devs don't understand how LLM tokens work," clarifies why the tokenization process is so important from an engineering perspective.
Watch the first two sections, up to the 4:49 mark. Pay close attention to these key points: Tokens as the "currency" of LLMs, with separate billing for input and output. How the same input can result in different token counts (and costs) for different models. The process of encoding text into numerical tokens and decoding them back.
As the video demonstrated, the process of converting text into tokens is called tokenization. LLMs don't see words; they see numerical IDs that correspond to tokens in their vocabulary. Modern LLMs use subword tokenization, which is a clever compromise between breaking text into individual characters (inefficient) and whole words (inflexible).
Subword tokenizers, like Byte-Pair Encoding (BPE), are trained to find the most common sequences of characters in a large text corpus. This allows them to create a vocabulary of frequently used whole words, common subwords (like ing, pre, tion), and individual characters.
This image provides a clear visual of the process for a query you might write as a .NET developer.
This subword approach is why a single "word" doesn't necessarily equal one token. Common English words might be a single token, but less common words, code, or complex names are broken into multiple pieces. This also explains why different models produce different token counts: they are trained with different vocabularies and tokenization algorithms.
For a more structured overview of these tokenization strategies, the following article is a helpful reference.
What Is an LLM Token: Beginner-Friendly Guide for ...
This guide from The New Stack provides a solid foundation on tokenization.
Please read the following sections: Understanding Large Language Model Tokens and How Do LLMs Use Tokens? Word-Based, Character-Based, and Subword Tokenization, which formally defines the methods we've discussed. Popular Tokenization Algorithms and Their Differences, which names the specific algorithms like BPE and WordPiece that you'll encounter.
The Context Window: An LLM's Working Memory
Now that we have a solid grasp of tokens, let's discuss where they live during an LLM's operation: the context window. You can think of the context window as the model's short-term or working memory. It's the total amount of tokenized information the model can "see" at one time to process your request and generate a response.
This memory is not infinite. Every model has a maximum context window size, measured in tokens. This is one of the most important specifications of an LLM. For a great conceptual overview, watch this short video from IBM Technology.
What is a Context Window? Unlocking LLM Secrets
This video, "What is a Context Window?", uses a simple analogy to explain the concept and its limitations.
Watch the first three sections, from the beginning to the 8:52 mark. The video clearly explains: The context window as working memory. How conversations can exceed the window, causing the model to "forget". The various components that consume space in the context window.
As the video explained, the context window isn't just for your user prompt. It's a shared space that must hold everything the model needs for a given turn in a conversation. This includes:
- System Prompt: The initial instructions that define the agent's persona and task.
- Conversation History: Previous user messages and AI responses.
- RAG Chunks: External documents or data retrieved to answer a question (we'll cover this in Module 6).
- User Input: The current message from the user.
- Generated Response: The output the model is creating.
This is a zero-sum game. The total number of tokens for all these components cannot exceed the model's hard limit. A larger input means less available space for the model to generate a long and detailed response.
The following chart visualizes how you might budget tokens within a context window for a typical agentic application.
Designing for Token Limits
Understanding these constraints is the first step toward effective prompt engineering. As a developer, you must actively manage the context window to ensure your application is performant, cost-effective, and correct.
The most immediate consequence is the trade-off between input length and output length. If a model has a 128k token context and you send it a 124k token prompt (e.g., a very long document), you are only leaving a 4k token budget for the response. If the model hits that limit, its output will be abruptly cut off.
Furthermore, a larger context window isn't a silver bullet. Research has shown that models don't pay equal attention to all parts of the context. This leads to a phenomenon known as the "Lost in the Middle" problem.
The following article, written for .NET developers using Semantic Kernel, explains this and other practical context management issues very well.
How to Efficiently Manage AI Agent Context in .NET
This article by Beniamin Lenarcik connects the theoretical limits of context windows to the practical challenges you'll face when building agents in .NET.
Please read these sections: Context Window – Limitation of LLMs: This defines the context window and its components from a developer's perspective. Does More Context Always Help?: This introduces the critical "Lost in the Middle" research finding. Conclusions: This summarizes why proactive context management is a crucial architectural decision.
As the article concludes, simply feeding the model more and more context is not a scalable or effective strategy. It increases cost and latency, and due to the "lost in the middle" effect, may even decrease the quality of the response.
This is why, in later modules, we will spend significant time on techniques to manage the context window effectively, such as:
- Summarization: Using an LLM to periodically condense the conversation history.
- Chunking and RAG: Breaking large documents into smaller pieces and only retrieving the most relevant ones.
- History Management: Using techniques like sliding windows or filters to keep the conversation history from growing indefinitely.
These are core engineering practices for building robust AI agents.
Conclusion
In this lesson, we've moved from theory to the engineering reality of working with LLMs. The constraints imposed by tokens and context windows are not minor details—they are fundamental design considerations for any application you build.
Here are the key takeaways:
- Tokenization is Key: LLMs process numerical tokens, not words. The process of subword tokenization means that text length and token count are not 1:1, directly impacting cost and performance.
- The Context Window is Finite: The context window is the model's working memory, a finite resource measured in tokens. It must contain the system prompt, history, user input, and have room left for the generated output.
- Bigger Isn't Always Better: Large context windows are computationally expensive and can suffer from the "lost in the middle" problem, where the model may ignore information buried in a long prompt.
- Context Management is an Engineering Discipline: As developers, we must actively design our prompts and manage conversation history to work within these limits, ensuring our applications are both effective and efficient.
In our next lesson, we will continue building our foundational knowledge by exploring the different types of LLMs you'll encounter: base models, instruction-tuned models, and chat models, and discuss when to use each one.