In our last lesson, we established the fundamental principle of how Large Language Models operate: they are sophisticated next-token predictors that generate text through an iterative, autoregressive loop. We intentionally left a few key terms like "token," "context window," and "parameters" at a high level. Today, we will precisely define these concepts.
Mastering this vocabulary is essential for your goals of prompt engineering and fine-tuning. These are not just abstract terms; they are the levers that control a model's performance, capability, and cost. Understanding them allows you to move from being a casual user to a developer who can build reliable and efficient LLM-powered applications. This lesson will equip you with the foundational knowledge needed to diagnose model behavior, manage API costs, and understand the trade-offs between different models.
Part 1: Tokens and Tokenization
In the previous lesson, we described tokens as "words or pieces of words." Let's refine that. A token is the fundamental unit of text that an LLM processes. The model doesn't see characters or words as you do; it sees a sequence of numerical token IDs. The process of converting human-readable text into these tokens is called tokenization.
But why not just use words? The following article provides an excellent deep dive, explaining the problems with a word-based approach (like a massive vocabulary and out-of-vocabulary words) and how a method called Byte-Pair Encoding (BPE) solves them by creating subword units.
Temperature, Tokens, and Context Windows: The Three ...
This article from dev.to offers a very practical, developer-focused explanation of tokenization. It moves beyond theory to discuss the real-world implications for cost and performance.
Please read the entire section Tokenization Deep-Dive. As you read, focus on these key points: The definition of a token as a subword unit. The inefficiency of using whole words and how Byte-Pair Encoding (BPE) works. The practical implications for cost, especially regarding common vs. rare words, non-English languages, and structured data like JSON.
As the article highlights, tokenization is not uniform. The rule of thumb that "1 token ≈ 4 characters" or "100 words ≈ 130 tokens" applies reasonably well to common English text, but it breaks down quickly for other data types.
Since you have a background in programming, it's particularly relevant to see how code is tokenized. Notice how punctuation, whitespace, and keywords are often split into their own tokens.
Different models use different tokenizers, which are trained on different data. This means the same text can result in a different number of tokens depending on which model you use. The IBM Technology video below provides a clear, concise visual explanation of tokens and the tokenization process.
What is a Context Window? Unlocking LLM Secrets
This video, "What is a Context Window? Unlocking LLM Secrets," gives a great summary of tokenization.
Watch the segment from "what is a token?" up to the discussion of context window size. It provides clear examples of how a single letter's role in a sentence affects whether it becomes a distinct token.
Part 2: Model Parameters
If tokens are the language of the model, parameters are its brain. Parameters are the internal variables, or weights, that the neural network learns during its training phase. They store all the patterns, knowledge, and nuances of the vast dataset the model was trained on. When you hear that a model like GPT-3 has 175 billion parameters, it refers to the sheer number of these learned variables.
The number of parameters is a primary indicator of a model's potential complexity and capability.
From Tokens to Context Windows: Simplifying AI Jargon - Publications
The Tech Policy Institute provides a very direct and non-technical explanation of what parameters are and why they matter.
Read the section titled Parameters. Pay attention to the distinction that parameters are adjusted during training and are fixed afterward.
While a higher parameter count often correlates with a more powerful model, it's not the only factor. As the article notes, a smaller model trained on higher-quality, more diverse data can sometimes outperform a larger model. More parameters also mean the model requires more computational power and memory to run.
The image below gives you a concrete sense of the scale for both parameters and another key concept we'll discuss next: the context window.
Part 3: The Context Window
The context window is the model's working memory. It defines the maximum number of tokens (including both your input and the model's generated output) that the model can "see" or pay attention to at any single moment. Anything outside this window is forgotten.
This is arguably the most critical concept for you to master for both prompt engineering and practical application development. The size of the context window dictates how long of a conversation you can have, how much document text you can analyze, and how complex your instructions can be.
The IBM video provides an excellent high-level explanation of what a context window is and the challenges associated with increasing its size.
What is a Context Window? Unlocking LLM Secrets
This video provides a solid conceptual foundation for understanding context windows.
First, watch the introduction from the beginning to get a clear analogy for the context window as a "working memory." Next, watch the segment that explains what fills the window, including system prompts and documents for Retrieval-Augmented Generation (RAG). Finally, watch the section on the challenges of large context windows, which introduces the critical concepts of quadratic scaling (O(n²)) and the "lost in the middle" problem.
To see these concepts in action, the next video gives a hands-on demonstration. It shows a model literally "forgetting" information once the context window is full and illustrates the immense hardware (VRAM) cost of using a large context window—a critical consideration for when you begin fine-tuning models locally.
Why LLMs get dumb (Context Windows Explained)
This video from NetworkChuck, "Why LLMs get dumb," provides a very practical and visual demonstration of context window limitations.
First, watch the introduction for an analogy of the context window as short-term memory. Next, see a demonstration of a model "forgetting" information when the context is exceeded. Finally, and most importantly for your future work, watch the segment on hardware impact, which shows how increasing the context window size consumes a massive amount of GPU memory (VRAM).
The key trade-off with the context window is this: a larger window allows the model to maintain more context for longer, more complex tasks, but it comes at a significant cost in terms of computation, speed, and potential performance degradation (the "lost in the middle" effect).
Conclusion
In this lesson, you have learned the precise definitions of the fundamental components of any Large Language Model. These concepts are the bedrock for everything that follows in this course.
Here are the key takeaways:
- Tokens are the subword units that an LLM processes. Tokenization is the process of converting text to tokens, and the number of tokens generated depends on the language, text complexity, and the specific tokenizer used.
- Model Parameters are the billions of weights learned during training that encode the model's knowledge. More parameters generally lead to more capable models but require more computational resources.
- The Context Window is the model's fixed-size working memory, measured in tokens. It limits how much information the model can process at once and presents a trade-off between contextual awareness and computational cost.
With this foundational theory in place, you are now ready to start interacting with a model programmatically. In our next lesson, we will begin Module 2 by setting up the OpenAI Python client and making your first API call to a state-of-the-art LLM. You'll see immediately how these concepts of tokens and context windows translate into practical API parameters and costs.