Welcome back! In our last lesson, you saw firsthand how different LLMs have unique "personalities" by comparing their responses to the same prompt. Today, we'll dive into one of the most critical technical constraints that shapes a model's behavior: its context window.
This lesson will explain what a context window is, why it's like a model's short-term memory, and what happens when you push that memory to its limits. By the end, you'll have run your own experiment to observe how a model's performance can degrade as you feed it more and more information. This is a fundamental concept for building reliable AI products.
What is a Context Window?
Think of a context window as the total amount of information an LLM can "see" at one time. This includes everything: the initial instructions (system prompt), your questions (user messages), and the model's own answers (assistant messages). Every time you send a new message in a conversation, the entire history is typically sent back to the model so it can follow along.
But this memory isn't infinite. Each model has a fixed-size context window, measured in tokens. When a conversation gets too long, the model starts to "forget" things, just like a person might forget details from the beginning of a long talk.
To get a solid grasp of this concept, watch this short video.
Most devs don’t understand how context windows work
Matt Pocock's video "Most devs don’t understand how context windows work" gives a very clear and concise explanation of what makes up the context window and what happens when you exceed its limit.
Focus on these two parts: What's in the window: This part defines exactly what gets counted as context. Hitting the limit: Here, you'll see what happens when the input or output exceeds the model's hardcoded limit.
The size of the context window varies dramatically between models. Here’s a quick look at the advertised context windows for some of the models you've already seen, as detailed in the article "Context Rot" from ProductTalk.
| Model | Context Window |
|---|---|
| Claude 3.5 Sonnet | 200K tokens |
| GPT-4o | 128K tokens |
| Gemini 1.5 Pro | 1M tokens |
A million tokens sounds huge, but as you'll see next, a bigger context window doesn't always mean better performance.
The "Lost in the Middle" Problem
Even with massive context windows, LLMs struggle to pay attention to all information equally. Research has shown that models tend to remember information best from the beginning and the end of the context. Information buried in the middle often gets ignored or "lost." This phenomenon is sometimes called "context rot" or the "Needle in a Haystack" problem.
This next clip from Matt Pocock's video illustrates this U-shaped attention span perfectly.
Most devs don’t understand how context windows work
This segment explains the "Lost in the Middle" problem and why it's so important for anyone building with AI.
Watch from Primacy and Recency Bias. Pay close attention to the simple graph he draws. It clearly shows how the model's attention is highest at the start and end of a conversation, and dips significantly in the middle.
This isn't just a theory; it's a measurable effect. Engineers and researchers test for this using a "Needle in a Haystack" test. They place a specific piece of information (the "needle") into a long document (the "haystack") and see if the model can find it. By moving the needle to different positions and changing the haystack's length, they can map out the model's performance.
Look at this heatmap from a real Needle in a Haystack test run on GPT-4.
You can clearly see the model performs well when the needle is at the very beginning (0% depth) or the very end (100% depth) of the document. But there's a large blue area in the middle where the model consistently fails, especially as the context gets longer.
Your Turn: Run a "Needle in a Haystack" Test
Now, let's run a simplified version of this test yourself. You'll give a model a piece of secret information (the needle) buried in a long text about event planning (the haystack) and see if it can retrieve it.
We'll use OpenRouter again, just like in the last lesson.
1. The Needle:
Here's our "needle." It's a specific, unusual fact related to your interests.
The secret password for the DJ's audio-reactive visuals is "Cattaneo-Cox-2024".
2. The Haystack:
Here is a long, generic text about event planning. Copy this entire block of text.
Event planning is the process of managing a project such as a meeting, convention, tradeshow, ceremony, team building activity, party, or concert. It involves understanding the client's vision, establishing a timeline, selecting and reserving event sites, acquiring permits, planning food, coordinating transportation, developing a theme, arranging for activities, selecting speakers and keynotes, arranging for equipment and facilities, managing risk, and developing contingency plans. The event manager is the person who plans and executes the event, taking responsibility for the creative, technical, and logistical elements. This includes overall event design, brand building, marketing and communication strategy, audio-visual production, scriptwriting, logistics, budgeting, negotiation, and client service. Because of the complexities involved and the numerous details that must be handled, professional event planners are often hired to take charge of the event's execution. They are experts in managing the intricate details of an event, ensuring that everything runs smoothly and according to plan. This allows the client to focus on their guests and the purpose of the event without being bogged down by the operational aspects.
The initial stage of planning involves determining the purpose and objectives of the event. Is it for a celebration, education, promotion, or commemoration? Clarifying the 'why' behind the event is crucial as it guides all subsequent decisions. Once the objectives are clear, the next step is to establish a budget. The budget will dictate many of the choices, from the venue to the catering to the entertainment. It is essential to be realistic about costs and to include a contingency fund for unexpected expenses. With a budget in place, the planner can begin to scout for venues. The choice of venue is critical; it must not only fit the budget but also be appropriate for the size and style of the event, be accessible to attendees, and have the necessary facilities. After securing a venue, the focus shifts to vendors. This includes caterers, decorators, entertainers, photographers, and transportation services. It is important to work with reputable and reliable vendors who can deliver high-quality service. Contracts should be reviewed carefully, and all terms should be clearly understood before signing.
Marketing and promotion are also key components of event planning, especially for public events. A solid marketing strategy is needed to reach the target audience and drive attendance. This might involve social media campaigns, email marketing, public relations, and advertising. For corporate events, internal communications are just as important to ensure employees are informed and engaged. As the event date approaches, the logistical coordination intensifies. This involves creating a detailed timeline or run-of-show, coordinating the arrival and setup of all vendors, managing staff and volunteers, and preparing for any potential issues. On the day of the event, the event planner is on-site to oversee everything, ensuring that the plan is executed flawlessly and that any problems are dealt with swiftly and discreetly. After the event, post-event activities include collecting feedback, settling final payments, and conducting a debrief to evaluate the event's success against its objectives. This entire lifecycle requires a unique blend of creativity, organization, and communication skills.
3. The Experiment:
Use a model like Anthropic: Claude 3.5 Sonnet or OpenAI: GPT-4o in OpenRouter for these tests. After each prompt, clear the chat to start fresh.
-
Test 1: Needle at the Beginning
Paste the needle first, then paste the haystack immediately after it in the same prompt. Then, on a new line, ask:What is the secret password for the visuals?
-
Test 2: Needle at the End
Start a new chat. Paste the haystack first, then paste the needle at the very end. Then ask:What is the secret password for the visuals?
-
Test 3: Needle in the Middle
Start a new chat. Copy the first two paragraphs of the haystack. Paste them. Then, paste the needle. Finally, paste the third paragraph of the haystack. Now, the needle is buried. Ask the model:What is the secret password for the visuals?
Did the model find the password in all three cases? You will likely see that it succeeds easily in tests 1 and 2 but might struggle or fail in test 3. You have just replicated the "Lost in the Middle" problem.
Conclusion
Fantastic work today. You've gone beyond just using an LLM and have started to probe its limitations like a true AI engineer.
Key Takeaways:
- The context window is an LLM's finite short-term memory, which includes all prompts and responses in a conversation.
- Models tend to remember information best from the beginning and end of the context.
- The "Lost in the Middle" problem causes models to ignore information buried deep inside a long input, degrading their performance.
- The "Needle in a Haystack" test is a standard method for evaluating how well a model handles long contexts.
In our next lesson, we'll explore another key setting that controls an LLM's output: temperature. You'll experiment with how to make a model's responses more creative and varied, or more focused and deterministic.