Summarizing Older Messages for Context Preservation

Introduction

In our last lesson, you implemented a token-aware sliding window, a robust solution for preventing context window overflows by truncating the oldest messages. While effective at managing API limits and costs, this strategy introduces a significant drawback: amnesia. Any information outside the recent window is lost forever, limiting your agent's ability to reason about long-term conversational context.

Today, we will directly address this limitation. We'll elevate our context management from simple truncation to intelligent compression by implementing a summarization strategy. You will learn to build a system that periodically condenses older parts of the conversation into a dense summary. This preserves the essential context of the entire interaction, allowing your agent to "remember" key facts and decisions from the beginning of a conversation while still respecting the model's token limits. This is a pivotal step toward creating agents with more sophisticated and persistent memory.

The Problem with a Full Context and Simple Truncation

As conversations with an agent grow, two competing problems emerge. On one hand, sending the entire history becomes expensive and eventually hits the model's hard token limit. On the other hand, as we saw previously, simply truncating old messages causes the agent to forget crucial early context.

Furthermore, LLMs don't weigh all parts of the context equally. Research and empirical evidence show a "lost in the middle" problem, where information at the very beginning and very end of the context window has the most impact on the model's output.

This "lost-in-the-middle" effect means that blindly keeping a long history isn't just inefficient; it can actually degrade performance. Summarization offers a solution by transforming the "fluffy" middle of the conversation into a compact, potent piece of context that we can place at the beginning of the prompt.

Core Strategy: Hybrid Summarization and Sliding Window

The most practical and widely-used summarization strategy is not to summarize the entire conversation on every turn, but to use a hybrid approach. This method combines the retention of recent, high-fidelity messages with a compressed summary of older messages.

Here's the logic:

1. The conversation proceeds normally, appending messages to the history.
2. Once the history reaches a predefined threshold (e.g., 75% of the context budget), a summarization process is triggered.
3. The process splits the history: the oldest portion is taken to be summarized, while the most recent portion is kept as-is.
4. A separate LLM call is made to generate a concise summary of the older messages.
5. The old messages are then replaced in the history with a single new message containing the summary.

The new conversation history looks like this: [System Prompt, Summary of old messages, Recent messages]. This structure is highly effective because it places the broad context at the beginning of the prompt and follows it with the immediate, detailed context of recent turns.

The following resource provides an excellent implementation of this logic in Python, which we will adapt for our Node.js environment.

Implementing Summarization in Node.js

Let's translate this powerful concept into TypeScript. Our goal is to create a function that takes the current message history and, if necessary, summarizes it before returning a pruned list of messages ready to be sent to the LLM.

A crucial point to understand is that summarization requires an additional, separate LLM call. This has implications for both cost and latency, a trade-off we accept for the benefit of improved long-term context.

Step 1: The Summarization Helper

First, we need a function that can take a list of messages and ask an LLM to summarize them.

import { openai } from '@ai-sdk/openai';
import { UIMessage } from 'ai'; // Or your message type

/**
 * Creates a summary of a list of messages using an LLM.
 * @param messages The messages to summarize.
 * @returns A string containing the summary.
 */
async function createSummary(messages: UIMessage[]): Promise<string> {
  const model = openai('gpt-4o'); // You can use a smaller/cheaper model for summarization

  // Format the conversation for the summarization prompt
  const conversationText = messages
    .map(msg => `${msg.role}: ${msg.content}`)
    .join('\n');

  const systemPrompt = `You are a conversation summarizer. Your task is to create a concise summary of the following conversation. The summary should preserve key facts, user intents, and important decisions that might be referenced later. The output must be a single block of text, not a list.`;

  const { text } = await model.generate({
    system: systemPrompt,
    prompt: `Summarize this conversation:\n\n${conversationText}`,
    maxTokens: 500, // Limit the summary size
  });

  return text;
}

This function is self-contained. It takes messages, formats them, and uses a dedicated prompt to generate a summary. Note that you could opt for a cheaper, faster model here (like GPT-3.5 Turbo or Claude Haiku) to manage costs.

Step 2: The Main History Management Function

Now, let's create the main function that orchestrates the summarization and truncation. This function will replace the simple truncateHistory function from our previous lesson.

import { getEncoding } from 'js-tiktoken';
import { UIMessage } from 'ai';

const encoding = getEncoding('cl100k_base');
const MODEL_CONTEXT_LIMIT = 8192; // Adjust for your model, e.g., gpt-4o has 128k
const RESPONSE_RESERVATION = 1000;
const CONTEXT_BUDGET = MODEL_CONTEXT_LIMIT - RESPONSE_RESERVATION;

// Trigger summarization when history reaches 75% of the budget
const SUMMARIZE_TRIGGER_THRESHOLD = CONTEXT_BUDGET * 0.75; 

/**
 * Manages conversation history by summarizing old messages when the context
 * grows too large, and truncating recent messages if still over budget.
 * 
 * @param messages The full conversation history.
 * @returns A promise that resolves to a potentially modified message history.
 */
export async function manageHistory(messages: UIMessage[]): Promise<UIMessage[]> {
  // Separate system message, we always keep it
  const systemMessage = messages.find(m => m.role === 'system');
  const conversation = messages.filter(m => m.role !== 'system');

  // Calculate current token count
  const tokenCount = encoding.encode(
    conversation.map(m => m.content).join('')
  ).length;

  let finalMessages: UIMessage[] = [...conversation];

  if (tokenCount > SUMMARIZE_TRIGGER_THRESHOLD) {
    console.log('Summarization triggered. Current tokens:', tokenCount);

    // Find the split point: keep ~40% of recent messages
    const splitIndex = Math.floor(conversation.length * 0.6);
    const toSummarize = conversation.slice(0, splitIndex);
    const keepRecent = conversation.slice(splitIndex);

    // Avoid summarizing if there's nothing significant to summarize
    if (toSummarize.length > 1) { 
      const summaryText = await createSummary(toSummarize);
      const summaryMessage: UIMessage = {
        id: 'summary-' + Date.now(),
        role: 'system', // Treat the summary as part of the system instructions
        content: `[PREVIOUS CONVERSATION SUMMARY]:\n${summaryText}`
      };
      
      // The new history is the summary + recent messages
      finalMessages = [summaryMessage, ...keepRecent];
      console.log('Summarization complete. New message count:', finalMessages.length);
    }
  }

  // --- Fallback to sliding window truncation ---
  // This ensures we are always under budget, even after summarization.
  let finalTokenCount = encoding.encode(
    finalMessages.map(m => m.content).join('')
  ).length;

  while (finalTokenCount > CONTEXT_BUDGET && finalMessages.length > 0) {
    const removedMessage = finalMessages.shift(); // Remove oldest message
    if (removedMessage) {
        finalTokenCount -= encoding.encode(removedMessage.content).length;
    }
  }
  
  // Re-add the original system message at the beginning
  if (systemMessage) {
    finalMessages.unshift(systemMessage);
  }

  return finalMessages;
}

How to integrate this into your API route:
Your API handler now calls manageHistory. A critical difference is that if summarization occurs, you should update your persisted history in the database to reflect this change.

// app/api/chat/route.ts
// ... imports

export async function POST(req: Request) {
  const { messages, id }: { messages: UIMessage[]; id: string } = await req.json();

  // 1. Load the full history from the database (e.g., `fullHistory`).

  // 2. Manage the history. This is now an async operation.
  const prunedMessages = await manageHistory(fullHistory);

  // 3. IMPORTANT: If `prunedMessages` is different from `fullHistory`,
  //    it means summarization happened. You should now save `prunedMessages`
  //    back to your database to replace the old history.
  //    if (prunedMessages.length < fullHistory.length) {
  //      await saveHistoryToDB(id, prunedMessages);
  //    }

  const result = streamText({
    model: openai('gpt-4o'),
    messages: prunedMessages, // Use the managed message list
    // ...
  });
  
  // ... rest of handler
}

This implementation provides a complete, robust context management solution that intelligently preserves long-term memory.

Practical Demonstration and Trade-offs

Seeing this in action solidifies the concept. The following video from LangChain demonstrates a "summarization middleware" that does exactly what our manageHistory function accomplishes.

While powerful, summarization is not a "free lunch." As a senior developer, it's crucial to understand the trade-offs:

• Cost & Latency: As mentioned, each summary is an extra API call. For applications requiring very low latency, this might be a concern.
• Information Loss: Summarization is a lossy compression. While it aims to keep key facts, subtle details or nuances can be lost. The quality of the summary is dependent on the model and the prompt used.
• Trajectory Elongation: Sometimes, a summary can smooth over signals that would have otherwise told an agent to stop. This can lead to agents running for longer than necessary, inadvertently increasing costs.

The following article from JetBrains Research provides a deep dive into these trade-offs, comparing LLM summarization to other techniques.

Advanced Summarization Patterns

For extremely long documents or conversations, a simple one-shot summary may not be sufficient. A more advanced pattern is Hierarchical Summarization, also known as the "Map-Reduce" strategy.

This approach is highly scalable and is a standard technique for processing large volumes of text that far exceed any single context window.

Conclusion

You have now moved beyond basic context management and implemented a sophisticated summarization strategy. By compressing older conversation history instead of discarding it, your agent can maintain long-term context, leading to more coherent and intelligent interactions.

Key Takeaways:

• Summarization solves the "amnesia" problem of simple sliding-window truncation.
• A common and effective pattern is a hybrid approach: summarize the oldest part of the history while keeping the most recent messages in full detail.
• Implementing summarization requires an additional LLM call, introducing a trade-off between context preservation, cost, and latency.
• The new, summarized history should be persisted to your database to make the memory compression permanent for the conversation.
• Advanced techniques like hierarchical summarization exist for handling extremely large contexts.

We have now established the core building blocks of our agent's memory system: persistence, truncation, and summarization. The final step in this module is to abstract these functionalities into a clean, reusable interface. In the next lesson, you will design a memory interface that integrates with the agent hierarchy and orchestrator, making our memory system a modular component that any agent in our infrastructure can use.