Token-Aware Conversation Truncation

Introduction

In our previous lesson, you successfully implemented a persistent memory layer, enabling your agentic system to store and retrieve entire conversation histories. This was a critical step towards creating stateful agents that can maintain context across multiple interactions. However, this solution introduces a new, inevitable challenge: as conversations grow longer, the total number of tokens in the history will eventually exceed the context window limit of the language model, leading to API errors and escalating costs.

Today, we will tackle this problem head-on by implementing a token-aware truncation strategy. You will learn how to manage the size of the conversation history by intelligently pruning older messages, ensuring your application remains robust and cost-effective even during extended interactions. This is a foundational technique in production-grade LLM applications, moving us from simple storage to active context management.

The Problem of Finite Context

An LLM's context window is analogous to a computer's RAM—it's a finite workspace. Every piece of information we send, including the system prompt and the entire conversation history, must fit within this window. Mismanaging this space leads to significant problems.

To effectively manage the context window, we must first understand its fundamental unit of measurement: the token.

What Are Tokens?

You've likely encountered the term "token" frequently, but a precise understanding is crucial for implementation. Tokens are the atomic units of text that an LLM processes. A word can be one or more tokens, and the way text is converted into tokens—a process called tokenization—is specific to each model family.

As you saw, managing history based on message count is unreliable because messages have vastly different token lengths. A truly robust solution must be "token-aware."

The Sliding Window Truncation Strategy

The most direct strategy to manage context size is the sliding window. The concept is simple: as the conversation grows, we maintain a fixed-size window of the most recent messages. When a new message is added, the oldest message "falls out" of the window if the context is full.

When we build the prompt to send to the LLM, we are essentially assembling these pieces. The sliding window determines which part of the conversation history makes it into the final payload.

This approach is a form of context "compression" via trimming, which is a practical and widely used pattern. Now, let's translate this concept into code.

Implementing a Token-Aware Sliding Window in Node.js

We will implement a function that takes the full message history and truncates it to fit within a specified token limit. This function will be called on the server right before you make the streamText call, ensuring the payload sent to the LLM is always valid.

The key is to work backward from the most recent message, adding messages to our context until we approach our token limit. A more straightforward implementation, however, is to start with the full history and trim from the beginning until the total token count is under the limit.

The following resource provides a complete, practical guide to implementing this in a Node.js environment.

Let's synthesize this into a reusable function for our agentic system.

Step 1: Install the Tokenizer

First, you need the library to count tokens for OpenAI models.

npm install js-tiktoken

Step 2: Create the Truncation Logic

Now, let's create a function that enforces our token limit. Based on the logic from the article, it would look something like this.

import { getEncoding } from 'js-tiktoken';
import { UIMessage } from 'ai'; // Or your message type

// Get the tokenizer for a specific model, e.g., 'gpt-4o'
const encoding = getEncoding('cl100k_base'); // This encoding is used by gpt-4, gpt-4o, gpt-3.5-turbo

const MODEL_CONTEXT_LIMIT = 8192; // Example for gpt-4, modern models have more
const RESPONSE_RESERVATION = 1000; // Reserve tokens for the AI's response
const CONTEXT_BUDGET = MODEL_CONTEXT_LIMIT - RESPONSE_RESERVATION;

/**
 * Truncates conversation history to fit within a token budget.
 * Keeps the system message (if any) and the most recent messages.
 * @param messages The full list of messages.
 * @returns A truncated list of messages.
 */
export function truncateHistory(messages: UIMessage[]): UIMessage[] {
  let tokenCount = 0;
  const truncatedMessages: UIMessage[] = [];

  // Start from the end of the history, which has the most recent messages
  for (let i = messages.length - 1; i >= 0; i--) {
    const message = messages[i];
    
    // Always include the system message if it's the first one
    if (message.role === 'system') {
      // Prepend system message to ensure it's at the start
      truncatedMessages.unshift(message);
      continue; // Move to next message
    }

    const messageTokenCount = encoding.encode(message.content).length;
    
    if (tokenCount + messageTokenCount > CONTEXT_BUDGET) {
      // This message would push us over the budget, so we stop here
      break;
    }

    tokenCount += messageTokenCount;
    truncatedMessages.unshift(message); // Prepend to keep chronological order
  }
  
  // Find the original system message if it wasn't added yet (in case it wasn't at index 0)
  const systemMessage = messages.find(m => m.role === 'system');
  if (systemMessage && !truncatedMessages.some(m => m.role === 'system')) {
      truncatedMessages.unshift(systemMessage);
  }

  return truncatedMessages;
}

How to use this in your API route:

// app/api/chat/route.ts
// ... imports

export async function POST(req: Request) {
  const { messages, id }: { messages: UIMessage[]; id: string } = await req.json();

  // 1. Load the full history from the database if needed (as per previous lesson).
  //    For this example, we assume `messages` contains the full history.

  // 2. Truncate the history before sending it to the model.
  const prunedMessages = truncateHistory(messages);

  const result = streamText({
    model: openai('gpt-4o'),
    messages: prunedMessages, // Use the truncated message list
  });

  // ... rest of the handler with onFinish for persistence
}

This implementation is slightly more robust than a simple shift() loop, as it explicitly protects the system message, which often contains critical instructions for the agent's behavior. It builds the context from the most recent messages backward, ensuring relevance, while staying within our defined CONTEXT_BUDGET.

Trade-offs and A Look Ahead

The sliding window strategy is simple and effective at preventing errors, but its major drawback is amnesia. Any information in messages that fall outside the window is completely lost. If a user refers to a detail from the beginning of a long conversation, the agent will have no memory of it.

This limitation is the primary motivation for more advanced context management techniques. While simple truncation is a necessary first step, it's not the final answer for sophisticated agents.

Conclusion

You have now implemented a crucial piece of infrastructure for managing long-running conversations. By making your system "token-aware," you can prevent context window overflows, control costs, and ensure your application remains stable.

Key Takeaways:

• Context windows are finite and measured in tokens, not messages or words.
• The sliding window is a fundamental truncation strategy that keeps the most recent messages while discarding the oldest to stay within a token budget.
• Using js-tiktoken allows you to count tokens accurately for a given model, enabling a truly token-aware implementation in your Node.js backend.
• The primary trade-off of this strategy is the complete loss of information from older, truncated messages.

While we've solved the immediate problem of overflowing the context window, the issue of long-term memory loss remains. In our next lesson, we will address this by exploring a more sophisticated approach: implementing a summarization strategy for older messages to preserve context within token limits. This will allow your agent to "remember" the gist of the entire conversation, not just the most recent part.