Agent Memory: Short-Term Context Retention

Part 1: Why Conversational Memory is Necessary

At their core, Large Language Models are stateless. Each time you send a request to an LLM's API, it processes that request in isolation, with no inherent knowledge of past requests. To create the illusion of a continuous conversation, we must manually provide the relevant history with each new query.

This "memory" is simply the history of messages (both from the user and the agent) that we include in the prompt we send to the LLM. This allows the model to understand context, resolve references (e.g., knowing what "it" refers to), and build upon previous turns of the conversation.

To start, let's watch a brief, clear explanation of why memory is so crucial for building conversational agents.

The fundamental technique, as the video explains, is to put the conversation history back into the prompt. The immediate challenge this creates is managing the LLM's finite context window. A long conversation can easily exceed the token limit, leading to errors or significant costs.

Part 2: Strategies for Managing Short-Term Memory

Since we can't just append messages indefinitely, we need strategies to keep the conversation history a manageable size while retaining the most important context. Your choice of strategy involves trade-offs between context fidelity, token usage (cost), and complexity. Given your background in product management and system design, thinking about these trade-offs is key.

The following video segments provide a practical walkthrough of the most common memory strategies available in LangChain. While the syntax is from an older version of the library, the concepts are universal and provide a clear mental model for each approach.

Strategy 1: The Full History (ConversationBufferMemory)

This is the simplest approach: append every message to the history. It's perfect for short conversations but will quickly fail as the conversation grows.

Strategy 2: The Sliding Window (ConversationBufferWindowMemory)

A simple and effective fix for the context limit problem. This strategy only keeps the last k number of conversational turns. It's computationally cheap but risks losing important information from earlier in the conversation.

Strategy 3: The Summary (ConversationSummaryMemory)

Instead of just dropping old messages, this strategy uses an LLM to progressively summarize the conversation. This can preserve information over a longer dialogue but introduces extra latency and cost due to the summarization calls.

Strategy 4: The Hybrid (ConversationSummaryBufferMemory)

This approach combines the speed of a sliding window with the context preservation of a summary. It keeps recent messages in their raw form while compressing older parts of the conversation into a summary. This often represents the best balance for many applications.

Part 3: Implementing Persistent Memory in LangGraph

Now, let's move from theory to practice. The modern way to handle memory in LangChain is with LangGraph, which treats agent execution as a stateful graph. Memory isn't just an add-on; it's a core part of the agent's state that gets persisted between runs.

The key components are:

• Thread ID: A unique identifier for a single conversation. When you make calls with the same thread_id, the agent loads the history for that conversation.
• Checkpointer: An object responsible for saving and loading the agent's state (which includes the message history) to a backend. This is what makes the memory persistent.

For this lesson, we will use an in-memory checkpointer (InMemorySaver), which is perfect for development. In a production system, you would use a persistent backend like Postgres or Redis.

Let's begin by studying the official LangChain documentation, which is the most up-to-date resource for this implementation.

Code Implementation

Now, let's put this into practice. We'll build a simple ReAct agent similar to the one from our last lesson, but this time we'll make it stateful using a checkpointer.

The following code sets up a basic search tool and an agent. The crucial additions are the InMemorySaver and the config dictionary with a thread_id.

import os
from langchain_openai import ChatOpenAI
from langchain.agents import create_tool_calling_agent
from langgraph.checkpoint.memory import InMemorySaver
from langgraph.prebuilt import create_agent_executor
from langchain_core.tools import tool
from langchain_core.runnables import RunnableConfig

# Set your OpenAI API key
# os.environ["OPENAI_API_KEY"] = "your_api_key"

@tool
def search_project_database(query: str) -> str:
    """Searches a dummy project database for information on renewable energy projects."""
    print(f"--- Searching database for: {query} ---")
    if "greenfield solar project in lagos" in query.lower():
        return "Project 'Helios One' is a 50MW C&I solar project in Lagos, Nigeria. Its primary offtaker is a major manufacturing firm. The projected DSCR is 1.4x."
    else:
        return "No specific project found for that query."

# 1. Setup the model and tools
llm = ChatOpenAI(model="gpt-4o")
tools = [search_project_database]
agent_runnable = create_tool_calling_agent(llm, tools)

# 2. Define the checkpointer for memory
# This will store the state of our conversations in memory
memory = InMemorySaver()

# 3. Create the agent executor
# The checkpointer is passed here to make the agent stateful
app = create_agent_executor(agent_runnable, tools, checkpointer=memory)

# 4. Define a unique ID for our conversation thread
config: RunnableConfig = {"configurable": {"thread_id": "project-finance-thread-1"}}

# --- Run 1: Ask an initial question ---
print("--- First Interaction ---")
response1 = app.invoke(
    {"messages": ["Hi, can you find me information on a greenfield solar project in Lagos?"]},
    config=config
)
print(f"Agent Response: {response1['messages'][-1].content}\n")

# --- Run 2: Ask a follow-up question in the same thread ---
print("--- Second Interaction (in the same thread) ---")
response2 = app.invoke(
    {"messages": ["What is the projected DSCR for that project?"]},
    config=config
)
print(f"Agent Response: {response2['messages'][-1].content}\n")

# --- Run 3: Start a new, separate conversation thread ---
print("--- Third Interaction (in a NEW thread) ---")
config_new_thread: RunnableConfig = {"configurable": {"thread_id": "new-unrelated-thread-2"}}
response3 = app.invoke(
    {"messages": ["What was the DSCR we were just talking about?"]},
    config=config_new_thread
)
print(f"Agent Response: {response3['messages'][-1].content}\n")

Run this code. You will observe:

1. In the first interaction, the agent uses the search_project_database tool to find the 'Helios One' project.
2. In the second interaction, because we use the same thread_id, the agent remembers the context of 'Helios One' and correctly answers the question about its DSCR without needing to search again.
3. In the third interaction, we use a new thread_id. The agent, now in a separate conversation, has no memory of 'Helios One' and cannot answer the question.

This demonstrates that you have successfully implemented persistent short-term memory, with conversations cleanly separated by thread_id.

Part 4: Production Architecture and Patterns

While InMemorySaver is great for getting started, a production system requires more robust solutions for both persistence and managing long conversations.

For managing long conversations in LangGraph, you don't use the old memory objects directly. Instead, you use middleware that modifies the agent's state before the LLM call.

The LangChain documentation provides code examples for the modern way to implement trimming and summarization.

This middleware approach is more modular and aligns better with the graph-based structure of modern agents, allowing you to declaratively add capabilities like summarization to your agent executor.

Conclusion

You have now equipped your agent with one of the most fundamental capabilities: a memory. You've moved from a stateless calculator to a stateful conversational partner.

Key Takeaways:

• Memory is Context: Short-term memory is about providing conversation history to the stateless LLM so it can maintain context.
• Strategies are Trade-offs: You must choose a strategy (full history, windowed, summary, or hybrid) that balances context preservation with token limits and cost for your specific use case.
• Checkpointers and thread_id are the Implementation Keys: In modern LangGraph, you make an agent stateful by adding a checkpointer and manage separate conversations using a unique thread_id in the config.
• Middleware for Management: Advanced memory management techniques like summarization are now implemented as middleware that plugs into your agent executor.

In this lesson, the agent's memory was limited to the conversation itself. It couldn't draw on external knowledge unless it explicitly used a tool. What if you want your agent to have access to a vast, private knowledge base, such as all your firm's previous investment memos, legal templates, or market research reports?

That requires a different kind of memory. In our next lesson, "Implement a long-term memory system using a vector database for Retrieval-Augmented Generation (RAG)," you will learn how to build this "external brain" for your agent, allowing it to pull in relevant information from a large corpus of documents to answer questions and complete tasks.