Agent Memory: Short-term and Long-term

Hello! In our last lesson, we explored how agents create and follow complex plans. We ended with a crucial question: how does an agent keep track of the conversation, its past actions, and what it has learned? Without this ability, every interaction would start from scratch, making complex, multi-step tasks impossible.

This lesson directly answers that question by focusing on a core component of any intelligent system: memory. We'll be addressing the learning outcome: Describe memory types in agents: short-term (context window) and long-term (external storage).

By the end of this lesson, you will understand:

• The fundamental difference between short-term and long-term memory in AI agents.
• How short-term memory works via the context window and its inherent limitations.
• How long-term memory is implemented using external storage to give agents persistence.
• The common categories of long-term memory: episodic, semantic, and procedural.

Let's begin by tackling the core problem that memory systems are designed to solve.

The Stateless Nature of LLMs

At their core, Large Language Models are stateless. Each time you send a request to an LLM API, the model processes that input without any inherent recollection of past interactions. This is analogous to a web server that treats every HTTP request as a completely new event. To build a coherent, multi-turn conversation or a stateful agent, we, as engineers, must build a memory system around the LLM.

The following article introduces this concept well, framing memory as a fundamental challenge in "context engineering."

This brings us to the first and most immediate type of memory that nearly all chatbot applications use.

Short-Term Memory: The Context Window

Short-term memory (STM) is the agent's active workspace. In the context of LLMs, this is implemented using the context window. Think of it as the model's RAM—it's fast, directly accessible, but volatile and limited in size.

Here’s how it works:

1. The user sends a message.
2. The application code prepends the entire conversation history (previous user messages and AI responses) to the new message.
3. This combined text is sent to the LLM.

This way, the model "sees" the recent conversation and can maintain context. The following video provides a simple, clear visual explanation of this process.

While simple and effective for brief conversations, relying solely on the context window presents several engineering challenges:

• Finite Size: Context windows have a token limit (e.g., 4k, 32k, 200k tokens). Long conversations will eventually exceed this limit, forcing you to drop older messages.
• Cost: API calls are often priced per token. As the conversation history grows, each turn becomes more expensive.
• Latency: Larger context windows require more processing time, leading to slower responses.
• Context Distraction: As mentioned in the Weaviate article, stuffing the context window with too much (or irrelevant) information can confuse the model, leading to poorer quality responses. The model might over-rely on historical patterns instead of reasoning about the current query.
• Volatility: The memory is lost once the session or application instance is terminated.

This image provides a great conceptual map, distinguishing the model's internal, volatile memory from the external, persistent memory we'll discuss next.

To overcome the limitations of STM, we need a way to store information permanently.

Long-Term Memory: External, Persistent Storage

Long-term memory (LTM) gives an agent the ability to recall and learn from information across different sessions and long periods. It functions like a computer's hard drive—a persistent storage system that the agent can read from and write to.

This is typically implemented using an external database. For an agent, the flow looks like this:

1. Store: During or after a conversation, the agent extracts important information.
2. Persist: This information is saved in an external database, often a vector database (like ChromaDB, Weaviate, or Pinecone) for semantic information, or a key-value store (like Redis) for structured data.
3. Retrieve: On a future interaction, the agent searches this database for memories relevant to the current query.
4. Inject: The retrieved memories are then loaded into the short-term memory (context window) along with the user's prompt to provide relevant context for the LLM's response.

The following article section provides an excellent architectural overview of this system.

As the article notes, LTM can hold different kinds of information. It's helpful to categorize them based on what they represent.

Types of Long-Term Memory

1. Semantic Memory (Facts & Knowledge):
   This is the agent's knowledge base of facts. It answers the question, "What is true?". This is often implemented using the Retrieval-Augmented Generation (RAG) pattern we'll cover in detail later. For now, think of it as taking documents, chunking them, embedding them, and storing them in a vector database for later retrieval.

   This video clip explains the concept clearly.

2. Episodic Memory (Events & Conversations):
   This is the agent's memory of specific past events or interactions. It answers the question, "What happened?". It's about remembering the history of experiences. This can be implemented by summarizing conversations and storing those summaries, or even storing full conversation logs linked to timestamps.

3. Procedural Memory (Workflows & Skills):
   This is the agent's memory of how to perform tasks. It answers, "How do I do something?". This could involve storing successful plans from previous tasks or step-by-step instructions for using a tool. When faced with a similar task, the agent can retrieve this procedure to guide its actions.

The image below shows a potential system architecture, illustrating how short-term and long-term memory (here, in MongoDB) fit into a larger agentic application.

Putting It All Together: A Hybrid System

No sophisticated agent relies on just one type of memory. They use a hybrid system where short-term and long-term memory work in synergy.

• Long-Term Memory acts as the vast, persistent library of knowledge and experience.
• Short-Term Memory (the context window) is the active, temporary workbench where information retrieved from LTM is combined with the current situation to perform a task.

The agent's intelligence lies in its ability to effectively manage this flow—deciding what to save to LTM, what to retrieve from LTM, and what to keep in the immediate STM to solve the problem at hand.

The following video provides an excellent summary of how these different memory systems interact to create a more capable agent.

Conclusion

In this lesson, we demystified how AI agents remember information. You've learned that they overcome the stateless nature of LLMs by employing a two-part memory system.

Key Takeaways:

• Short-Term Memory (STM) is the agent's active workspace, implemented via the context window. It's fast but volatile, costly, and limited in size.
• Long-Term Memory (LTM) is the agent's persistent knowledge store, implemented using external databases (like vector or key-value stores). It provides the agent with continuity across sessions.
• The two form a hybrid system: LTM is the library, and STM is the workbench. The agent's task is to bring the right books (memories) from the library to the workbench to solve the current problem.
• LTM can be categorized to hold different types of information: semantic (facts), episodic (events), and procedural (skills).

We now have a solid understanding of an agent's internal world: its ability to reason (ReAct), create strategies (Planning), and retain information (Memory). Now, we need to formalize how it interacts with the outside world.

Our next lesson, "Define tool schemas that describe function signatures for an LLM," will kick off a new module on tool use. We'll explore how we can provide an agent with a set of capabilities—tools—and, crucially, how we describe those tools so the agent knows what they do and how to use them.