Skip to main content
Back to path

Welcome to a Grasp lesson

Create a free account to keep studying and create your own study path.

Lesson 6

Memory Systems & Agent Context

Recommended resources

Hello! Welcome back to your course on AI agent development.

In our previous lessons, we built up the core of an agent. We started with the fundamental agent loop (Perceive, Reason, Act) and then gave the agent "hands" through tool use, allowing it to interact with external APIs and systems.

However, for an agent to perform any task that takes more than a single step, it needs to remember what's happened before—the user's last message, the result of a tool it just used, or a preference the user expressed last week. This is where memory comes in. Today's lesson addresses the crucial role of memory systems in maintaining an agent's context, transforming it from a simple "one-shot" tool into a coherent and stateful assistant.

The Need for Memory: Overcoming Statelessness

The Large Language Models that power agents are, by their nature, stateless. Each time you send a request to an LLM's API, it processes that request in isolation, with no inherent knowledge of your previous interactions. To build a continuous, coherent experience, we must explicitly provide the model with context from the past. This is the fundamental job of an agent's memory system.

Let's start with a short video that explains why this concept, inspired by human psychology, is so critical for building effective agents.

Building Brain-Like Memory for AI | LLM Agent Memory Systems

The video 'Building Brain-Like Memory for AI' from Adam Lucek's channel provides an excellent introduction to why memory is essential for agentic systems due to the stateless nature of LLMs.

Watch the first two minutes of the video (00:00 - 01:58). Focus on the core problem: LLMs lack prior recollection, and how memory systems are designed to compensate for this by simulating human cognitive approaches.

As the video explains, without memory, an agent would be like a person with severe amnesia, unable to build on past conversations or learn from experience. To solve this, we architect memory systems that are broadly categorized into two types: short-term memory and long-term memory.

  • Short-Term Memory (STM): Holds information relevant to the current, ongoing conversation or task. It's volatile and typically forgotten once the session ends. Think of it as the agent's "working memory" or scratchpad.
  • Long-Term Memory (LTM): Stores information persistently across multiple sessions. This is how an agent "learns" about a user's preferences, remembers past conversations, and accumulates knowledge over time.

Short-Term Memory: The Conversational Scratchpad

Short-term memory is what allows an agent to follow a conversation from one turn to the next. Its primary role is to maintain immediate context.

How It Works: The Conversation Buffer

The most common implementation of STM is a conversation buffer. This is simply a list of the back-and-forth messages between the user and the agent. With each new user query, the recent chat history is included in the prompt sent to the LLM. This gives the model the context of the immediate past, allowing it to generate a relevant response.

Let's see a practical demonstration of this.

Building Brain-Like Memory for AI | LLM Agent Memory Systems

We'll return to Adam Lucek's video, where he explains and demonstrates this fundamental form of memory, which he refers to as 'working memory'.

Watch the segment from 03:03 to 08:32. Pay close attention to how the message_history list is appended with each new user and AI message and then passed back to the LLM in subsequent calls. This is the core mechanism of short-term memory.

STM in an Agent Development Kit (ADK)

While a simple list is a good illustration, production-grade frameworks provide more robust abstractions. In Google's Agent Development Kit (ADK), short-term memory is managed through the concepts of a Session and its State.

  • Session: A container for a single, continuous interaction with the agent. It holds the chat history (the conversation buffer).
  • State: A key-value store within the Session that acts as a temporary scratchpad. Both the agent's LLM and its tools can read from and write to the state, allowing them to track information like a user's current quiz score or items in a shopping cart.

This video from Google Cloud Tech explains how ADK implements these concepts.

Remember this: Agent state and memory with ADK

The video 'Remember this: Agent state and memory with ADK' provides a framework-specific view of short-term memory.

Watch from 02:18 to 04:34. Notice how Session and State are presented as the agent's 'scratchpad for the current conversation'. This is a direct, practical application of the STM concept.

As a software engineer, you can think of the Session object as the in-memory representation of a user's current interaction, which is managed by a SessionService (e.g., an InMemorySessionService for prototyping or a database-backed service for production).

Long-Term Memory: Building a Persistent Identity

While STM is essential for coherence within a single session, LTM is what gives an agent the ability to learn and personalize experiences over time. It allows the agent to recall information from past conversations.

Storing entire raw conversations can be inefficient and costly. A more sophisticated approach is to categorize long-term memories, much like human cognition. A common and useful taxonomy includes:

  1. Episodic Memory: Stores specific past interactions or "episodes." This is like the agent's diary, remembering entire conversations and their outcomes. It's often implemented using a vector database to find semantically similar past conversations.
  2. Semantic Memory: Stores general knowledge, facts, and concepts. This is the agent's reference library. In practice, this is often implemented using Retrieval-Augmented Generation (RAG), where the agent retrieves relevant documents from a knowledge base (like a vector database of product manuals) to ground its responses in facts.
  3. Procedural Memory: The most abstract type, this is "how-to" knowledge or skills the agent has learned. In an agent, this can be represented by the agent's core code, its system prompt, or a set of learned instructions that guide its behavior.

This architecture is illustrated very well in the following diagram.

Agentic AI Memory Management Architecture
This architectural diagram shows the distinction between Short-Term Memory (STM) and Long-Term Memory (LTM). It breaks LTM into its constituent parts—Semantic, Episodic, and Procedural—and shows how they interact with underlying technologies like vector databases and knowledge graphs.

Let's watch how these LTM concepts are implemented.

Building Brain-Like Memory for AI | LLM Agent Memory Systems

Adam Lucek's video provides excellent explanations for implementing episodic and semantic memory.

Watch the following two segments: Episodic Memory (08:32 - 17:18): See how past conversations are processed to extract 'reflections' and stored in a vector database for later retrieval. Semantic Memory (21:56 - 29:44): Understand how this is implemented as a classic RAG system to provide factual grounding.

LTM in an Agent Development Kit (ADK)

Frameworks like Google's ADK provide services to manage LTM. The VertexAiMemoryBankService is a powerful example. Instead of just storing raw chat logs, it uses an LLM to intelligently process and summarize a conversation, extracting the most important facts. This is highly efficient for production systems.

Remember this: Agent state and memory with ADK

Let's return to the Google ADK video to see how long-term memory is handled in a production-oriented framework.

Watch from 04:34 to the end (08:00). Pay close attention to how Memory Bank summarizes facts from a conversation (e.g., 'My name is Megan', 'I scored 75% on the dictionaries quiz') rather than storing the whole chat. This is a key production practice.

The official ADK documentation provides a clear breakdown of the different memory services and the internal workflow. As a developer, understanding these details is key to choosing the right tool for the job.

Memory - Agent Development Kit

The official ADK documentation on 'Memory' provides the technical details behind the concepts we've seen in the videos.

Please read the following sections: Introduction and 'The MemoryService Role': This clearly separates Session/State (STM) from MemoryService (LTM). 'Choosing the Right Memory Service': Focus on the comparison table. It highlights the trade-offs between a simple in-memory service for prototyping and the advanced, persistent VertexAiMemoryBankService. 'How Memory Works in Practice': This section details the internal step-by-step workflow, which will be very familiar to you as a software engineer.

Test your understanding!

You are designing a customer support agent for an e-commerce site. The agent needs to handle the following two requirements:

  1. During a single chat session, it must remember which products the user has asked about so it can offer a summary at the end.
  2. When the same user returns a week later, the agent should greet them by name and be aware of their previously expressed interest in a specific product category (e.g., "running shoes").

For each requirement, state whether it would be handled by short-term memory or long-term memory, and describe a typical technical implementation for it.

Show answer

  1. Requirement 1 (Remembering products in-session): This is a job for short-term memory (STM).

    • Implementation: You would use the agent's State object (in an ADK context) or a similar temporary data structure within the current Session. As the agent identifies products the user asks about, it would add them to a list in the State. At the end of the chat, it can read from this list to provide the summary. The State and Session are discarded after the conversation ends.

  2. Requirement 2 (Remembering user info across sessions): This requires long-term memory (LTM).

    • Implementation: At the end of the first conversation, you would use a MemoryService (like Google's Memory Bank or a custom solution with a vector database) to save key information. The service would extract and store facts like user_name: 'John' and interested_category: 'running shoes'. When the user returns, the agent would query this LTM service using the user's ID to retrieve this information and personalize the greeting.

Conclusion

In this lesson, we've established memory as a foundational pillar of agent architecture, enabling them to maintain context and learn over time.

Key Takeaways:

  • Memory counters LLM statelessness: It's the mechanism for providing context to otherwise isolated API calls.
  • Short-Term Memory (STM) provides context within a single conversation, typically via a chat history buffer or a session state object. It's the agent's working scratchpad.
  • Long-Term Memory (LTM) provides context across conversations, enabling personalization and learning. It's the agent's persistent knowledge store.
  • LTM has different flavors: We can model LTM as episodic (past events), semantic (facts), and procedural (skills) to build more sophisticated agents.
  • ADKs offer abstractions: Frameworks like Google's ADK provide building blocks like Session, State, and MemoryService to simplify the implementation of both STM and LTM.

So far, we have covered the agent loop (the engine), tool use (the hands), and now memory (the context). We have all the individual components needed to build a powerful agent. In our next lesson, we will explore the ReAct pattern, a powerful orchestration strategy that weaves these three components together, allowing an agent to reason and plan its way through complex, multi-step tasks.