Building an In-Memory State Manager

Introduction

Welcome back! In our last lesson, we implemented a simple "post-tool hook" by enriching the response of our get_order_details tool directly within the endpoint. While effective, the summary we added was ephemeral—it existed for a single response and was then forgotten.

In conversational AI, however, memory is crucial. Your experience with Rasa Pro has shown the importance of "slots," which are key-value stores that remember information across multiple turns of a conversation. For example, after fetching an order, you might want to "remember" the order_id so that a subsequent user request like "now show me the customer for that order" can be fulfilled without asking for the ID again.

This lesson directly addresses that need. Our learning outcome is to design a simple in-memory dictionary to act as a state manager, analogous to 'slots' in conversational AI. We will explore how to create a state that persists between tool calls, setting the foundation for more complex, multi-step interactions.

State Management: Global vs. Session

In a web framework like FastAPI, there are two primary ways to think about in-memory state:

Global State: A single data structure (like a dictionary) that is shared by every user and every request. This is useful for loading shared resources like a machine learning model once at startup. However, for conversational "slots," this is problematic. If User A's order_id is stored, User B might accidentally retrieve it. There is no isolation.
Session State: A separate, isolated data structure for each user session. This is the perfect model for conversational slots, as it ensures one user's conversation state does not interfere with another's.

Fortunately, the fastapi-mcp library provides a powerful, built-in mechanism for session state, which is exactly what we need. While we could build a global state manager using FastAPI's lifespan events, the library's solution is purpose-built for this use case and much simpler to implement.

Let's briefly look at the lifespan approach to understand the concept, and then pivot to the fastapi-mcp way, which will be our chosen design.

The `lifespan` Approach (For Global State)

FastAPI allows you to run code at application startup and shutdown. This is ideal for creating a global, shared resource.

Lifespan Events - FastAPI

The official FastAPI documentation on 'Lifespan Events' explains how to manage resources that should exist for the entire lifetime of the application. Please review this to understand the concept of a global state.

Read the sections 'Lifespan Events', 'Use Case', and 'Lifespan'. Pay close attention to the code example where a dictionary ml_models = {} is created within a lifespan function. This is a perfect illustration of how a global, shared dictionary is managed in FastAPI.

From the reading, you can see how you could create a global dictionary:




# Conceptual example - do not add this to your code
from contextlib import asynccontextmanager
from fastapi import FastAPI

app_slots = {} # A global dictionary

@asynccontextmanager
async def lifespan(app: FastAPI):



    # Code to run on startup
    print("Server starting up, initializing global slots...")
    app_slots["global_counter"] = 0
    yield



    # Code to run on shutdown
    print("Server shutting down, clearing global slots...")
    app_slots.clear()




# app = FastAPI(lifespan=lifespan) # You would initialize the app like this

While this works, every single user interacting with your MCP server would be reading and writing to the same app_slots dictionary. This is not what we want for conversational memory.

The `fastapi-mcp` `Context` Approach (For Session State)

The fastapi-mcp library solves the isolation problem for us with its Context object. It automatically manages an in-memory dictionary-like state for each unique client session. This means we get the "slot" behavior you're familiar with, right out of the box.

The key to this functionality is the Context object, which is accessible via dependency injection within your tool endpoints.

MCP Context

The fastapi-mcp documentation on 'Context' details a powerful object for interacting with the MCP client. We will focus on its session state capabilities.

Please read the introduction ('What Is Context?'), 'Accessing the Context', and most importantly, the 'Session State' section. Focus on how the CurrentContext() dependency is used and the purpose of the ctx.get_state() and ctx.set_state() methods.

Our State Manager Design

Based on that reading, our design for a state manager is clear and simple: we will use the session state feature provided by the fastapi-mcp Context object.

This design fulfills the learning outcome perfectly. We are using what is conceptually a "simple in-memory dictionary," but its creation and session-based isolation are elegantly handled by the framework.

The interface for this state manager consists of two primary methods on the Context object (ctx):

await ctx.set_state(key: str, value: Any): This is how you "set a slot." It saves a value to the current session's state.
await ctx.get_state(key: str) -> Any: This is how you "read a slot." It retrieves a value from the current session's state. If the key doesn't exist, it returns None.

This dictionary-like get/set interface is intuitive and directly analogous to the slot-filling mechanisms you've used in Rasa Pro. By adopting this design, we don't need to write any code for managing the lifecycle of the state dictionary (lifespan) or worry about isolating data between users—fastapi-mcp does it for us.

Conclusion

In this lesson, we designed our state management system for remembering information between tool calls.

Key Takeaways:

Conversational AI requires session state to remember information for each user, a concept analogous to "slots."
While FastAPI provides lifespan events for managing global state, this is unsuitable for per-user conversational memory.
The fastapi-mcp library provides a built-in session state manager via the Context object, which is the correct design for our use case.
The state is managed through two simple async methods: ctx.set_state(key, value) and ctx.get_state(key).

We have now officially "designed" our state manager by selecting the appropriate tool for the job. We understand what it is, why we're using it, and what its interface looks like.

Next Up: In the next lesson, we will put this design into practice. We will modify our get_order_details tool to use ctx.set_state() to save the customer_id after fetching an order, effectively setting a slot.

Recommended resources