Recommended resources
The freely available resources this lesson will direct you to.
Hello! In our last lesson, we successfully decoupled our AI system by implementing the producer-consumer pattern. You learned how to enqueue a long-running agentic task and have a worker execute it in the background, all without blocking the user. This architecture returns a task_id to the user, which is a promise of a future result.
Today, we'll fulfill that promise. We're going to tackle the critical question: "What happens to the task after the worker is done?" This lesson focuses on managing the lifecycle of that task—its status, its intermediate steps, and its final result. We will explore how to use persistent stores like Redis and Postgres to build a robust state management system for our asynchronous workflows. This is a cornerstone of production-grade AI engineering, transforming a simple fire-and-forget task into a trackable, resumable, and auditable process.
The Problem: Stateless Workers and the Need for Memory
In the architecture we built, the web server (producer) and the worker (consumer) are separate, often stateless processes. The message queue connecting them is a brilliant buffer, but it's not a database. If a worker process crashes mid-task, the state of its work is lost. If the user wants to check the status of their task, how do we provide an answer?
This is where a dedicated, persistent state store comes in. It acts as the "single source of truth" for every task in the system.
An asynchronous task isn't just a single state; it's a state machine. It transitions from QUEUED, to IN_PROGRESS, to COMPLETED or FAILED. For complex agents, it might have many more intermediate steps. Our state store needs to track this.
Choosing Your State Store: Redis vs. Postgres
The choice of database for state management is a classic system design problem with significant trade-offs. For our purposes, the most common and effective choices are Redis and Postgres. An excellent article on SitePoint breaks down this decision by categorizing agent memory.
Agent State Management: Redis vs Postgres for AI Memory
This article introduces the core problem and defines different types of agent memory, which helps clarify the roles of different databases.
Please read the section The "Context Window Isn't Memory" Problem. Focus on the definitions of Working memory, Episodic memory, and Semantic memory.
As the article explains, these different memory types have different requirements for latency, durability, and cost. This is why a one-size-fits-all approach is often suboptimal.
- Redis is the champion of working memory. It's an in-memory data store, which means it's incredibly fast for reading and writing the current state of a task.
- Postgres is the bedrock for episodic and semantic memory. It's a relational, disk-based database that provides full ACID durability and powerful querying, making it perfect for storing final results, audit logs, and long-term knowledge.
Let's see how this plays out in practice.
The Polling Pattern with Redis
A common and effective pattern for managing task state is to use Redis to store the status and results, which a client can then poll via an API endpoint.
A video by "Code with Irtiza" provides a fantastic walkthrough of this exact pattern. While it's in Python, the architectural concepts are universal, and it will reinforce your understanding of how the pieces fit together.
Asynchronous Python LLM APIs | FastAPI, Redis, AsyncIO
This video demonstrates building an asynchronous API with FastAPI and Redis to handle long-running LLM tasks.
Please watch the following segments to see how Redis acts as the state manager: The User Experience: First, watch the demonstration from the beginning of the demo. Notice how submitting multiple prompts returns a list of task_ids, and a separate GET endpoint is used to poll for their status and results. This is the exact user-facing flow we're enabling. Storing Initial State: Next, see how the state is initialized in Redis. The segment from this section shows the code that creates a Redis Hash for each task_id with an initial status of "queued" and progress "0". Updating State from the Worker: Now, see the worker in action. The segment from here explains the process_llm_task function. This is our "consumer." Notice how it updates the status in Redis to "processing," updates the progress, and finally writes the "result" and "completed" status to the same Redis key. It also sets a Time-To-Live (TTL) on the result. Retrieving State: Finally, watch the explanation of the GET endpoint from this section. It's very simple: it takes the task_ids, queries Redis for the corresponding keys, and returns the stored state. This video perfectly illustrates using Redis for managing the working memory of a task—its current status, progress, and final result, available for quick retrieval. The Hybrid Architecture: The Best of Both Worlds Using Redis alone is great for speed but has drawbacks for long-term storage. Redis is optimized for memory, not disk, and its durability guarantees are configurable but often weaker than a traditional database. Full conversation histories or audit logs can become expensive to store in Redis at scale. This is where the hybrid architecture comes in, combining the strengths of Redis and Postgres.
This video perfectly illustrates using Redis for managing the working memory of a task—its current status, progress, and final result, available for quick retrieval.
The Hybrid Architecture: The Best of Both Worlds
Using Redis alone is great for speed but has drawbacks for long-term storage. Redis is optimized for memory, not disk, and its durability guarantees are configurable but often weaker than a traditional database. Full conversation histories or audit logs can become expensive to store in Redis at scale.
This is where the hybrid architecture comes in, combining the strengths of Redis and Postgres.
Agent State Management: Redis vs Postgres for AI Memory
This article details the hybrid pattern, which is a robust solution for production systems.
Please read the following sections to understand this powerful architecture: Component Roles: First, review the sections Redis as Agent Working Memory and Postgres as Agent Episodic and Semantic Memory. This will solidify your understanding of what each database is best at. The Hybrid Pattern: Now, read the section The Hybrid Architecture. Focus on the "Write-Through with Async Consolidation" data flow. This is a key system design pattern. Implementation Abstraction: Finally, study the Python code for the AgentMemoryBridge class. You don't need to memorize the syntax, but understand the purpose of each method: save_working_state: Writes the current, "hot" state to Redis. This is what the worker calls frequently. consolidate_to_long_term: A separate process calls this to flush completed task data from Redis to Postgres for permanent storage. hydrate_agent: When an agent starts, it uses this to load its state, pulling hot context from Redis and relevant long-term memories from Postgres.
This hybrid model gives you the sub-millisecond latency of Redis for in-progress tasks while providing the durable, queryable, and cost-effective long-term storage of Postgres for completed tasks and historical context.
Orchestrating State with Temporal
For highly complex, multi-step agentic workflows, managing state manually with status flags in a database can become cumbersome. This is where a workflow orchestration engine like Temporal shines. Temporal provides a developer-centric way to write distributed systems without having to implement the plumbing for state management, retries, and fault tolerance yourself.
In the Temporal model, the state of the workflow is preserved automatically as part of the workflow's execution history.
Deep-Dive: AI Agent Code Walkthrough with Temporal
This deep-dive by Temporal provides a walkthrough of an AI agent built on their platform, revealing how it handles state.
Please watch two key segments: State as Code: Watch from this point, where the presenter shows a basic chat loop. Notice how conversation_history is just an instance variable in the workflow class. Temporal's platform ensures that the value of this variable is durable and survives any process restarts. The workflow code itself is the state machine. Scaling with External Stores: Storing large amounts of data directly in the workflow history is not scalable. Watch the Q&A segment from here. The presenter explains that for production scale, you would store large data blobs (like chat history) in an external store like Redis or S3 and only keep a reference or pointer in the workflow history. This elegantly connects back to the hybrid pattern we just discussed!
Temporal represents a higher-level abstraction. You still need persistent stores like Redis and Postgres, but Temporal manages the logic of how and when state is saved and loaded, allowing you to focus on the business logic of your agent.
The Interview Perspective
Your ability to discuss these trade-offs is crucial for AI Engineer interviews. Let's look at a typical question.
10 Essential Agentic AI Interview Questions for AI Engineers
This article from KDnuggets lists common interview questions for this role.
Read question #3: "How Do You Handle State Management in Long-Running Agentic Workflows?" and pay close attention to the "Answer along these lines" part.
Notice how the ideal answer includes everything we've covered: explicit state storage, checkpointing (saving state), managing short-term vs. long-term memory, and ensuring recoverability. You are now equipped to give a detailed, expert-level answer drawing on the relative strengths of Redis, Postgres, and orchestration engines like Temporal.
Conclusion
You have now bridged the gap between initiating an asynchronous task and delivering its result. Managing state is what makes a distributed system robust, observable, and resilient.
Key Takeaways:
- Persistence is Non-Negotiable: Asynchronous workflows require a persistent store to track status, store results, and enable failure recovery.
- Use the Right Tool for the Job:
- Redis is ideal for "working memory"—the fast-changing, hot state of currently active tasks.
- Postgres is the foundation for "episodic memory"—the durable, auditable, long-term history of completed tasks.
- The Hybrid Model is a Production Standard: Combining Redis for latency-sensitive state and Postgres for durable storage offers a scalable and cost-effective solution.
- Orchestration Engines Abstract Complexity: Tools like Temporal manage state persistence and fault tolerance for you, allowing you to build complex, multi-step workflows with greater confidence.
You have now built a complete, scalable, and stateful system for running agentic tasks. But how do you know if it's working well? How do you track costs, latency, and quality? In our next lesson, we will enter the world of LLMOps, learning how to integrate monitoring platforms to gain deep visibility into our AI system's performance.