Hello and welcome back!
In our last lesson, we saw how the ReAct pattern orchestrates the agent's core components—the agent loop, tool use, and memory—to solve multi-step tasks. We saw a conceptual blueprint for how an agent works.
Now, we'll shift from the "what" to the "how" and, more importantly, the "how hard." As a software engineer, you know there's a world of difference between a design diagram and a production-ready system. Today, we'll explore the engineering realities of building an agent from scratch. This lesson addresses the fundamental "why" behind Agent Development Kits (ADKs), which is the focus of this module.
By the end of this lesson, you will be able to identify the key challenges in building agents from scratch, specifically focusing on orchestration, error handling, and state management.
The Engineering Behind the Agent Loop
In previous lessons, we've used diagrams to represent the agent architecture. Let's look at one again, but this time, view it through an implementer's lens.
Implementing this system requires you to build the "plumbing" that manages the flow of information, maintains the agent's memory, and handles the inevitable cascade of things that can go wrong. Let's break down the three primary categories of these challenges.
1. The Orchestration Challenge: Managing the Flow
Orchestration is the process of managing the agent's execution loop: parsing the LLM's output, deciding what to do next, calling tools, and feeding the results back to the LLM. When building from scratch, you quickly find yourself writing a significant amount of "boilerplate" code to manage this flow.
To see what this looks like in practice, we're going to watch a developer build a simple agent without using any frameworks.
Intro to Agents - Create an Agent from Scratch (No Frameworks)
The video 'Intro to Agents - Create an Agent from Scratch (No Frameworks)' by Alejandro AO demonstrates the raw implementation of a basic agent. It provides a clear view of the code required to get a simple agent running.
Watch the segment from 00:18:06 to 00:31:26. As you watch, pay attention to the execute method within the Agent class. Notice the manual, low-level tasks the developer has to implement: The Main Loop: A while True loop is used to keep the agent running until it produces a final answer. State Manipulation: The messages list (the agent's memory) is manually appended with user messages, tool calls, and tool results. Output Parsing: The code must check if tool_calls exist in the LLM's response. Tool Dispatching: The code dynamically finds and calls the correct Python function based on the function name returned by the LLM. Result Formatting: The output from the tool must be formatted into a specific message structure, including the tool_call_id, before being added back to the history.
As you saw in the video, even a simple agent requires a non-trivial orchestration engine. The code has to meticulously manage the conversation history, parse structured data from the LLM, dispatch function calls, and handle the response loop. This logic is common to nearly all agents. Writing and maintaining this for every agent you build is inefficient and error-prone. This is the core "boilerplate" problem that ADKs are designed to solve.
As the creator of the video himself notes at the end, this approach is not robust enough for production. One of the key missing pieces is robust state management.
2. The State Management Challenge: The Problem of Memory
In the agent we just saw, "state" was just a simple list of messages held in memory. This works for a single, short-lived interaction. But what happens in a real-world application?
An agent needs to remember context:
- Within a long conversation.
- Across multiple, separate user sessions.
- After a system crash or restart.
- When handing off a task to another agent.
Your first instinct might be to simply feed the entire history back to the LLM in its context window. However, this strategy quickly fails in production.
State Management in Multi-Agent AI Systems
The article 'Building Agents That Remember: State Management in Multi-Agent AI Systems' by Ranjan Kumar provides an excellent breakdown of why agent memory is a difficult engineering problem.
Read the sections '1. The Memory Problem Nobody Wants to Talk About' and '2. The Context Window Trap'. Focus on the three reasons why relying solely on the context window is not a scalable solution.
As the article explains, relying on the context window as your sole memory system is a trap due to:
- Finite Capacity: Even multi-million token context windows can be exhausted by long conversations, complex tool outputs, and retrieved documents.
- Context Rot & "Lost in the Middle": LLM performance degrades as the context window fills. Information buried in the middle of a long prompt is often ignored or misremembered.
- Exploding Cost and Latency: Because LLMs are stateless, you pay to re-process the entire history with every single turn. This makes the agent progressively slower and more expensive as the conversation continues.
Solving this "from scratch" means you are now in the business of designing a memory architecture. You need to decide on strategies for summarizing conversations, storing state in an external database (like Redis or Postgres), and implementing retrieval mechanisms to fetch only the most relevant information for the current task. This is a significant architectural undertaking that goes far beyond simple prompt engineering.
3. The Error Handling & Reliability Challenge: The "Stochastic Fog"
Traditional software is deterministic: the same input produces the same output, making debugging a predictable process. Agents are fundamentally different. They are stochastic, or non-deterministic. Running the same query twice can produce two different reasoning paths, one succeeding and one failing.
This introduces a whole new class of challenges related to reliability and debugging.
The article 'Production Hell of AI Agents' vividly describes the practical difficulties of making agents reliable. It contrasts the clean world of demos with the messy reality of production.
Please read three sections from this article: Start with 'Debugging AI Agents: Tracing Failures Through Stochastic Fog'. This section explains why traditional debugging methods don't work for agents. Next, read 'Hitting AI Agent Reliability Ceilings: When Brittleness Wins'. This discusses the inherent unreliability of agents and the edge cases they struggle with. Finally, read 'Integration Labyrinth: Connecting AI Agents to Real-World Systems'. This details the immense complexity of tool integration.
From these readings, we can distill several critical reliability challenges you must solve when building from scratch:
- LLM Failures: The LLM might generate a response that isn't valid JSON, hallucinate a parameter for a tool, or simply fail to follow the correct reasoning path. Your code needs to be able to catch these errors, potentially retry the request, or fail gracefully.
- Tool Failures: The external tools and APIs your agent relies on can fail. They can time out, return error codes, or be temporarily unavailable. A production agent needs robust error handling for every tool it uses, including mechanisms like retries with exponential backoff.
- Integration Complexity: As the article highlights, connecting to real-world systems is a "labyrinth." You are responsible for handling authentication (e.g., OAuth), rate limits, API version changes, and parsing inconsistent response formats. This integration "plumbing" often becomes a larger and more complex part of the codebase than the agent logic itself.
- The Reliability Ceiling: Even with extensive prompt tuning and error handling, agents often plateau at a certain success rate (e.g., 80-90%). The remaining failures are often subtle edge cases that are difficult to predict and fix, making it challenging to use agents for mission-critical, high-stakes tasks without a human in the loop.
Test your understanding!
Imagine you're building an agent from scratch that uses a third-party API to book flights. The API call to book_flight occasionally fails with a 503 Service Unavailable error, which usually resolves in a few seconds.
Without an ADK, what specific error-handling logic would you need to implement in your orchestration code to make this tool call more reliable?
Show answer
You would need to wrap the book_flight API call in a try-except block. Inside the except block, you'd check for the specific 503 error. If detected, you would implement a retry mechanism. A robust implementation wouldn't just retry immediately; it would use a strategy like exponential backoff, where it waits for an increasing amount of time between retries (e.g., 1s, 2s, 4s, 8s) to avoid overwhelming the service. You would also need to set a maximum number of retries to prevent an infinite loop. If all retries fail, the agent must have a way to handle this final failure—perhaps by informing the user or trying an alternative approach. All of this logic would have to be written manually.
Conclusion
Building a basic agent demo is straightforward. Building a robust, reliable, and scalable agent is a significant software engineering challenge.
Today, we've focused on three of the most immediate and critical challenges you'll encounter:
- Orchestration: Requires writing and maintaining significant boilerplate code to manage the agent's execution loop, parse LLM outputs, and dispatch tool calls.
- State Management: Relying on the context window is not a scalable solution. Building a proper memory system requires designing a complex architecture for storing, retrieving, and summarizing information, often involving external databases.
- Error Handling & Reliability: Agents are non-deterministic and interact with fallible external systems. This demands custom, complex error-handling logic for everything from parsing LLM outputs to managing API failures.
These challenges are not unique to specific tasks; they are universal problems in agent development. This is precisely why Agent Development Kits exist.
In our next lesson, we will explore how Agent Development Kits (ADKs) abstract these common patterns. We'll see how they provide pre-built components for orchestration, state management, and tool integration, allowing you to focus on the unique logic of your agent rather than reinventing the wheel.