Welcome to the first lesson of your course on building AI agents. Your goal is to gain a deep, technical understanding of agent design patterns, particularly for creating sophisticated AI tutors. Given your background in programming and system architecture, we will dispense with the basics and dive directly into the high-level concepts that define modern agentic systems.
This initial lesson focuses on establishing a clear taxonomy. We will distinguish between simple Large Language Model (LLM) calls, deterministic agent-like "workflows," dynamic "agent" architectures where the LLM acts as a controller, and the traditional Reinforcement Learning (RL) agents from which they descend. Understanding these distinctions is fundamental, as the choice of architecture has profound implications for a system's capabilities, reliability, and cost—concerns that are central to production engineering.
The Spectrum of Agentic Systems: From Simple Calls to Autonomous Agents
At the most basic level, interacting with an LLM involves a simple, stateless call: you provide a prompt, and the model generates a text completion. While powerful, this is not an agent. Agency emerges when a system can perform a sequence of actions, perceive the results, and decide what to do next to achieve a goal. However, not all systems that do this are architected in the same way.
A critical distinction, popularized by Anthropic, is between "workflows" and "agents." A video from LangChain, based on this distinction, provides an excellent overview.
Building Effective Agents with LangGraph
Please watch this introductory segment from the LangChain video, "Building Effective Agents with LangGraph." It clearly defines the difference between a workflow, where an LLM is embedded within predefined code paths, and an agent, where the LLM itself directs the action.
Focus on the explanation from workflow versus agent. The speaker uses the analogy of "scaffolding" for workflows, which is a useful mental model.
To elaborate on this, let's explore the definitions and patterns provided in Anthropic's blog post, which the video is based on.
This post provides a practical, industry-standard taxonomy of agentic systems. It's a valuable guide to thinking about how to structure applications that go beyond single LLM calls.
First, read the section What are agents? to solidify the distinction between workflows (predefined code paths) and agents (LLM-directed processes). Then, in the "Building blocks, workflows, and agents" section, review the descriptions of the various workflow patterns, such as prompt chaining, routing, and orchestrator-workers. Notice that in all these cases, the overall logic is determined by external code, even if an LLM is used to make a routing decision at a specific branch point.
In essence:
- Simple LLM Calls: The foundational block.
prompt -> LLM -> completion. - Deterministic Workflows (or Pipelines): An external program orchestrates a series of LLM calls and other operations (e.g., tool use). The control flow is rigid and pre-scripted. This is akin to a traditional program that happens to use an LLM as a function for certain steps. These systems are more predictable and reliable but less flexible.
- Controller-based Agents: The LLM itself functions as the central control loop. It receives a high-level goal, reasons about how to achieve it, decides which tools to use, executes them, observes the outcome, and then decides the next step. This process repeats until the goal is met. The control flow is dynamic and determined by the model's reasoning at runtime.
The following diagram from Anthropic illustrates the dynamic, multi-turn flow of a controller-based agent designed for a coding task.
This sequence diagram shows an agentic workflow. A human provides a query, which the LLM iteratively clarifies. The LLM then enters a loop: it interacts with the environment (e.g., searching files, writing and testing code), receives results, and uses this feedback to inform its next action until the task is complete.
This loop—often summarized as Perceive → Reason → Act → Observe—is the defining characteristic of a controller-based agent architecture. The autonomy to decide the next action based on environmental feedback is what separates it from a deterministic workflow.
Let's watch two more clips from the LangChain and AI Engineer videos that further clarify this agent loop and its contrast with workflows.
Building Effective Agents with LangGraph
This segment shows how a simple agent is constructed as a loop and explicitly discusses when you might choose this architecture over a more structured workflow.
Watch the section from now let's remove the scaffolding to understand the agent's unbounded nature and the types of open-ended problems it's suited for.
Reinforcement Learning for Agents - Will Brown, ML Researcher at Morgan Stanley
This clip from the "Reinforcement Learning for Agents" talk provides another perspective, distinguishing between "pipelines" (workflows) and more autonomous agents, and situates them in a broader landscape of LLM capabilities.
Watch the segment from distinguish between agents and pipelines, where the speaker discusses the low degree of autonomy in pipelines and contrasts it with systems designed for longer, more complex tasks.
A Formal Framework: The Markov Decision Process (MDP)
The concept of an agent interacting with an environment is not new; it is the cornerstone of Reinforcement Learning. We can formalize the architecture of an LLM agent using the mathematical framework of a Markov Decision Process (MDP). This provides a precise language to distinguish it from a simple LLM call and to connect it to traditional RL agents.
An MDP is defined by a tuple , representing the State Space, Action Space, Transition Probability, Reward Function, and a discount factor. The "Agent-R1" paper offers a superb technical breakdown of how these components are adapted when moving from a static LLM to a dynamic LLM agent.
Agent-R1: Training Powerful LLM Agents with End-to-End Reinforcement Learning
This section of the paper formalizes the architectural shift from a static text generator to an interactive agent using the language of MDPs. This is a critical perspective for understanding agent behavior at a fundamental level.
Read all of Section 2, "From Large Language Models to Agents: An MDP Perspective". Pay close attention to the tables and the detailed explanations for each component: State Space (\mathcal{S}): Notice how the state evolves from just the text generated so far (static LLM state) to a comprehensive history including multi-turn interactions and environmental feedback (LLM agent state). Action Space (\mathcal{A}): Understand that while the fundamental action is still token generation, for an agent, specific token sequences are interpreted as commands. State Transition (\mathcal{P}): This is a key differentiator. The transition is no longer deterministic. It can be stochastic due to environmental interaction, as captured by the distinction between a generative transition (P_G) and an environmental transition (P_E). Reward Function (\mathcal{R}): Grasp the shift from a single, sparse reward for the final output to a richer structure that can include dense, intermediate process rewards for successful steps.
Connecting to Traditional Reinforcement Learning
So, how does this relate to traditional RL agents, like those used to master games like Go or control robotic arms?
A traditional RL agent is a learning entity that aims to find an optimal policy, , which dictates the best action to take in a given state to maximize its cumulative future reward. It learns this policy through trial-and-error interaction with its environment.
An LLM-based agent is a modern, powerful instantiation of this classical paradigm:
- The LLM itself is the policy . Given the state (the context, including history and observations), the LLM generates the next action (the response, which might be a tool call).
- The environment is everything external to the LLM: the user, the tools it can call, the files it can access, etc.
- The learning happens when the LLM's parameters are updated to favor actions that lead to higher rewards. This is exactly what happens during Reinforcement Learning from Human Feedback (RLHF) or more advanced end-to-end RL training schemes, which are used to produce models that are better at following instructions and using tools.
The following clip provides context on how RL is the mechanism that "unlocks" more complex, agentic behavior in LLMs.
Reinforcement Learning for Agents - Will Brown, ML Researcher at Morgan Stanley
This final set of clips connects the concept of an agent back to its roots in RL and explains how RL is used to train the very capabilities that enable agency.
First, watch the brief segment on the traditional definition of an RL agent. Then, see how this learning principle is applied to LLMs to teach them complex reasoning and problem-solving strategies, as discussed from the world of reinforcement learning. Finally, the speaker introduces a framework for doing RL in multi-step environments from what this really is, which is precisely the setting for a controller-based agent.
Summary of Distinctions
To consolidate, here is the hierarchy of systems we've discussed:
| System Type | Control Flow | Autonomy | Key Characteristic |
|---|---|---|---|
| Simple LLM Call | Single, stateless request-response. | None | A function that maps text to text. |
| Deterministic Workflow | Pre-scripted, defined by external code. | Low | LLM calls are embedded in a fixed logic graph. Predictable but inflexible. |
| Controller-based Agent | Dynamic, directed by the LLM itself. | Medium to High | The LLM operates in a loop, deciding its own next actions based on observations. |
| Traditional RL Agent | Learned policy . | Varies (Learning System) | A formal framework for learning optimal behavior through trial and error. LLM agents are an instance of this. |
Conclusion
In this lesson, we have established a foundational taxonomy for understanding systems that utilize LLMs. We've distinguished between simple LLM calls, deterministic workflows where control is external, and controller-based agents where the LLM itself directs the process dynamically. By framing this within the formal language of Markov Decision Processes, we've connected modern LLM agents to their theoretical roots in traditional Reinforcement Learning, where the LLM acts as a learned policy.
This architectural understanding is crucial. As we proceed, you will see that the most capable and interesting systems, including the AI tutors that are your ultimate focus, are built upon these controller-based agent patterns.
In our next lesson, we will build on this foundation by classifying agents along several key dimensions, such as their degree of autonomy, their access to tools, their planning horizon, and whether they operate alone or in groups. We will also dissect the canonical agent loop in greater detail, identifying the core components—the LLM, controller, tools, and memory—that you will learn to design and implement.