Interacting with the Environment: Agents, Tools, and APIs

Recommended resources

The freely available resources this lesson will direct you to.

Introduction

In our previous lesson, we established a clear architectural map of an AI agent, deconstructing it into four core components: the LLM backbone, the orchestration layer, tools, and memory. We emphasized the crucial separation of concerns: the LLM acts as the reasoning engine that decides on an action, while the orchestration layer is the code that actually executes that action.

This lesson drills down into the precise mechanism of that execution. Our learning outcome is to explain how agents invoke external tools and APIs to act on an environment. We will demystify the process, moving from the abstract concept of "tool use" to the concrete protocol of data exchange that makes it possible. You will learn how an agent communicates its capabilities to the LLM and how it interprets the LLM's instructions to interact with real-world systems like databases, APIs, or internal services. This is the bridge between the agent's "thinking" and "doing."

The Core Mechanism: A Protocol of Structured Data

A common misconception is that the LLM somehow directly runs code or calls a function. This is incorrect and, from a security and systems design perspective, would be highly problematic. The reality is more elegant and controlled. An LLM is a text-in, text-out system; its only ability is to generate a sequence of tokens. The "magic" of tool use lies in constraining the LLM to generate text in a highly structured format, typically JSON, which the orchestration layer can then parse and act upon.

The process is not an execution but a request. The LLM generates a data structure that says, "I have decided that the next logical step is to run this specific function with these specific arguments." The orchestration code then handles that request.

Function Calling in LLM Applications — This diagram illustrates the fundamental flow. A user's input is processed by the LLM, which, instead of answering directly, may decide to use a tool. It generates a structured "Function Call" request. This request is intercepted by the application's code (the orchestrator), which calls the actual external API or function. The result is then formatted and sent back to the LLM, which uses this new information to generate the final, context-aware response to the user.

Let's start by reinforcing this core concept, as it's fundamental to a rigorous understanding of agent architecture.

How does function calling with tools really work?

The video 'How does function calling with tools really work?' from Matt Williams directly addresses the misconception that LLMs call functions. This clip provides a clear, no-fluff explanation that aligns with your preference for precision.

Watch the segment from 00:52 to 03:42. The key takeaway is the explicit statement that function calling involves the LLM outputting a JSON blob that conforms to a predefined schema, not executing a function itself. This is a critical distinction for any architect.

The Tool-Calling Protocol: A Step-by-Step Breakdown

The interaction between the orchestration layer and the LLM follows a well-defined, multi-step protocol. Understanding this sequence is key to designing and debugging agentic systems.

Function calling using LLMs

Martin Fowler's article, 'Function calling using LLMs', provides an excellent, high-level overview of this process from a software engineering perspective. We will use it as a structural guide.

First, read the introductory section that starts with 'One of the key applications of LLMs...'. This will formally define function calling and reinforce the point that the LLM generates a structured JSON output for the program's runtime environment to execute.

The protocol can be broken down into four distinct steps:

Step 1: Define and Describe the Tools
The LLM cannot use tools it doesn't know exist. In each turn of the conversation, the orchestrator must provide the LLM with a list of available tools. This is done by passing a description of each tool in a specific format, typically a JSON Schema.

This schema defines:

name: The name of the function (e.g., get_stock_price).
description: A clear, natural language explanation of what the function does and when to use it. This is arguably the most important field, as the LLM uses this description to reason about the tool's utility.
parameters: An object describing the arguments the function accepts, including their names, types (string, integer, etc.), and whether they are required.

Function calling using LLMs

Let's return to the Martin Fowler article for a concrete example of these schemas.

Read the section that begins 'This part of the Chat Completions API call defines the available functions...' and review the SEARCH_SCHEMA, PRODUCT_DETAILS_SCHEMA, and CLARIFY_SCHEMA examples. Note how each schema clearly communicates the function's name, purpose, and required parameters.

Step 2: The LLM Decides to Use a Tool
The orchestrator sends the user's prompt, the conversation history, and the tool schemas to the LLM. The LLM then reasons over this context. If it determines that it needs external information or needs to perform an action to answer the user's request, it will generate a special response. Instead of a natural language reply, it outputs a JSON object indicating the name of the tool it wants to call and the arguments it has extracted from the user's query.

Step 3: The Orchestrator Executes the Tool
The orchestration code receives this structured response. It parses the JSON, identifies the requested function name (e.g., get_stock_price), and extracts the arguments (e.g., {'ticker': 'AAPL'}).

Critically, the orchestrator then calls the actual, executable function within its own environment, passing in the arguments provided by the LLM. This is the point where the agent interacts with the outside world—by making an API call, running a database query, or executing any other pre-defined piece of code.

Step 4: The Orchestrator Feeds the Result Back
The tool execution returns a result (e.g., the current stock price of Apple). The orchestrator takes this result, packages it into a special message (often with a role of tool or function), and appends it to the conversation history.

It then calls the LLM again, but this time with the original prompt, the LLM's decision to call the tool, and the result of that tool call. With this new information, the LLM can now generate a final, informed, natural language response to the user (e.g., "The current stock price for AAPL is $175.30.").

This complete loop is the engine of agentic action. The video below walks through this entire sequence with code.

LLM Function Calling - AI Tools Deep Dive

The video 'LLM Function Calling - AI Tools Deep Dive' provides a detailed walkthrough of this four-step loop using the OpenAI API.

Please watch the following segments: The Basic Sequence (00:50 - 03:12): This part outlines the exact four-step flow we just discussed. Defining the Schema (03:12 - 05:50): See how a tool is defined in code using the JSON schema format for an exchange rate function. The Full Loop in Action (05:50 - 12:53): This is the most important part. Watch how the code first calls the model, gets a tool_call response, parses it, executes the corresponding Python function, and then sends the result back to the model in a second call to get the final answer. Pay attention to how it handles multiple tool calls in one turn.

From Theory to Practice: Abstraction and Interoperability

As an architect, it's crucial to recognize that while the concept of tool calling is universal, the specific API syntax can differ between LLM providers (e.g., OpenAI, Anthropic, Google). Building a system directly against one provider's API creates vendor lock-in.

This is where abstraction layers, such as agentic frameworks like LangChain, become essential. They provide a standardized, provider-agnostic way to define and use tools. You define your tool once, and the framework handles the "translation" to the specific format required by whichever LLM backbone you choose to use.

LLM Function Calling - AI Tools Deep Dive

Let's return to the 'LLM Function Calling' video, which provides an excellent explanation of why and how to use LangChain for this purpose.

Watch the final part of the video on LangChain: The Need for a Standardized Framework (19:50 - 23:10): This explains the problem of differing provider syntaxes and introduces LangChain's standardized approach for defining tools using decorators or Pydantic classes. LangChain in Action (23:10 - 29:06): Observe how LangChain automates the multi-step protocol we've been studying. The .bind_tools() method and the agent executor abstract away the complexity of parsing tool calls and managing the conversation history, allowing you to focus on the business logic of your tools.

Using an abstraction framework is a key architectural decision that promotes flexibility, maintainability, and allows you to leverage the best model for a given task without rewriting your tool integration logic.

Conclusion

In this lesson, we have dissected the mechanism that allows an AI agent to act on its environment. You now have a precise, technical understanding of how tool invocation works, moving beyond metaphor to the level of a data-exchange protocol.

Key Takeaways:

Agents act on the world via tool calling (or "function calling"). The LLM does not execute code; it generates a structured JSON request that specifies a tool and its arguments.
This process follows a four-step protocol:
1. The orchestrator defines available tools using JSON schemas.
2. The LLM decides to use a tool and outputs a JSON request.
3. The orchestrator parses the JSON and executes the corresponding code.
4. The orchestrator feeds the result back to the LLM for a final response.
The quality of a tool's description in its schema is critical, as it's the primary information the LLM uses for reasoning.
Architectural best practice involves using abstraction frameworks like LangChain to create a provider-agnostic tool-use layer, avoiding vendor lock-in.

Preview of the Next Lesson:

We now understand how an agent can use a single tool. But how does it decide which tool to use when multiple are available? And how does it handle complex tasks that require a sequence of tool calls? This leads directly to our next topic: reasoning patterns. We will explore techniques like Chain-of-Thought and ReAct (Reason+Act) that enable LLMs to formulate and execute multi-step plans.