LLMs and External Tools: Selection, Calling, and Interpretation

Welcome to your fifth lesson on the foundations of LLM agents. In our previous lesson, we explored a variety of architectural patterns for agents, from the adaptive ReAct loop to the more structured Plan-and-Execute approach. We established that regardless of the high-level strategy, the agent's ability to interact with the outside world is fundamental. All these planning patterns ultimately rely on a core mechanism: tool use.

Today, we will dissect that mechanism. This lesson fulfills your goal of understanding the fundamental building blocks of agentic systems by explaining precisely how a large language model selects, calls, and interprets the results from external functions, often called tools. You'll learn how an LLM moves from understanding natural language intent to triggering a programmatic action and incorporating its outcome.

The Tool-Calling Lifecycle: A Four-Step Process

At its heart, enabling an LLM to use a tool is about bridging the gap between probabilistic text generation and deterministic code execution. This process can be broken down into a reliable, four-step cycle that forms the basis of almost all agentic tool use.

The following diagram illustrates this core loop. A user's query comes in, the LLM decides a tool is needed, your application code invokes that tool externally, the tool's output is returned, and finally, the LLM processes that output to formulate a response.

A visual representation of the tool-calling loop: (1) A user provides input, (2) the LLM outputs a tool invocation, (3) the tool is executed externally, and (4) the tool's output is fed back to the LLM to generate a final answer.

Let's explore the conceptual architecture that makes this loop possible. The video from IBM Technology provides a great high-level overview.

AI Tool Calling via Natural Language: LLMs, APIs & Docker in Action

This video frames tool calling as a system architecture, which aligns well with your goal of building production-grade applications. It clearly separates the distinct responsibilities within the process.

Watch from the first step, where the model detects that a tool is needed. Then, continue through the explanations of generating a structured call, executing the call in isolation, and finally re-inserting the result into the conversation.

As you saw, the four key steps are:

Detect: The LLM recognizes that the user's request cannot be fulfilled with its internal knowledge alone and requires an external action.
Generate: The LLM creates a structured, machine-readable request to call a specific tool with the necessary arguments.
Execute: Your application code receives this structured request, runs the corresponding function, and captures the output. This step happens outside the LLM.
Interpret: The tool's output is passed back to the LLM as new information, which it then uses to synthesize a final, user-facing response.

Now, let's dive into the technical details of how each step is implemented.

Step 1 & 2: Tool Selection and Structured Call Generation

This is arguably the most "magical" part of the process. How does an LLM, trained on text, know to call your specific get_weather function and correctly extract "San Francisco" as the location parameter?

The answer lies in how we describe the tools to the model. When we make a set of tools available, we provide the LLM with a manifest—essentially a menu—detailing each tool's:

Name: A unique identifier, like get_weather.
Description: A clear, natural language explanation of what the tool does, when to use it, and what it's for. This is the most critical piece. The LLM performs a semantic search over these descriptions to find the best tool for the user's query. A good description is the difference between a tool that is used correctly and one that is ignored.
Parameters Schema: A structured definition of the arguments the function accepts, including their names (e.g., location), types (string), and whether they are required. This is often defined using JSON Schema or, more conveniently, through Python constructs like Pydantic models or type-hinted function signatures.

When the LLM receives a prompt, it considers both the prompt and the provided tool descriptions. If it determines a tool is needed, it doesn't execute anything. Instead, its output is a structured JSON object specifying the chosen tool's name and the arguments it has extracted from the prompt.

The video "LLM Function Calling - AI Tools Deep Dive" provides an excellent, code-level walkthrough of this entire process.

LLM Function Calling - AI Tools Deep Dive

This video walks through the full end-to-end process with clear code examples, starting with the raw OpenAI API and then showing how frameworks like LangChain simplify it.

First, watch the introduction from the beginning to get a solid definition of function calling and an overview of the flow. Next, focus on the section from defining tools to generating arguments. Pay close attention to how a Python function is described using a JSON schema, especially the description and parameters fields. Notice how the LLM's output is not an answer, but a JSON object representing a function call request.

Step 3 & 4: Execution and Interpretation

Once the LLM has generated the tool call request, your application takes over. This is a crucial point: the LLM does not run the code. Your Python application is responsible for:

Parsing the LLM's response: Check if the response contains a tool call request.
Routing the call: Based on the name of the tool in the request, call the corresponding Python function in your codebase.
Executing the function: Pass the arguments from the LLM's request to your Python function and execute it.
Capturing the result: Store the return value of the function.
Formatting the feedback: Package the result into a special message (e.g., a ToolMessage) that includes the original tool call ID and the output.
Calling the LLM again: Make a second call to the LLM, this time including the original prompt, the LLM's initial tool-call request, and the new ToolMessage containing the result.

Now, with the full context, the LLM can "interpret" the tool's output and generate a natural language response that incorporates the new information.

Let's continue with the same video to see this second half of the loop in action.

LLM Function Calling - AI Tools Deep Dive

This section completes the loop, showing how the application code executes the function and feeds the result back to the model.

Watch the segment from executing functions and returning context. This part clearly demonstrates the multi-step process of getting the initial tool call from the LLM, executing it in Python, and then sending the result back to get the final, polished answer.

Standardizing Tool Calling with LangChain

As you just saw, the raw process involves a fair bit of manual JSON formatting and API-specific logic. Furthermore, each major LLM provider (OpenAI, Anthropic, Google) has its own subtle variations in how they expect tool schemas to be formatted and how they return tool-call requests. This creates friction when you want to switch models or build a system that can leverage multiple providers—a key consideration for cost and performance optimization in a production environment.

This is where a framework like LangChain becomes invaluable. LangChain provides a standardized, model-agnostic interface for tool calling. You define your tools once, using convenient Python constructs, and LangChain handles the complexities of formatting them correctly for whichever backend LLM you're using.

Tool Calling with LangChain

This blog post from the LangChain team introduces their standardized interface for tool calling, which directly addresses the problem of provider-specific implementations. It's a great example of how a framework can help you write more portable and maintainable agentic code.

Start by reading the introduction and the section on provider incompatibilities to understand the problem LangChain is solving. Next, read the section on ChatModel.bind_tools. Pay attention to the different ways you can define a tool: as a Pydantic class, with the @tool decorator on a function, or as a raw dictionary. The decorator method, combined with type hints and a docstring, is particularly powerful and Pythonic. Finally, read the section on AIMessage.tool_calls. This explains the standardized output format. Instead of parsing provider-specific additional_kwargs, you can now reliably find all tool call requests in the .tool_calls attribute of the message.

By using llm.bind_tools() and reading from AIMessage.tool_calls, your code becomes cleaner and portable across different models, a significant advantage for building production-ready agents.

For more complex tools with many parameters, you can use Pydantic models to define the schema, which provides robust validation and autocompletion in your IDE. This is a common pattern in production code.

Bonus: Hierarchical Tool Selection

For agents with a very large number of tools, a simple linear search through all tool descriptions can become inefficient and less accurate. A more advanced technique is hierarchical tool selection.

In hierarchical selection, the agent first chooses a general category of tools (e.g., "Compute") before selecting the specific tool (e.g., a calculator), narrowing down the search space.

In this approach, the agent first makes a high-level decision to select a "tool group" (e.g., "Computation Tools," "Database Tools," "Communication Tools"). Once a group is selected, it then performs a second selection to choose the most relevant tool from within that smaller, more focused set. This two-step process can improve both speed and accuracy for complex agents.

Conclusion

In this lesson, we have demystified the process of LLM tool calling. You've moved from the high-level "what" of agent planning to the low-level "how" of agent action.

Here are the key takeaways:

The Four-Step Cycle: Tool use follows a detect, generate, execute, and interpret loop.
Descriptions are Key: The LLM selects tools based on a semantic understanding of their natural language descriptions. Clear, concise descriptions are crucial for reliable tool selection.
The LLM Generates, You Execute: An LLM's role is to produce a structured request to call a tool. Your application code is responsible for the actual execution and for feeding the results back.
Frameworks Provide Standardization: Tools like LangChain abstract away provider-specific APIs, allowing you to write cleaner, more portable code with methods like bind_tools and a standard AIMessage.tool_calls output format.

You now have a solid grasp of the mechanics that empower an LLM to act on the world. However, this power comes with risks. What happens when the LLM hallucinates a tool that doesn't exist? Or gets stuck in a loop calling the same tool over and over? In our next lesson, we will explore these common agent failure modes and begin discussing how to build more robust and reliable systems.