Structured Output for Consistent Agents

Introduction

In our last lesson, we focused on designing fallback and escalation behaviors to manage agent failures. We established that a robust system must gracefully handle unexpected events, including when an agent produces a malformed or syntactically incorrect output. The self-correction loop we discussed is a powerful reactive pattern for this.

Today, we shift from reaction to prevention. Why build complex loops to handle broken JSON when you can architect the system to guarantee the JSON is always valid? This lesson directly addresses that question. Our learning outcome is to apply structured output formatting (e.g., JSON schemas) to enforce consistent agent responses.

This is a cornerstone of building reliable agentic systems. When an agent's output is guaranteed to be machine-readable and conform to a predefined data contract, you unlock the ability to:

• Reliably call tools and APIs without parsing errors.
• Chain agents together in predictable workflows.
• Store agent outputs in structured databases for analysis.
• Build user interfaces that can confidently bind to agent data.

We will explore the underlying theory of how this enforcement works and then translate that theory into practical design patterns for building your agentic solutions.

1. From Unpredictable Text to Guaranteed Structure

At their core, Large Language Models are text generators. Without constraints, they will often wrap their output in helpful but programmatically useless explanatory text. For an agentic system that needs to parse a response, this is a critical failure point.

The technique that makes this possible is called constrained decoding. As a computer scientist, you can think of it as changing the model's token generation process from a free-form selection to one that is filtered by a set of rules.

This is more than just a convenience. As you design systems for clients, the distinction between hoping for valid JSON and guaranteeing it is the difference between a fragile prototype and a production-ready service.

Structured Output, Function Calling, or JSON Mode?

Modern APIs offer several ways to get structured data. A key architectural decision is choosing the right tool for the job.

To summarize the key design choice:

• Use Function Calling when you need the model to select one or more tools and generate the specific arguments to call them. This is for when the agent is acting.
• Use Structured Output when you need to enforce the format of the final response to a user, or, critically for our purposes, when defining the data contract for internal routing and communication between agents in a multi-agent system.

2. The Core Mechanism: Automata-Guided Generation

How does an LLM, a probabilistic model, provide a deterministic guarantee of structure? The answer lies in applying principles from formal language theory directly to the token generation process. Given your background, the underlying mechanics will be particularly interesting.

The process generally involves converting the desired schema (e.g., a JSON schema) into a regular expression, which is then used to construct a Finite State Machine (FSM). This FSM guides the LLM at every step.

The following diagram provides a fantastic visual summary of this token masking process. The FSM state dictates which tokens are "allowed," their probabilities are renormalized, and the model samples only from that valid subset.

For a more formal treatment, we can express this mathematically.

This mechanism is now being integrated into high-performance serving frameworks like vLLM, demonstrating its importance for production systems.

It's worth noting that while FSMs (recognizing regular languages) can handle most practical JSON needs, more complex nested structures theoretically require context-free grammars (CFGs). This introduces challenges like the "token-terminal mismatch problem," where LLM tokens don't align with the grammar's terminal symbols. This is an active area of research, but for most agentic design, FSM-based approaches are sufficient.

3. Application and System Design Patterns

With a solid grasp of the theory, let's turn to practical application. How do you, as a system architect, use these capabilities to build robust agentic workflows?

Pydantic: The Industry Standard for Schema Definition

While you can write raw JSON schemas, the overwhelming best practice in the Python ecosystem is to use Pydantic. Pydantic models are Python classes that provide data validation and schema definition in a clean, readable, and maintainable way.

Architectural Patterns for Structured Output

Adopting structured output is not just an implementation detail; it enables new architectural patterns for building reliable systems.

Case Study: Inter-Agent Handoffs

Perhaps the most powerful application of structured output in agentic systems is in orchestrating teams of specialized agents. By defining a strict JSON schema for the "handoff" of information from one agent to the next, you create a reliable communication bus.

This example demonstrates how the principles of API design and data contracts, familiar from traditional software engineering, are directly applicable and essential for building scalable and maintainable multi-agent systems.

Conclusion

Today, we've moved from hoping for correct formats to guaranteeing them. Enforcing structured output via constrained decoding is a fundamental technique for building production-quality agentic systems.

Key Takeaways:

• Guarantee, Don't Guess: Use structured output features to ensure agent responses are always machine-readable and conform to your data contracts.
• Mechanism Matters: The underlying process involves converting schemas into Finite State Machines that guide the LLM's token generation by masking invalid tokens at each step.
• Pydantic is the Standard: For defining schemas in Python, Pydantic offers superior readability, maintainability, and validation compared to raw JSON schemas.
• Adopt a Schema-First Mindset: Design your data contracts before you write your prompts. Treat agent interfaces with the same rigor as you would a software API.
• Structure Enables Orchestration: Strict, schema-defined handoffs are the backbone of reliable multi-agent systems, enabling predictable communication and workflow execution.

Preview of the Next Lesson

We now have a powerful tool to control the structure of an agent's output. However, a perfectly formatted JSON response can still contain incorrect or low-quality information. The next step is to improve the content within that structure.

In our next lesson, we will focus on the iterative process of improvement. We will learn how to iteratively refine system instructions using failure case analysis, a systematic method for diagnosing why an agent produces a suboptimal response and updating its core programming to prevent that failure in the future.