Structured LLM Outputs with JSON Mode & Schemas

Welcome to the third lesson in our journey to becoming an AI Engineer. In our previous lessons, you've learned to guide a model's reasoning with advanced prompting and to manage its finite memory by mastering the context window. We've focused on controlling the input to the model. Now, we'll turn our attention to a challenge that's critical for any software engineer building with LLMs: controlling the output.

By default, LLMs produce unstructured text—a stream of words. This is fine for a chatbot, but for a production system that needs to parse data, trigger other functions, or save information to a database, it's a source of unreliability. This lesson will teach you how to tame that chaos. You will learn how to enforce structured outputs, forcing the model to respond in a predictable, machine-readable format like JSON that adheres to a strict schema you define. This skill is the bridge between the probabilistic world of LLMs and the deterministic world of traditional software.

The Problem: From Messy Strings to Reliable Data

As a software engineer, you build systems that rely on contracts and predictable data structures. An API that sometimes returns a string and other times an object with a different shape would be considered broken. Yet, this is the default behavior of LLMs. Early attempts to solve this involved prompt engineering ("Please return a JSON object...") and brittle post-processing with regular expressions. These methods are unreliable and not suitable for production.

The modern solution is to enforce structure at the inference level, guaranteeing the output is not just syntactically valid, but also conforms to a specific schema. This transforms the LLM from a simple text generator into a powerful component for data extraction and classification that you can reliably integrate into larger software systems.

Levels of Enforcement: JSON Mode vs. Strict Mode

The path to reliable structured output has evolved. It's crucial to understand the distinction between two key features offered by modern LLM providers.

As the guide explains, there are two primary modes you'll encounter:

1. JSON Mode: This guarantees that the model's output will be a syntactically valid JSON object. However, it offers no guarantee about the schema of that JSON. The model could return {"response": "Here is the data"} when you expected {"user_id": 123, "is_active": true}. It's a step up from raw text, but still requires validation.

2. Strict Mode (Schema Enforcement): This is the production-grade solution. It guarantees that the model's output will not only be valid JSON but will also perfectly match a JSON Schema you provide. It will have the correct fields, data types, and structure, every single time. This is what we mean by "structured output" in a professional context.

For virtually all production use cases where you need to parse the LLM's response, you should use Strict Mode with an explicit schema.

How Does It Work? Constrained Decoding

You might be wondering how this 100% guarantee is possible. It's not just clever prompting; it's a change to the fundamental generation process of the LLM itself, a technique called constrained decoding.

In essence, when you provide a schema, the LLM provider's inference engine compiles it into a state machine. At each step of generating the response, the engine looks at all the possible next tokens in its vocabulary. It then "masks" any token that would violate the schema, effectively setting its probability of being chosen to zero.

Imagine the model has just generated {"rating": . If your schema specifies that rating must be an integer, the engine will only allow tokens that are digits (0-9). It physically cannot generate a quotation mark or a letter, even if the model's internal "thought" was to write out the word "five". This is what provides the mathematical guarantee of compliance.

For a slightly more technical look at this process, the following article provides a great explanation.

Defining Schemas with Pydantic

To use strict mode, you need to provide a schema. While you can write raw JSON Schema files, the industry standard in the Python ecosystem is to use Pydantic. Pydantic allows you to define data structures using standard Python type hints, which are then automatically converted into JSON Schema. This is more readable, less error-prone, and integrates perfectly with the rest of your Python code.

Since you have a background in Go and JavaScript, you can think of Pydantic models as being analogous to defining a struct in Go or a type/interface in TypeScript. They define the shape and types of your data.

This video from Dave Ebbelaar gives a fantastic demonstration of using Pydantic for this exact purpose and explains why it's the recommended approach.

Let's look at some key Pydantic patterns for defining schemas.

The Reasoning-First Pattern

This is the single most important design pattern for structured output. Always place a reasoning or thought_process field before the fields that contain the final answer. Because LLMs generate text token by token from left to right, this forces the model to articulate its reasoning before committing to an answer, leading to more accurate results. This is essentially Chain-of-Thought prompting embedded directly into your schema.

from pydantic import BaseModel, Field
from typing import Literal

# GOOD: Reasoning comes first
class SentimentAnalysis(BaseModel):
    reasoning: str = Field(description="Think step-by-step. Analyze the user's text and explain why you are choosing the sentiment.")
    sentiment: Literal["positive", "negative", "neutral"] = Field(description="The final sentiment classification.")
    confidence: float = Field(description="Confidence score from 0.0 to 1.0.", ge=0.0, le=1.0)

# BAD: Answer comes first, encouraging rationalization
class BadSentimentAnalysis(BaseModel):
    sentiment: Literal["positive", "negative", "neutral"]
    confidence: float
    reasoning: str # The model has already decided the sentiment before it gets here.

Field Descriptions as Prompts

The description argument in Field is not just for documentation. These descriptions are passed to the LLM as part of the schema and act as targeted prompts, guiding the model on what to populate in that field. Be specific and clear in your descriptions.

Hands-on Exercise: Structuring Bug Reports

It's time to apply these concepts. As a software engineer, you often deal with bug reports. Let's build a simple tool to automatically parse an unstructured bug report from a user into a structured format suitable for a ticketing system like Jira or GitHub Issues.

Your Task:
Create a Python script that uses OpenAI and Pydantic to extract structured information from a raw text bug report.

1. Define the Schema: Create a Pydantic BaseModel called BugReport with the following fields:

   • reasoning: A str field. Use a description to instruct the model to think through the report before filling out the other fields.
   • title: A str field for a concise, one-line summary of the bug.
   • priority: A Literal type that can only be one of "Low", "Medium", or "High".
   • is_ui_bug: A bool field to indicate if the bug is related to the user interface.
   • reproduction_steps: A list[str] to capture the steps to reproduce the bug.

2. Implement the Extraction Function: Write a function extract_bug_report(text: str) -> BugReport that:

   • Takes a raw text string of a bug report as input.
   • Uses the OpenAI Python client's client.beta.chat.completions.parse() method.
   • Provides a simple system prompt like "You are an expert at analyzing bug reports and extracting key information."
   • Passes the user's bug report text.
   • Sets the response_format to your BugReport Pydantic model.
   • Returns the parsed, structured BugReport object.

3. Test Your Function: Call your function with a sample bug report and print the resulting structured object.

Starter Code:

import os
from openai import OpenAI
from pydantic import BaseModel, Field
from typing import Literal, List

# Ensure you have your OPENAI_API_KEY set as an environment variable
client = OpenAI(api_key=os.environ.get("OPENAI_API_KEY"))

# --- 1. Define your Pydantic schema here ---
class BugReport(BaseModel):
    # Your field definitions go here.
    # Remember the reasoning-first pattern and use descriptive fields!
    reasoning: str = Field(description="First, think step-by-step about the content of the bug report to understand the core issue, its severity, and its context.")
    title: str = Field(description="A concise, one-line title for the bug ticket.")
    priority: Literal["Low", "Medium", "High"] = Field(description="The estimated priority of the bug.")
    is_ui_bug: bool = Field(description="True if the bug is related to the user interface (e.g., a button not working, layout issues).")
    reproduction_steps: List[str] = Field(description="A list of clear, numbered steps to reproduce the bug.")


# --- 2. Implement the extraction function ---
def extract_bug_report(text: str) -> BugReport:
    """
    Uses OpenAI's structured output to parse a bug report.
    """
    response = client.beta.chat.completions.parse(
        model="gpt-4o", # Or another model that supports structured outputs
        messages=[
            {"role": "system", "content": "You are an expert bug report analyst. Your task is to extract structured information from user-submitted reports."},
            {"role": "user", "content": text},
        ],
        response_format=BugReport,
    )
    # The .parse() method directly returns an instance of your Pydantic model
    return response.choices[0].message.parsed


# --- 3. Test your function ---
raw_bug_report = """
Hey team, I think I found a problem. When I go to the user profile page,
the 'Save Changes' button doesn't do anything. I click it, and the spinner
shows for a second, but my new profile picture doesn't get saved.
I'm on Chrome on my Mac. To reproduce: 1. Log in. 2. Go to /profile.
3. Click 'Upload new picture' and select a file. 4. Click 'Save Changes'.
The page just sits there and nothing happens. It's pretty annoying.
"""

structured_report = extract_bug_report(raw_bug_report)

print("--- Structured Bug Report ---")
print(f"Title: {structured_report.title}")
print(f"Priority: {structured_report.priority}")
print(f"Is UI Bug: {structured_report.is_ui_bug}")
print("\n--- Reasoning ---")
print(structured_report.reasoning)
print("\n--- Reproduction Steps ---")
for i, step in enumerate(structured_report.reproduction_steps, 1):
    print(f"{i}. {step}")

# You can also get the model as a dictionary
# print(structured_report.model_dump_json(indent=2))

This exercise demonstrates how you can take messy, human-generated text and reliably convert it into a data structure your application can work with, a foundational skill for any AI Engineer.

Conclusion

You have now learned one of the most powerful techniques for building production-grade AI systems. By enforcing structured outputs, you eliminate a major source of runtime errors and create a reliable bridge between your LLM and your application code.

Key takeaways from this lesson:

• From Hope to Guarantee: We've moved beyond unreliable prompt engineering to guaranteed schema compliance using Strict Mode and constrained decoding.
• Pydantic is the Standard: For Python developers, Pydantic is the best-in-class tool for defining schemas. It's clean, readable, and integrates seamlessly with LLM APIs.
• Schema Design is Prompt Design: The names and descriptions of your schema's fields are powerful prompts that guide the model's output quality.
• Reasoning First: Always structure your schemas to let the model "think" before it "answers" to improve the accuracy of the final output.

In our last few lessons, we've focused on getting reliable data into and out of the model. However, the API calls themselves can fail due to rate limits, network issues, or provider downtime. In our next lesson, we will address this by learning how to implement robust error handling for LLM APIs, including strategies like exponential backoff and provider fallbacks.