Recommended resources
The freely available resources this lesson will direct you to.
Introduction
In our previous lesson, we established how to write a system prompt, which acts as the agent's "constitution"—defining its role, objectives, and high-level rules. We framed this using the "Agent Contract" model, which formalizes the agent's operational boundaries.
This lesson addresses the next logical question: how do we technically enforce those boundaries? Our goal is to learn how to use constraint-based prompting to prevent an agent from acting outside its scope. We will move beyond simply stating rules in a prompt to implementing mechanisms that compel the agent to follow them.
We'll explore a multi-layered approach to agent constraint, which is essential for building safe, reliable, and predictable systems. These layers function like a series of security checkpoints, each providing a different kind of guarantee.
We will investigate three primary types of constraints:
- Semantic Constraints: Filtering an agent's tasks based on the meaning of the input.
- Syntactic Constraints: Forcing an agent's output into a specific structure or format.
- Architectural Constraints: Designing the surrounding system to limit the agent's capabilities and enforce control flow.
1. Semantic Guardrails: Constraining the Agent's Scope
The first line of defense is often at the input layer. A semantic guardrail acts as a router, deciding whether an incoming query falls within the agent's designated scope of operation. This is not about understanding every nuance of the query but about making a quick, probabilistic judgment based on semantic similarity.
However, relying purely on the semantic similarity of dense vector embeddings can be brittle. For example, the queries "Can I sell my BYD car?" and "Can I sell my Tesla?" are semantically very close, but a chatbot for BYD should only handle the former. This is where a hybrid approach becomes necessary.
Advanced Guardrails for AI Agents | Full Tutorial
This video from James Briggs provides an excellent explanation of semantic guardrails, their limitations, and how a hybrid approach using both dense (semantic) and sparse (term-matching) vectors provides more robust control.
Please watch the following two segments: 00:00 - 07:52: This section introduces the concept of guardrails and explains why semantic similarity alone is insufficient for enforcing brand- or keyword-specific boundaries. 07:52 - 13:20: This part demonstrates the practical implementation of a hybrid router. Focus on how predefined routes for 'allowed' and 'blocked' topics are created and how a sparse encoder is added to account for specific terms.
The key idea is that semantic guardrails are a "soft" constraint. They are highly effective for routing and topic-scoping at the entry point of your system, preventing the agent from ever starting work on an out-of-scope task. This is an efficient way to enforce high-level operational boundaries before more expensive LLM calls are made.
2. Syntactic Guardrails: Constraining the Agent's Output
Once a task is deemed in-scope, we need to constrain the agent's output. Syntactic constraints don't directly control the meaning of the output, but by forcing it into a rigid structure, they significantly reduce the agent's freedom of action. This is a powerful technique for ensuring that an agent's output is machine-readable and conforms to a predefined specification, which is critical for reliable tool use and predictable workflows.
Common syntactic constraints include forcing the output to be a specific JSON schema or to match a regular expression.
Structured Output from LLMs: Grammars, Regex, and State Machines
The video 'Structured Output from LLMs' from the Efficient NLP channel delves into the mechanics of these constraints.
Please watch these three parts: 00:00 - 02:37: This introduces the problem and shows how to enforce a JSON schema using Pydantic with OpenAI's API. This guarantees the output format. 03:01 - 08:44: This is the core technical segment. It explains how libraries like Outlines use regular expressions to guide token generation. Given your background, you will find the explanation of how this is implemented using Finite State Machines (FSMs) particularly insightful. It's a classic CS concept applied to LLM generation. 15:49 - 17:16: This final section discusses a critical trade-off: forcing structure can sometimes decrease the accuracy of the model's output. This is a subtle but vital point for system design.
To summarize, syntactic constraints provide a hard guarantee on the form of the LLM's output. By tightly defining the acceptable "grammar" of a response, you prevent the agent from generating freeform text, making unsanctioned tool calls, or producing outputs that would break downstream components. However, the trade-off noted in the video is important: over-constraining the format might force the model into a sub-optimal reasoning path, impacting the quality of the content itself.
3. Architectural Guardrails: Constraining the Agent's Capabilities
The most robust and secure way to constrain an agent is to build the system architecture around it in a way that makes out-of-scope actions impossible. This approach, often called "security by design," doesn't rely on the LLM's compliance; instead, it uses a non-LLM orchestrator to enforce rules, similar to how a reference monitor in an operating system mediates access to resources.
This architectural approach can be implemented through several design patterns.
Design Patterns to Secure LLM Agents In Action - Labs by Reversec
The article 'Design Patterns to Secure LLM Agents In Action' provides a practical catalog of such architectural patterns. It moves the security guarantee from the LLM's alignment to verifiable system design.
Please read the introduction and the first three patterns, which represent a spectrum of increasing flexibility and complexity: Section 2 'The Core Idea': This sets the stage by emphasizing architectural constraints over heuristic defenses. Pattern 1 'Action Selector': The most restrictive pattern. The LLM is just a switch, choosing from a predefined list of parameter-less actions. Note how this eliminates prompt injection risk by design. Pattern 2 'Plan-Then-Execute': This pattern provides a form of control-flow integrity. An LLM creates an immutable plan upfront, and a non-LLM orchestrator executes it, preventing the agent from deviating from the plan even if later steps are influenced by untrusted data. Pattern 3 'LLM Map-Reduce': This pattern is for safely processing batches of untrusted data by isolating each item's processing (the 'Map' phase) and then combining the sanitized, structured results (the 'Reduce' phase). The security lies in the strict validation between phases.
These patterns show that true constraint comes from a supervising orchestrator that is not an LLM. This orchestrator verifies each action against a plan, validates data schemas, and sandboxes the LLM's capabilities.
To formalize this concept, we can look at research into Domain-Specific Languages (DSLs) for agent safety. This work operationalizes the "Agent Contract" idea we discussed in the last lesson by providing a concrete syntax for rule enforcement.
[PDF] Customizable Runtime Enforcement for Safe and Reliable LLM Agents
The research paper 'AgentSpec' introduces a DSL for specifying and enforcing runtime constraints. This provides a formal structure for implementing the architectural patterns we just saw.
Review these sections to understand the concept and structure of AgentSpec: Section 2 'Motivating Example' and 'Problem Definition': This sets up the problem of needing customizable, runtime enforcement. Section 3 'The AgentSpec Language': Focus on Figure 3, which shows the abstract syntax. Understand the core components: trigger (the event), check (the condition), and enforce (the action, e.g., user_inspection, stop). Figure 2: Look at the concrete rule example rule @inspect_transfer. This clearly shows how a human-readable rule can define a specific constraint (require user inspection for transfers to non-family members).
AgentSpec demonstrates how architectural constraints can be made explicit and configurable. A non-LLM runtime intercepts agent actions, checks them against rules defined in this DSL, and applies enforcement actions, providing a verifiable layer of safety.
Conclusion
In this lesson, we have explored the critical techniques for enforcing the boundaries we define for our agents. We've seen that robust constraint is not a single action but a layered defense strategy.
Key Takeaways:
- Constraint is a multi-layered problem. Relying on a single method, especially just prompt instructions, is brittle.
- Semantic Guardrails act as an initial filter, using a hybrid of semantic and keyword matching to keep the agent focused on its designated domain and prevent work on out-of-scope queries.
- Syntactic Guardrails provide hard constraints on the form of an agent's output (e.g., JSON, regex), which is essential for reliable tool use and structured data processing. This is achieved by guiding the token generation process at a low level.
- Architectural Guardrails offer the strongest form of constraint by using non-LLM code to orchestrate, validate, and sandbox the agent's actions. This "security by design" approach moves the guarantee of safety from the probabilistic LLM to deterministic code.
By combining these techniques, you can design agentic systems that are not only capable but also predictable, reliable, and safe—preventing them from acting outside their intended scope.
Preview of the Next Lesson
We have now defined an agent's rules and learned how to enforce them. But what should an agent do when it hits one of these constraints, or when it encounters a situation it doesn't know how to handle? In our next lesson, we will design fallback and escalation behaviors for when an agent encounters ambiguity or edge cases, moving from preventing failure to gracefully managing it.