Defining Service Level Objectives and Operational Constraints

Recommended resources

The freely available resources this lesson will direct you to.

Welcome back! In our last lesson, we laid the essential groundwork for our agentic engineering assistant by defining its primary user journey and a clear set of acceptance criteria. We established what the system should do and how we'll know it has succeeded.

Now, we'll shift our focus from functional requirements to the critical non-functional requirements (NFRs) that distinguish a prototype from a production-ready, reliable service. In this lesson, you will define measurable objectives for latency and availability, along with constraints for cost, privacy, data retention, and auditability. These quality attributes are the guardrails that ensure our system is not just clever, but also dependable, efficient, and trustworthy.

The SRE Mindset: Beyond "It Works"

Before we start assigning numbers to our objectives, it's crucial to adopt the right mindset. As an experienced developer, you know that building a system that is 100% reliable, infinitely fast, and completely free is impossible. The practice of Site Reliability Engineering (SRE) provides a formal framework for navigating these trade-offs.

A central idea in SRE is that of appropriate reliability. Not all systems need to be as reliable as a pacemaker. The goal is to be reliable enough for the user's needs and to make explicit, data-driven decisions about where to invest engineering effort.

To get a deeper understanding of this philosophy, please watch a few key segments from the talk "Implementing SRE practices: SLI/SLO deep dive" by David Blank-Edelman.

Implementing SRE practices: SLI/SLO deep dive - David Blank Edelman - DevOpsDays Tel Aviv 2018

This video provides a foundational understanding of Service Level Indicators (SLIs), Service Level Objectives (SLOs), and Error Budgets, which are the core tools for managing reliability.

Please watch the following sections: What is SRE? This segment introduces the key concepts of reliability, appropriate reliability, and sustainability. SLIs and SLOs. Here, he introduces the fundamental concepts of Service Level Indicators and Objectives. Types of SLIs. Pay attention to the different dimensions of reliability, such as availability, latency, and quality. Crucially, note the emphasis on measuring from the customer's perspective. The SLO Recipe. This part gives a simple, concrete recipe for how to construct an SLO.

The key takeaway is that we need to define our targets (SLOs) based on measurable indicators (SLIs) and that doing so gives us a common language to discuss system performance with all stakeholders.

Defining Objectives for Our Agentic Assistant

With the SRE mindset as our guide, let's define the specific objectives for our Task Analysis Agent. We'll categorize them into performance, cost, and compliance constraints. An agentic system introduces unique challenges, particularly around its non-deterministic behavior, which we must account for.

First, let's get an overview of the kinds of metrics that are essential for AI systems.

What is AI Observability? Key to Monitoring Your LLM ...

This article from Kong's blog provides an excellent breakdown of observability signals specific to AI systems, covering performance, cost, and quality.

Please read the following sections to get a broad view of what we need to measure: Start from the beginning and read up to the heading "AI-specific signals". Read the subsections on Performance metrics and Cost metrics. Finally, skim the sections under "Essential metrics & SLIs for LLM systems", paying close attention to the examples for performance and cost.

As you can see, metrics for LLM-based systems are more nuanced than for traditional APIs. Now, let's apply these concepts to our project.

1. Latency and Availability Objectives (SLOs)

In our last lesson, we set a preliminary acceptance criterion: "The entire analysis and report generation must complete in under 30 seconds." Let's refine this into a formal Service Level Objective (SLO).

Averages can be misleading because they hide the long tail of slow requests that frustrate users. Percentiles are much more effective. For example, a P95 latency of 2 seconds means that 95% of requests are faster than 2 seconds, while 5% are slower. This gives you a much better picture of the user experience.

For LLM applications, we care about several types of latency:

Time to First Token (TTFT): How quickly the user sees the start of a response. This is crucial for perceived performance in streaming applications.
Total Latency: The time for the entire response to be generated. This is what we referenced in our initial acceptance criterion.

Let's define our SLOs using a more detailed resource.

How to Create Latency Monitoring

This article provides concrete examples of how to define latency components and formalize them into SLOs for an LLM application.

Please read these two sections: Under "Understanding LLM Latency Components", review the table and diagram that break down the different phases of latency. Jump down to the "SLO Configuration" section and read the code examples.

Based on this, we can now formulate clear SLOs for our Task Analysis Agent. We'll set targets for a rolling 30-day window.

Availability SLO: 99.5% of analysis requests will complete successfully (without unhandled errors).
- SLI: The ratio of successful requests (HTTP 2xx status codes) to total valid requests, measured at the API load balancer over 1 minute intervals.
Latency SLO (Total): 95% of analysis requests will complete in under 30 seconds (P95 < 30s).
- SLI: The total time from when the API receives the request to when the final analysis report is sent, measured at the service level.

These two SLOs give us an error budget. For availability, an SLO of 99.5% means we can tolerate 0.5% of requests failing over a 30-day period. This is approximately 216 minutes of downtime per month. If we exceed this budget, it's a signal to halt new feature development and focus exclusively on reliability.

2. Cost Constraints

LLM-powered features can become expensive quickly. It's vital to monitor costs from day one. The primary cost driver for our agent will be API calls to the model provider, which are typically billed per token.

This example dashboard from Braintrust shows key metrics for an LLM application, including Total LLM cost, Token count, and Latency percentiles. This is the kind of visibility we need to build.

Our goal is not to eliminate costs, but to make them predictable and align them with the value provided. We can set a budget as a constraint.

Cost Constraint: The average cost per task analysis must be less than $0.05.
Monitoring: We will track the number of prompt tokens and completion tokens for each analysis request and calculate the estimated cost based on the model provider's pricing (e.g., Anthropic's Claude 3.5 Sonnet at $3/million input tokens and $15/million output tokens). We will set up alerts if the daily cost exceeds a predefined budget (e.g., $10/day).

3. Privacy, Retention, and Auditability Constraints

These constraints are about building a trustworthy and transparent system. This is especially important for an agent that will be handling potentially sensitive information, like task descriptions or code snippets from a private repository.

The non-deterministic nature of agentic systems makes auditability particularly challenging and crucial. Unlike a traditional program, you can't always predict an agent's exact execution path. Therefore, we need to be able to answer: "What did the agent decide, why did it decide that, and what did it do next?"

How to Monitor, Debug, and Trust Agentic AI Systems - Observability in Agentic AI

This presentation on observability for agentic AI explains why traditional monitoring falls short and what new approaches are needed.

Please watch the following segments: The Need for Agentic Observability. This part explains the fundamental shift from deterministic to non-deterministic systems and the need to establish trust. What to Observe. The speaker breaks down the "MELT" framework (Metrics, Events, Logs, Traces) and highlights the key questions we need to answer about agent behavior. Use Cases for Observability. This final section discusses practical applications, including the concept of "Chain of Thought" for regulatory compliance and auditability.

With this context, let's define the final set of constraints for our project.

Privacy Constraint:
- All user-provided task descriptions and agent-generated analyses will be treated as confidential data.
- No Personally Identifiable Information (PII) will be intentionally collected or stored.
- The connection to the model provider API must be encrypted (HTTPS).
Data Retention Constraint:
- Task inputs, agent analysis outputs, and associated logs/traces will be stored for a maximum of 90 days to support debugging and evaluation.
- Data will be permanently deleted after 90 days.
Auditability Constraint:
- Every agent run must have a complete, immutable trace.
- The trace must include the initial prompt, the full model request/response for each step, any tool calls made, and the final generated output. This "chain of thought" is essential for debugging and understanding agent behavior. We'll use a tool like Langfuse (as shown in the video) or a similar observability platform to capture this.

This diagram illustrates how defining goals and SLOs is the first step in a larger process to deploy and monitor a system with full observability.

This flowchart shows the end-to-end process of setting up observability for LLM systems, starting with defining goals, risks, and SLOs.

Summary of Quality Targets

Let's consolidate our decisions into a single table.

Category	Objective / Constraint	Measurement / Implementation
Availability	99.5% of requests succeed (over a 30-day window).	Ratio of HTTP 2xx responses to total valid requests at the load balancer.
Latency	95% of requests complete in < 30 seconds (P95 < 30s).	Total request-to-response time, measured at the service level.
Cost	Average cost per analysis < $0.05.	Track input/output tokens per request and calculate cost based on model pricing. Alert on budget thresholds.
Privacy	User-provided content is confidential. No PII is stored.	Access controls on stored data. Encryption in transit.
Retention	Data is stored for a maximum of 90 days.	Automated data deletion policy for all logs, traces, and stored artifacts.
Auditability	Every agent run has a complete, immutable trace.	Log the full "chain of thought" (prompts, model I/O, tool calls) to an observability platform.

Conclusion

In this lesson, we've moved beyond simple acceptance criteria to define a robust set of production-quality targets for our agentic assistant. By embracing an SRE mindset, you've learned how to use SLIs and SLOs to make conscious trade-offs and manage reliability, latency, and cost in a data-driven way. You have also established critical constraints for privacy, data retention, and auditability that are essential for building trust in non-deterministic AI systems.

Key Takeaways:

SRE Mindset: Aim for appropriate reliability, not perfection. Use SLIs, SLOs, and error budgets to make deliberate engineering trade-offs.
Measure What Matters: Use percentiles (P95, P99) for latency, not averages, to better reflect user experience. Break down LLM latency into components like TTFT and Total Latency.
Cost is a Feature: Treat cost as a key metric to be monitored and managed, just like performance.
Trust Through Transparency: For non-deterministic agentic systems, comprehensive audit trails ("chain of thought") are non-negotiable for debugging, governance, and building trust.

These quality targets are not just numbers on a page; they will directly influence our architectural decisions. In the next lesson, we will begin to design the system's architecture, and these constraints will guide us in creating a system that is not only functional but also reliable, scalable, and secure.