Setting Up Alerts for SLO and Anomaly Detection

Hello! In our previous lessons, we've focused on collecting the right data and visualizing it. You built a Grafana dashboard to get a real-time view of your LLM service's health and then learned how to instrument your application to track its operational costs. This gives you the power to observe.

Today, we'll make your system proactive. Instead of you having to watch a dashboard, the system will tell you when something needs attention. Our goal is to configure alerts for SLO violations and anomalous inference patterns, such as a sudden spike in latency. Given your SRE background, the principles of Service Level Objectives (SLOs) and alerting will be familiar territory. We'll focus on applying them to the specific metrics and failure modes of large language models.

This lesson will bridge the gap between passive monitoring and automated, active response, a critical capability for any production-grade system.

1. Defining What Matters: SLOs for LLM Inference

Before we can alert on a problem, we must first define what "good" looks like. This is the purpose of Service Level Objectives (SLOs), which are formal, measurable reliability targets from a user's perspective.

To refresh the core concepts and see how they fit together, please watch the first few minutes of the following video.

While these SLI categories are universal, for LLMs, we need to be more specific, especially with latency.

As you can see, our key SLIs for an LLM service are not just "latency" but more granular metrics:

• TTFT (Time to First Token): Measures how quickly the user sees the start of a response. Crucial for perceived performance.
• Total Latency: The end-to-end time for the full response. Important for overall system load and long-running tasks.
• Availability/Error Rate: The percentage of requests that succeed. A fundamental measure of reliability.
• Throughput: Often measured in tokens per second, this indicates the generation speed and capacity of the system.

With these SLIs, we can define our SLOs. For example:

• SLO 1 (TTFT): 95% of requests will have a TTFT under 2 seconds over a 30-day window.
• SLO 2 (Availability): The service will have a 99.5% success rate over a 30-day window.

These SLOs now give us clear, measurable targets. When we are at risk of missing these targets, we need an alert.

2. The Alerting Pipeline

In a modern observability stack, alerts follow a standard pipeline. Your experience with SRE tooling will make this process familiar.

This diagram shows the flow of an alert from a Prometheus server, through Alertmanager for processing (grouping, inhibiting, silencing), and finally out to notification channels like Slack or PagerDuty.

The key components are:

1. Prometheus Server: Continuously scrapes metrics and evaluates pre-configured alert rules against them. When a rule's condition is met for a specified duration, it fires an alert.
2. Alertmanager: Receives alerts from Prometheus. Its job is to de-duplicate, group, and route them to the correct notification channels based on labels (e.g., severity=critical goes to PagerDuty, severity=warning goes to Slack). It also handles silencing alerts during maintenance.
3. Grafana: While it can be used to define and manage alerts via its UI, it also serves as the primary tool for visualizing the metrics that the alerts are based on, helping you diagnose the issue.

You can define alerts directly in Prometheus configuration files or through the Grafana UI. Let's explore both methods.

3. Method 1: Configuring Alerts in Prometheus

Defining alerts directly in Prometheus YAML files is a common practice in environments that use Infrastructure-as-Code (IaC). It allows you to version control your alerts alongside your application and infrastructure configuration.

Here is a practical example of a Prometheus alert rule for high LLM latency.

Let's break down that rule:

- alert: LLMHighP95Latency
  # The PromQL expression to evaluate.
  expr: histogram_quantile(0.95, sum by (le) (rate(tgi_request_duration_bucket[5m]))) > 3
  # The condition must be true for this duration before firing. Prevents flapping.
  for: 10m
  # Labels are used by Alertmanager for routing and grouping.
  labels:
    severity: page # 'page' often means "wake someone up"
  # Annotations add human-readable context to the alert notification.
  annotations:
    summary: "TGI p95 latency > 3s (10m)"

The expr is the heart of the alert. It calculates the 95th percentile latency over a 5-minute window and checks if it's greater than 3 seconds. The histogram_quantile function is essential for working with histogram metrics, which you configured in previous lessons to capture latency distributions.

To build a robust alerting suite, you need rules for various scenarios. The article below provides a fantastic "cheat sheet" of alert definitions.

From that resource, here are some critical alerts you should always have, along with the conceptual PromQL:

Alert Description	Key Metric	Example PromQL Expression	Why It's Important
High P95 Time-to-First-Token	llm.ttft.p95	llm_ttft_p95 > 2000	Catches degradation in perceived responsiveness.
Critical P99 Total Latency	llm.request.duration	histogram_quantile(0.99, sum(rate(llm_request_duration_bucket[5m])) by (le)) > 30000	Catches outlier requests that hog resources or time out.
Elevated Error Rate	llm.request.outcome	sum(rate(llm_request_outcome_total{outcome="error"}[5m])) / sum(rate(llm_request_outcome_total[5m])) > 0.05	Indicates systemic failures or provider issues.
Anomalous Throughput Drop	llm.tokens.generated	rate(llm_tokens_generated_total[10m]) < 100	Can signal a silent failure or a processing bottleneck.

4. Method 2: Configuring Alerts in Grafana

For many teams, creating alerts directly from a Grafana dashboard panel is a more intuitive workflow. It allows you to visually identify a trend and then attach an alert to it in a few clicks.

This video provides a clear, step-by-step walkthrough of the entire process in Grafana.

By following this process, you could go to the "P95 TTFT Latency" panel on your dashboard, enter the "Alert" tab, and visually configure the same logic as the Prometheus YAML rule: evaluate every 1 minute, fire if the value is above 2000ms for 5 consecutive minutes, and send the notification to your team's Slack channel.

Conclusion

Today, you've learned how to transform your observability stack from a passive reporting tool into an active defense mechanism for your LLM services. By defining clear, model-specific SLOs and configuring automated alerts, you can ensure that you are the first to know when your service's reliability is at risk.

Key Takeaways:

• LLM-Specific SLOs are Key: Your reliability targets must be based on metrics that matter for user experience, like Time to First Token (TTFT), error rate, and total latency.
• The Alerting Pipeline is Standardized: Alerts flow from Prometheus (evaluation) to Alertmanager (routing) and finally to notification channels.
• Configuration Can Be Code or UI: You can define alerts as code in Prometheus for version-controlled, automated deployments, or you can use the Grafana UI for a more interactive, visual workflow.
• Start with Critical Alerts: A minimal set of alerts covering P95/P99 latency, error rate, and throughput provides excellent coverage for most common failure modes.

In our previous lesson, you learned how to calculate cost metrics. In this lesson, you learned how to get alerted on performance issues. In the final lesson of this module, you will bring the cost aspect back to the forefront by building a dedicated cost analytics dashboard to track dollars-per-token and dollars-per-request over time, giving you a complete FinOps command center.