Automated SLO-based Alerting and Rollback

Hello! In our previous lesson, we built a powerful safety net by configuring automated evaluation gates within a CI/CD pipeline. Using Argo Rollouts, we ensured that a new model version would be automatically rolled back if it failed to meet performance criteria during the canary deployment process.

Today, we'll address a different but equally critical scenario: what happens if a problem emerges after a deployment has been successfully completed and is serving 100% of production traffic? A memory leak, a dependency failure, or a degradation in a semantic SLO might only become apparent over time. We will complete our reliability story by learning how to configure automated alerting and rollback based on live Service Level Objective (SLO) violations from the monitoring system.

This closes the loop between your observability stack and your deployment system, creating a self-healing infrastructure that can react to issues 24/7 without manual intervention. This is a hallmark of a mature SRE practice and a vital skill for an LLM Ops engineer.

1. The Foundation: Live SLO Alerting with Prometheus

Before we can automate a response, we must first reliably detect that an SLO has been breached. Your extensive experience with monitoring systems is key here. We will use the industry-standard stack of Prometheus for metrics collection and Alertmanager for handling alert logic and notifications.

The core workflow is:

1. Prometheus continuously scrapes metrics from our LLM serving endpoint.
2. We define alerting rules in Prometheus that codify our SLOs (e.g., "P99 latency must be less than 2 seconds").
3. If a metric breaches a rule's condition for a specified duration, Prometheus fires an alert to Alertmanager.
4. Alertmanager then deduplicates, groups, and routes the alert based on its configuration.

This video masterclass provides an excellent, in-depth walkthrough of setting up this entire stack. We'll focus on the sections covering alert rule definition and Alertmanager configuration.

Defining LLM-Specific Alerting Rules

For an LLM service, your SLOs—and therefore your alert rules—will go beyond basic CPU and memory. Drawing from your experience and our previous lessons, you can define rules for critical LLM metrics.

Here is an example alert.rules.yml file for an LLM service:

groups:
- name: llm-service.rules
  rules:
  - alert: HighP99RequestLatency
    expr: >
      histogram_quantile(0.99, sum(rate(vllm_e2e_request_latency_seconds_bucket{app="my-llm-service"}[5m])) by (le)) > 2.0
    for: 5m
    labels:
      severity: critical
    annotations:
      summary: "High P99 request latency on {{ $labels.app }}"
      description: "The 99th percentile latency is {{ $value }}s, which is above the 2s SLO."

  - alert: HighErrorRate
    expr: >
      sum(rate(http_requests_total{app="my-llm-service", code=~"5.."}[5m]))
      /
      sum(rate(http_requests_total{app="my-llm-service"}[5m])) > 0.01
    for: 10m
    labels:
      severity: critical
    annotations:
      summary: "High 5xx error rate on {{ $labels.app }}"
      description: "The error rate is {{ $value | humanizePercentage }}, which is above the 1% SLO."

  - alert: SemanticQualityDegradation
    expr: avg_over_time(semantic_relevance_score{app="my-llm-service"}[15m]) < 0.85
    for: 15m
    labels:
      severity: warning # May not warrant an automatic rollback, but needs investigation
    annotations:
      summary: "Semantic relevance score has dropped for {{ $labels.app }}"
      description: "The average relevance score over the last 15 minutes is {{ $value }}, below the 0.85 threshold."

With these rules in place, Prometheus will now continuously evaluate the health of our live service against our defined SLOs.

2. From Alert to Action: Triggering an Automated Rollback

Now for the crucial link: how does a critical alert in Alertmanager trigger a rollback in Argo Rollouts? This is typically achieved with an event-driven automation pattern using webhooks.

1. Alertmanager Webhook: We configure Alertmanager to send a webhook for any alert with severity: critical. This webhook contains a payload with all the alert's details.
2. Automation Endpoint: This webhook is sent to a dedicated endpoint that can initiate an action. In a sophisticated setup, this would be a tool like Argo Events or Keptn. For simplicity, you can think of it as a serverless function or a small microservice.
3. Rollback Execution: Upon receiving the webhook, the automation service validates the request and executes the command to roll back the corresponding application. For Argo Rollouts, this command is elegantly simple: kubectl argo rollouts undo <rollout-name>.

Here is how you would configure Alertmanager to send these webhooks. In your alertmanager.yml, you define a webhook_configs receiver and a route that directs critical alerts to it.

route:
  receiver: 'slack-notifications' # Default receiver
  group_by: ['alertname']

  routes:
  - match:
      severity: critical
    receiver: 'auto-rollback-webhook'
    continue: true # Also send to the default receiver if you want

receivers:
- name: 'slack-notifications'
  slack_configs:
  - api_url: 'https://hooks.slack.com/services/...'
    channel: '#alerts'

- name: 'auto-rollback-webhook'
  webhook_configs:
  - url: 'http://my-automation-service.default.svc.cluster.local/rollback'

When a critical alert fires, Alertmanager POSTs a JSON payload to http://my-automation-service/.../rollback. The automation service would parse this JSON to find the app label and then execute kubectl argo rollouts undo my-llm-service.

This architecture decouples your monitoring system from your deployment system, a robust pattern that aligns well with your SRE background.

3. Argo Rollouts: The Safety Net for Rollbacks

Why do we use kubectl argo rollouts undo instead of just running helm rollback or kubectl apply -f <previous-version.yaml>?

Because the Rollout object maintains a version history. When you run undo, Argo Rollouts:

• Identifies the current stable ReplicaSet (the problematic one).
• Identifies the previous stable ReplicaSet (the last known good one).
• Intelligently scales up the previous ReplicaSet and scales down the problematic one, managing the transition safely.

This is much safer than manual rollbacks, which can be error-prone under pressure. The Rollout object acts as the source of truth for your deployment's history.

The article below discusses the full progressive delivery lifecycle, including the rollback mechanism and the importance of monitoring integration.

Conclusion

You have now bridged the gap between passive monitoring and active, automated remediation. By configuring Prometheus alerts based on your LLM service's SLOs and using Alertmanager webhooks to trigger an Argo Rollouts undo command, you've created a closed-loop system that protects your production environment even after a deployment is complete.

Key Takeaways:

• SLOs Drive Automation: Production-grade alerting is based on SLOs for key metrics like latency, error rate, and semantic quality.
• Alertmanager is the Router: It categorizes alerts by severity and routes them to appropriate destinations—humans for warnings (via Slack/tickets) and automation for critical failures (via webhooks).
• Webhooks Trigger Actions: A webhook is the event-driven glue that connects a detected problem (the alert) to a corrective action (the rollback).
• Argo Rollouts Manages History: Using kubectl argo rollouts undo is the reliable way to revert to the last known good version, as the Rollout object safely manages the state transition.

In our next and final lesson of this module, we will consolidate our knowledge by configuring a complete CI/CD pipeline that integrates automated evaluation gates and is monitored for live SLO violations, preparing you for the final capstone project.