Monitoring Backend Health with RED Metrics

1. From Four Golden Signals to the RED Method

Modern monitoring practices were heavily influenced by Google's Site Reliability Engineering (SRE) culture. In their foundational book, they outlined "Four Golden Signals" for monitoring any system: Latency, Traffic, Errors, and Saturation.

While comprehensive, the Four Golden Signals can be complex to apply universally. For request-driven services, like the APIs you build, there is a more focused and pragmatic approach: the RED method. Coined by Tom Wilkie, it distills the golden signals into the three most crucial metrics for a service:

• Rate: The number of requests your service is receiving, per second.
• Errors: The number of those requests that are failing, per second.
• Duration: The amount of time each request takes to process.

This method gives you a straightforward, service-centric dashboard for understanding service health.

Now, let's explore each of these metrics in detail.

2. Rate: Measuring Throughput

Rate (or throughput) is the measure of demand on your service. It's typically expressed as requests per second (RPS).

• Why it matters: Monitoring the request rate helps you understand traffic patterns, plan for capacity, and detect anomalies. A sudden, unexpected drop to zero might indicate an outage or a deployment failure. A sharp spike could be a sign of a DDoS attack or the effect of a new marketing campaign.
• How it's measured: In monitoring systems like Prometheus, this is typically implemented with a Counter. A counter is a metric that only ever increases. You instrument your application (often in a middleware) to increment this counter for every single incoming request. The monitoring tool can then calculate the rate of increase over a time window (e.g., the last 5 minutes) to give you the requests per second.

3. Errors: Measuring Failures

Errors represent the rate of failed requests.

• Why it matters: This is a direct measure of user-facing problems. A high error rate is a clear signal that something is wrong. Defining what constitutes an "error" is a critical decision. Typically, any request that results in a 5xx HTTP status code is considered an error. You might also decide to include certain 4xx codes, depending on the context.
• How it's measured: Like the request rate, this is also based on a Counter. You can either have a separate counter for errors (http_requests_errors_total) or, more commonly, use a single request counter with a label to distinguish outcomes. For example, your http_requests_total counter could have a status_code label. You can then query the rate for all requests where the status code is 500, 502, etc.

4. Duration: Measuring Latency and the Problem with Averages

Duration (or latency) measures how long your service takes to process a request. This is arguably the most important signal for user satisfaction, but also the most nuanced to measure correctly.

It's tempting to measure and plot the average latency. However, averages can be dangerously misleading. A single, extremely slow request (an outlier) can dramatically skew the average, while the experience for the majority of users might be fine. Conversely, a good average can hide a poor experience for a significant minority of users.

This is where percentiles become essential.

As the video explained, percentiles tell a much richer story:

• p50 (Median): 50% of your requests are faster than this value. This tells you the experience of a typical user.
• p95: 95% of your requests are faster than this value. This is a common metric to track the "unhappy path" or tail latency. It represents the experience of a user on a slow connection or a request hitting an uncached resource.
• p99: 99% of your requests are faster than this value. This represents the worst-case experience for the vast majority of your users. SLOs (Service Level Objectives) are often set against p95 or p99 latency.

How it's measured: Percentiles cannot be calculated from simple counters. They require a more complex metric type called a Histogram. A histogram groups observations into configurable buckets (e.g., requests faster than 10ms, 25ms, 50ms, etc.) and maintains a total count and sum. The monitoring tool can then use this bucketed data to estimate any percentile.

5. Seeing RED in Action with Prometheus

Now that you understand the concepts, let's see how they are implemented and queried in a real-world monitoring tool like Prometheus.

From the video, we can summarize the key implementation points:

• Instrumentation: A piece of middleware is the perfect place to record metrics for every request.
• One Metric to Rule Them All: A single Histogram metric (e.g., http_request_duration_seconds) is powerful enough to provide all three RED signals.
  • Rate is derived from the histogram's total _count.
  • Errors are distinguished by adding a status_code or success label.
  • Duration percentiles are calculated from the histogram's _buckets.
• Querying (PromQL examples):
  • Rate: sum(rate(http_request_duration_seconds_count[5m]))
  • Error Rate: sum(rate(http_request_duration_seconds_count{status_code=~"5.."}[5m]))
  • p99 Latency: histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[5m])) by (le))

6. From Metrics to Action: Dashboards and Alerts

Collecting metrics is only half the battle. To make them useful, you need to visualize them on dashboards and configure alerts to notify you when things go wrong.

This tiered approach to dashboards ensures that everyone from an executive to an on-call engineer can get the information they need at the right level of detail.

Conclusion

In this lesson, we explored metrics as the second pillar of observability. You learned how the RED method provides a simple, standardized framework for monitoring the health of any request-based service.

Key Takeaways:

• The RED method (Rate, Errors, Duration) is a focused approach for monitoring the health of backend services.
• Rate measures throughput and helps identify traffic anomalies. It is implemented with a counter.
• Errors measure the failure rate, a direct indicator of user-facing problems. It is also implemented with a counter, often using labels.
• Duration (latency) must be measured with percentiles (p50, p95, p99) to accurately reflect user experience, as averages can be misleading. Percentiles are calculated from histogram metrics.
• A single histogram metric, enriched with labels, can be used to generate all three RED signals.
• These metrics are visualized on dashboards and used to trigger alerts, making them actionable.

Preview of the Next Lesson

The RED method tells us about the quality of the requests our service is handling. But it assumes our service is up and running. What if an application has crashed, is deadlocked, or is so overloaded it can't even respond?

In our next lesson, we will address this by learning how to implement liveness and readiness health check endpoints. These are simple, standardized APIs that orchestration systems like Kubernetes use to determine if a service instance is healthy enough to receive traffic or if it needs to be terminated and replaced. This will connect our observability knowledge directly to the automated management of our deployed applications.