Adaptive Rate Limiting with System Health Integration

Hello! Welcome to your next lesson in the "Rate Limiting and Traffic Shaping" module.

In our last session, we designed a distributed rate limiter that was correct, scalable, and fault-tolerant, using Redis with Lua scripting to ensure atomicity. However, its rules were static—pre-configured and rigid. While effective, such a system can't react to the dynamic nature of a service's health.

Today, we'll address this limitation. Our goal is to implement an adaptive rate limiting strategy that adjusts limits based on system health metrics. This is a more sophisticated approach that moves from simple traffic policing to intelligent, real-time capacity management. Your background in building resilient services like Rudi and your experience with capacity allocation and load shedding will be directly applicable here, as we're essentially creating a system that automatically tunes itself to prevent overload.

1. The Fragility of Static Limits

A static rate limit, such as "100 requests per second," is a blunt instrument. In a complex distributed environment, the actual capacity of a service is not constant. It can fluctuate based on:

• Request Complexity: Some requests are more computationally expensive than others.
• Downstream Health: A slowdown in a dependency (like a database or another microservice) reduces your service's effective capacity.
• Resource Contention: "Noisy neighbors" on the same host can impact performance.

Manually tuning static limits becomes a constant, reactive battle.

To break this cycle, we need a system that observes its own health and adjusts its behavior accordingly, much like how TCP congestion control works on a network.

2. The Solution: A Feedback Control Loop

An adaptive rate limiter is a feedback control system. It continuously measures the health of the service, compares it against a target, and adjusts the concurrency limit to maintain stability. The core components are:

1. Measure: Observe a health metric in real-time. The most common and effective metric is request latency (or Round-Trip Time, RTT). A rising latency is a clear signal that the system is becoming saturated. Other metrics can include CPU utilization or request queue depth.
2. Compare: Compare the observed metric against a predefined Service Level Objective (SLO). For example, "the p95 latency should remain below 200ms."
3. Act: Adjust the concurrency limit based on whether the SLO is being met.

This creates a closed loop where the system regulates itself.

The following article from Klaviyo Engineering provides an outstanding, in-depth look at building such a system to solve real-world congestion problems.

3. The AIMD Algorithm: Adjusting the Limit

The "Act" step of our feedback loop requires an algorithm to decide how to adjust the concurrency limit. The most common and proven approach is Additive Increase, Multiplicative Decrease (AIMD).

• Additive Increase (AI): When the system is healthy (e.g., latency is below the SLO), we gently increase the concurrency limit by a small, constant amount. This cautiously probes for more available capacity.
  • Rule: if latency < slo: new_limit = current_limit + 1
• Multiplicative Decrease (MD): When the system is unhealthy (e.g., latency exceeds the SLO), we drastically reduce the limit by a multiplicative factor (e.g., by 10%). This gives the system immediate breathing room to recover.
  • Rule: if latency > slo: new_limit = current_limit * 0.9

This asymmetric approach—gentle probing up, aggressive backing off—is key to maintaining stability. The system quickly reacts to stress but avoids oscillating wildly when healthy.

The diagram below illustrates this feedback loop.

Now, let's dive into the implementation details of this algorithm.

The Vector blog also provides a great, concise summary of how AIMD works in their product, which they call Adaptive Request Concurrency (ARC).

4. Implementation Sketch in Python

Let's outline how you would structure this in Python, suitable for an interview context. You'd likely define a class that encapsulates the state and logic.

import time
import threading
import math
from collections import deque

class AdaptiveConcurrencyLimiter:
    def __init__(self, initial_limit=50, max_limit=200, min_limit=10, 
                 latency_slo_ms=250, backoff_ratio=0.9, 
                 update_interval_sec=3, observation_window_size=100):
        
        self.limit = initial_limit
        self.min_limit = min_limit
        self.max_limit = max_limit
        
        self.latency_slo_ms = latency_slo_ms
        self.backoff_ratio = backoff_ratio
        



        # State for the feedback loop
        self.inflight_requests = 0
        self.lock = threading.Lock()
        



        # Latency observations
        self.observations = deque(maxlen=observation_window_size)
        



        # Background thread to update the limit
        self.update_interval_sec = update_interval_sec
        self._start_updater_thread()

    def _update_limit(self):
        """The core AIMD logic, run periodically."""
        with self.lock:
            if not self.observations:
                return




            # Use p95 to be robust against outliers
            latencies = sorted(list(self.observations))
            p95_latency = latencies[int(len(latencies) * 0.95)]




            # Multiplicative Decrease
            if p95_latency > self.latency_slo_ms:
                self.limit = math.floor(self.limit * self.backoff_ratio)

```grasp
{
  "type": "exercise",
  "id": "9c622b8c-bc3e-410f-9b5a-b43ff15743bd"
}

        # Additive Increase
        # Only increase if we are 'testing' the current limit
        elif self.inflight_requests >= self.limit * 0.8:
            self.limit += 1
        



        # Enforce bounds
        self.limit = max(self.min_limit, min(self.max_limit, self.limit))
        
        print(f"P95 Latency: {p95_latency:.2f}ms. New limit: {self.limit}")


def acquire(self):
    """Check if a new request can be accepted."""
    with self.lock:
        if self.inflight_requests < self.limit:
            self.inflight_requests += 1
            return True
        return False

def release(self, rtt_ms):
    """Release the permit and record the latency."""
    with self.lock:
        self.inflight_requests -= 1
        self.observations.append(rtt_ms)

def _updater_loop(self):
    while True:
        time.sleep(self.update_interval_sec)
        self._update_limit()

def _start_updater_thread(self):
    thread = threading.Thread(target=self._updater_loop, daemon=True)
    thread.start()

This is a conceptual example. In a real system, especially with asyncio,

you would use tasks and async locks instead of threads.

This implementation brings together the concepts from the Klaviyo article into a concrete class. Note the use of a background thread to periodically update the limit, separating the fast data path (`acquire`/`release`) from the slower control path (`_update_limit`). This pattern is analogous to running a background task in a system like Tokio.

---


```grasp
{
  "type": "exercise",
  "id": "39a805fd-dbec-4491-a267-ea9c4207ee8d"
}

5. Layers of Defense: Where Adaptive Limiting Fits

Adaptive concurrency control is a powerful tool, but it's one part of a multi-layered defense strategy against traffic spikes and overload. Your work on load shedding at AWS gives you a great perspective on this.

Let's clarify the roles:

• Autoscaling: The first layer. Adds more capacity (e.g., spins up new containers/VMs) in response to rising metrics like CPU. It's powerful but slow, taking minutes to react.
• Adaptive Rate Limiting: A second, faster layer. It operates on a sub-second to second timescale. It doesn't add capacity, but it manages the queue of incoming work to match the system's current processing capability, preventing it from tipping over.
• Load Shedding: The final emergency brake. If adaptive limiting isn't enough and the system is at imminent risk of failure (e.g., CPU at 95%), it starts intentionally dropping requests to protect core functionality.

This layered model is crucial for system design interviews, showing you can think about resilience at multiple levels of granularity.

Conclusion

In this lesson, we elevated our rate limiting design from a static gatekeeper to a dynamic, self-regulating system. This is a significant step towards building truly resilient, production-grade services.

Key Takeaways:

• Static limits are brittle: They fail to account for dynamic changes in request cost and system health.
• Adaptive control is a feedback loop: It measures system health (e.g., latency), compares it to an SLO, and acts to adjust concurrency.
• AIMD is the core algorithm: Additive Increase gently probes for capacity when healthy, while Multiplicative Decrease aggressively sheds load when stressed, ensuring stability.
• Implementation involves a control loop: A fast path handles requests (acquire/release), while a slower, periodic background task updates the limit based on recent observations.
• It's a layer of defense: Adaptive rate limiting is a fine-grained, proactive control that complements slower autoscaling and more drastic load shedding.

Next Lesson Preview

We've now covered how to police traffic with static rules and how to dynamically tune concurrency. But what happens when even our best adaptive efforts are overwhelmed? A massive, sudden traffic spike might still saturate the system before the adaptive limiter can fully react.

In our next lesson, we will focus on the last line of defense: designing and implementing degraded response strategies (e.g., queueing, shedding) for managing system overload. We'll explore what to do when you simply can't serve all incoming traffic and must make hard choices to protect the system's core.