Cache Stampede: Problem and Mitigation

1. What is a Cache Stampede?

A cache stampede (also known as a thundering herd problem) occurs when a very popular cached item expires or is evicted. At that moment, a large number of concurrent requests, which previously would have been served by the cache, all experience a cache miss simultaneously.

What happens next? They all try to regenerate the value by making a call to the primary data source—typically a database or a downstream microservice.

To understand this problem and its consequences, let's start with a short reading.

As the article explained, this sudden, massive spike in identical queries can overload your backend systems, leading to:

• Increased application latency: All users hitting that cache miss will experience a slow response.
• Database overload: Your database CPU and connection pool can be exhausted, slowing down the entire system.
• Cascading failures: In a microservices architecture, overwhelming one service can cause a domino effect, bringing down dependent services.

The following diagram perfectly illustrates the problem versus a potential solution.

On the left, you see the stampede in action. On the right, you see a controlled, managed refresh. Our goal for the rest of this lesson is to understand the strategies that enable this controlled behavior.

2. Strategies to Mitigate Cache Stampedes

In a system design interview, after defining the problem, you'll be expected to discuss various solutions and their trade-offs. Let's explore the most effective ones, many of which can be combined.

Strategy 1: Locking and Request Coalescing

This is one of the most robust solutions. The core idea is simple: only one thread or process gets to rebuild the cache. Everyone else waits.

1. When a request results in a cache miss, it first tries to acquire a lock for that specific cache key.
2. The winner: The first thread to acquire the lock is responsible for fetching the data from the database and populating the cache.
3. The waiters: Other concurrent threads that miss the cache fail to acquire the lock. They wait for a short period and then retry reading from the cache, which should now be populated by the "winner" thread.

This technique is called request coalescing because it coalesces multiple requests for the same data into a single database query.

For an in-depth look at how this is implemented, please read the following section.

• Trade-offs:
  • Pros: Guarantees that the backend is protected. Provides strong consistency as clients wait for the fresh data.
  • Cons: Adds complexity (lock management). The "waiter" threads experience slightly higher latency as they wait for the cache to be repopulated.

Strategy 2: Stale-While-Revalidate

This strategy prioritizes low latency and high availability over immediate consistency. The idea is to serve stale data while asynchronously refreshing the cache in the background.

1. A cache entry has two expiration times: a "soft" TTL (refreshAfterWrite) and a "hard" TTL (expireAfterWrite).
2. When a request arrives after the soft TTL has passed but before the hard TTL, the cache immediately returns the stale value (a cache hit).
3. Simultaneously, the cache triggers a single asynchronous background task to refresh the value from the database.

This ensures that users never experience the penalty of a cache miss, while still preventing a stampede because only one refresh task is kicked off.

The next section of the same article explains this with a clear Spring Boot and Caffeine example.

• Trade-offs:
  • Pros: Very low latency for users, as they rarely have to wait for a database query. Excellent backend protection.
  • Cons: Users may be served stale data for a short period. This is unacceptable for some use cases (e.g., real-time financial transactions) but perfectly fine for others (e.g., product recommendations, user timelines).

Strategy 3: Probabilistic Approaches (Jitter and Early Expiration)

Instead of complex locking or asynchronous refreshes, you can use randomness to prevent many cache keys from expiring at the same time.

• Randomized Expiry (Jitter): Instead of a fixed TTL, add a small, random amount of time (jitter) to the expiration of each key. For a base TTL of 10 minutes, the actual TTL might be set to a random value between 10 and 11 minutes. This desynchronizes the expiration times across many keys.
• Probabilistic Early Expiration: The client fetching the data can decide to proactively refresh it before it expires based on a probability calculation. For example, if a cache item is 95% through its TTL, the client might "roll the dice" and, with a small probability, decide to refresh it.

The following reading provides code snippets for these ideas.

• Trade-offs:
  • Pros: Very simple to implement. Reduces the probability of a stampede.
  • Cons: Does not guarantee prevention. A stampede is still possible, just less likely.

This visual from another source nicely contrasts the impact of these strategies on database load over time.

A Note on Non-Existent Data

A related problem can occur when users repeatedly request data that doesn't exist (e.g., looking up a user with an invalid ID). If you don't cache this fact, every request will hit your database, creating a potential vector for a denial-of-service attack.

The solution is simple: cache the null result. Store a special value (or simply null) in the cache to indicate that the data does not exist. It's crucial to use a much shorter TTL for these "negative" cache entries so that if the data is created later, it can be fetched correctly.

Conclusion

You've now moved beyond basic caching to understand a critical failure mode of high-traffic systems and how to engineer for resilience. Being able to discuss the cache stampede problem and its solutions demonstrates a mature understanding of system design.

Key Takeaways:

• Cache Stampede: A concurrency problem where multiple processes simultaneously try to regenerate the same expired cache key, overwhelming the backend.
• Mitigation Strategies:
  • Locking/Request Coalescing: Prioritizes consistency by allowing only one refresh at a time. Best for critical data.
  • Stale-While-Revalidate: Prioritizes availability and low latency by serving stale data while refreshing asynchronously. Best for non-critical data.
  • Jitter/Probabilistic Expiration: A simple way to desynchronize expirations and reduce the likelihood of a stampede.
• Combining Methods: As the Cache Stampede Protection... article concludes, the most robust systems often layer these techniques.

Preview of the Next Lesson:
We've spent a lot of time optimizing the performance and resilience of our application code. The next step in a modern development lifecycle is to prepare our application for deployment. We are now moving into our next module: Containerization & Kubernetes.

In our next lesson, we will start this journey by learning how to create an optimized, multi-stage Dockerfile for a Spring Boot application. This will be our first step toward building portable, scalable, and cloud-native services.