Caching Strategies for Performance Improvement

1. Why Cache? And Where?

At its core, caching is about storing frequently accessed data in a faster, closer storage layer. The performance difference is staggering: accessing data from in-memory caches like Redis can be thousands of times faster than fetching it from a database stored on an SSD.

But where should this cache live? It's not just about placing a Redis instance between your app and your database. Caching can be implemented at multiple layers of a system. To get a high-level overview of these layers, let's watch a video from a staff engineer at Meta.

As the video explained, we can place caches in several key locations:

• External Cache: A separate service (like Redis or Memcached) shared by all application servers. This is the most common pattern in system design and will be our primary focus. It's great for sharing cached data across a horizontally scaled application.
• In-Process Cache: Storing data directly in the memory of the application server. This is incredibly fast as it avoids network calls, but the data is not shared between servers. It's best for small, frequently accessed, and rarely changed data like feature flags or configuration settings.
• Content Delivery Network (CDN): A geographically distributed network of servers that caches content close to users. We'll touch on this today, but it's the main topic of our next lesson. It's primarily used for static assets like images, videos, and CSS to reduce network latency.
• Client-Side Cache: Storing data on the user's device itself (e.g., in the browser or mobile app). This offers the fastest access but gives you the least control over data freshness and invalidation.

For the rest of this lesson, we will primarily focus on designing strategies for an external cache, as it's the most common and versatile solution for backend systems.

2. Core Caching Patterns: The "How"

Once you've decided to use an external cache, the next critical question is: how should your application interact with it? This is defined by your caching architecture or pattern. These patterns determine the logic for reading and writing data, and each comes with distinct trade-offs regarding performance, consistency, and complexity.

To explore these patterns, we'll use a detailed article that breaks them down clearly.

Let's summarize and compare the main patterns you just read about. The following diagram is an excellent visual reference for these flows.

Here’s a breakdown of the trade-offs, which is crucial for making architectural decisions:

Cache-Aside (Lazy Loading) is the most popular pattern. It's simple to implement and ensures that only data that is actually requested gets cached, keeping the cache lean. Its main downside is the "cold start" problem: the first request for a piece of data will always be slow (a "cache miss").

Write-Through prioritizes data consistency over write performance. It's more complex to set up because the cache itself needs to manage the database write.

Write-Behind (or Write-Back) prioritizes write performance above all else, accepting the risks of data loss and eventual consistency.

3. Managing Cache Contents: Eviction and Invalidation

A cache has limited memory. What happens when it gets full? And how do you handle data that has been updated in the database, making the cached version stale? A complete caching strategy must answer these two questions.

1. Cache Eviction: This is the process of removing items from the cache to make room for new ones.
2. Cache Invalidation: This is the process of marking cached data as "stale" or removing it because the original data has changed.

Let's quickly review the common policies.

Here's a summary of the key policies:

• Least Recently Used (LRU): Evicts the item that hasn't been accessed for the longest time. This is a great general-purpose policy and a very common default.
• Least Frequently Used (LFU): Evicts the item that has been accessed the fewest number of times. This is useful when you have a clear set of "popular" items that you want to keep cached, even if they aren't accessed for a short period.
• Time To Live (TTL): This isn't an eviction policy on its own but a mechanism. You set an expiration time on each cache entry. When the time is up, the item is automatically removed. This is essential for ensuring that stale data doesn't live in the cache forever.

For invalidation, the most common strategy used with Cache-Aside is invalidate-on-write. When your application writes an update to the database, it also sends a command to the cache to DELETE the old entry. The next time the data is requested, it will be a cache miss, and the application will fetch the fresh data from the database and populate the cache.

4. Putting It All Together: A Framework for Design

You now have the building blocks. Let's structure them into a repeatable design process. The best way to approach caching in a system design context is to follow a clear, logical sequence.

The "How to Introduce Caching" section in the article we've been using provides an excellent framework for this.

Let's recap that design process:

1. Identify the Bottleneck: Be specific. Is it high read latency on the database? Is it expensive computations? Quantify the problem (e.g., "User profile queries take 50ms and are our most frequent operation.").
2. Decide What to Cache: Focus on data that is read frequently and updated infrequently. Define your cache keys (e.g., user:{id}:profile).
3. Choose Your Cache Architecture: Select a pattern (e.g., Cache-Aside, Write-Through) based on your system's read/write patterns and consistency requirements. Justify your choice. For most general-purpose applications, Cache-Aside is the safe and correct default.
4. Set an Eviction/Invalidation Policy: Define how you'll manage cache size and stale data. A common combination is LRU with a TTL. Also, specify your invalidation strategy (e.g., "We will invalidate the user profile cache entry whenever a user updates their profile.").
5. Address the Downsides: Show you've considered the trade-offs. What happens if the cache fails? (Your app should gracefully fall back to the database). How will you prevent a Cache Stampede (Thundering Herd), where a popular item expires and thousands of requests hit the database at once? (Solutions include request coalescing or proactive refreshing).

Conclusion

Today we've moved from simply "adding a cache" to systematically "designing a caching strategy." This is a fundamental skill for building scalable systems and a core part of the architectural mindset you're developing.

Key Takeaways:

• Caching is a multi-layered concept, existing from the client all the way to the database.
• The Cache-Aside pattern is the most common and versatile strategy for read-heavy applications. Write-Through offers strong consistency at the cost of write speed, while Write-Behind offers maximum write speed at the cost of consistency and data safety.
• A complete caching strategy must include eviction policies (like LRU) to manage memory and invalidation strategies (like TTL or invalidate-on-write) to manage data freshness.
• Designing a strategy involves identifying the bottleneck, choosing what to cache, selecting an architecture, defining policies, and considering the potential problems.

Preview of the Next Lesson:
We briefly mentioned Content Delivery Networks (CDNs) as a form of globally distributed caching. In our next lesson, we will do a deep dive into the purpose and architecture of a CDN, exploring how it dramatically reduces latency for users around the world and offloads traffic from your origin servers.