Caching Patterns and Invalidation Strategies

The Role and Placement of Caches

At its heart, a cache is a high-speed data storage layer that stores a subset of data, typically transient in nature, so that future requests for that data are served faster than is possible by accessing the data's primary storage location. The performance gain comes from using faster media (like RAM instead of SSDs) or avoiding network latency.

To summarize the key placements for distributed backend systems:

• External/Distributed Cache: A separate service like Redis or Memcached, shared by multiple application servers. This is the most common pattern in microservice architectures as it provides a shared, global cache that all services can leverage. It adds a network hop but decouples the cache from the application lifecycle.
• In-Process Cache: Data is stored in the application's own memory space. This offers the lowest possible latency (no network hop) but the cache is not shared, leading to potential data duplication and inconsistency between service instances. It's best suited for small, very hot, and relatively static data like feature flags or configuration settings.

For the rest of this lesson, we will primarily focus on the interaction patterns with an external, distributed cache, as this is the most architecturally significant scenario in the systems we are studying.

Core Caching Patterns

The way an application reads from and writes to the cache and database defines its caching pattern. Each pattern represents a different trade-off between read/write performance, data consistency, and implementation complexity.

The article below provides an excellent, concise summary of the four primary patterns. We'll use it as our guide.

Let's review these patterns, using the following diagrams as a visual reference for the data flows.

1. Cache-Aside (or Lazy Loading)

   • Flow: This is the most common pattern. The application is responsible for managing the cache.
     • Read: The application first checks the cache. On a hit, it returns data. On a miss, it reads from the database, writes the result into the cache, and then returns.
     • Write: The application writes directly to the database. It may or may not invalidate the cache (we'll cover this shortly).
   • Trade-offs:
     • Pro: The cache only stores data that is actually requested, preventing it from being filled with unused data. It's also resilient to cache failures; if the cache is down, the application can fall back to the database (though with performance degradation).
     • Con: Every cache miss results in three distinct operations (cache read, DB read, cache write), increasing latency for that request. Data can become stale if it's updated in the database without being invalidated in the cache.

2. Read-Through

   • Flow: The application treats the cache as its main data source. The cache itself handles database interaction.
     • Read: The application requests data from the cache. On a hit, it's returned. On a miss, the cache is responsible for fetching the data from the database, storing it, and returning it to the application.
   • Trade-offs:
     • Pro: Application logic is simplified; it doesn't need to know about the database.
     • Con: This pattern requires a specific cache provider or library that abstracts the database interaction, adding a layer of complexity or vendor lock-in. It's less common for general application caching than Cache-Aside but is the principle behind CDNs.

3. Write-Through

   • Flow: On a write, data is written to the cache and the database in a single, synchronous operation.
     • Write: Application writes to the cache. The cache synchronously writes the data to the database. The operation is complete only after both writes succeed.
   • Trade-offs:
     • Pro: Guarantees that data in the cache is never stale relative to the database after a write. This is great for read-heavy workloads where consistency is key.
     • Con: Write latency is increased because it includes both the cache write and the database write. This pattern can also lead to the "dual-write problem": what if the database write fails but the cache write succeeded? The system is now in an inconsistent state.

4. Write-Behind (or Write-Back)

   • Flow: The application performs a fast write to the cache, and the cache asynchronously writes the data to the database in the background.
     • Write: Application writes to the cache and gets an immediate acknowledgment. The cache then handles writing to the database at a later time, often in batches.
   • Trade-offs:
     • Pro: Extremely fast writes and high write throughput, as the application isn't blocked by database performance.
     • Con: There is a significant risk of data loss if the cache fails before the data is persisted to the database. This pattern is suitable only when some data loss is acceptable, such as for collecting metrics, logging, or handling analytics events.

Invalidation and Consistency Strategies

A cache, by its nature, creates a second copy of data. Keeping that copy fresh is the hardest part of caching. A Principal Engineer must be able to articulate the trade-offs between different consistency strategies.

There are two primary ways to handle stale data:

• Time To Live (TTL): This is the simplest strategy. You set an expiration time on each cache entry. After the TTL passes, the entry is automatically evicted. The next request will result in a cache miss, forcing a refresh from the database. This provides eventual consistency but allows data to be stale for the duration of the TTL.
• Explicit Invalidation: When your application writes an update to the database, it also issues a command to the cache to delete the old entry. The next read for that data will miss, fetch the new data from the database, and populate the cache. This provides stronger consistency but introduces its own challenges. The most common implementation is a DELETE operation after a successful database UPDATE (the Cache-Aside pattern).

What happens if the cache invalidation DELETE command fails after the database UPDATE succeeds? The cache now holds stale data indefinitely (or until TTL). This is a classic distributed systems problem. There's no perfect solution, only trade-offs. Strategies to mitigate this include retries with backoff for the delete operation or having a background job that periodically sweeps for inconsistencies.

A Deep Dive into Strong Consistency: Chubby's Caching Model

Most application caches favor performance and availability, accepting eventual consistency. However, for systems where correctness is paramount, like a distributed lock service, a different model is required. Google's Chubby provides a fascinating case study in building a strictly consistent client-side cache.

The Chubby model is a powerful illustration of a design point that most application developers don't encounter. By tying cache consistency directly to the session lease mechanism, it enforces correctness at the protocol level. An application using Chubby is guaranteed not to see stale data because if it misses an invalidation (due to a network partition, for example), its session will eventually expire, forcing it to discard its entire cache. This is a much stronger guarantee than the "best-effort" nature of TTL or simple write-invalidation.

Eviction Policies and Common Pitfalls

A cache has a finite size. Eviction policies are the algorithms used to decide which items to discard when the cache is full.

• Least Recently Used (LRU): Evicts the data that hasn't been accessed in the longest time. This is the most common and often the best default choice.
• Least Frequently Used (LFU): Evicts the data that has been accessed the fewest times. Useful if some items are consistently popular over others.
• First-In, First-Out (FIFO): Evicts the oldest item, regardless of how often or recently it was accessed. Simple but generally inefficient.

Finally, while caching solves performance problems, it introduces new failure modes. A Principal Engineer must anticipate and mitigate these.

Conclusion

In this lesson, we have systematically analyzed the core architectural decisions involved in caching. This knowledge is fundamental to designing high-performance, cost-effective, and resilient distributed systems.

Our key takeaways include:

• Caching Patterns: The four primary patterns—Cache-Aside, Read-Through, Write-Through, and Write-Behind—offer a spectrum of trade-offs between read/write latency, data consistency, and implementation complexity. Cache-Aside is the most common and versatile pattern for general-purpose applications.
• Invalidation Strategies: Keeping caches fresh is a critical challenge. Strategies range from simple Time-To-Live (TTL), which offers eventual consistency, to explicit write-invalidation.
• Consistency Models in Caching: We saw how a system like Chubby implements a strongly consistent cache by linking invalidations directly to a session lease mechanism, providing a stark contrast to more common, relaxed consistency models.
• Operational Concerns: A robust caching strategy must also include an appropriate eviction policy (typically LRU) and mitigations for common pitfalls like cache stampedes and cache penetration.

In our next lesson, we will move from analysis to synthesis. You will apply the patterns and strategies learned today to design a multi-layer caching architecture for a given system scenario, a common task for a Principal Engineer seeking to optimize a complex system for performance and cost.