Skip to main content
Back to path

Welcome to a Grasp lesson

Create a free account to keep studying and create your own study path.

Lesson 61

Multi-Layer Caching Architecture Design

Recommended resources

Introduction

In our last lesson, we deconstructed the fundamental caching patterns and invalidation strategies that form the building blocks of high-performance systems. We analyzed the trade-offs of patterns like Cache-Aside and Write-Behind and even delved into the strong consistency guarantees provided by Google's Chubby. Now, we will move from analyzing the components to synthesizing them into a coherent whole.

This lesson is centered on the learning outcome: Design a multi-layer caching architecture for a given system scenario. As a Principal Engineer, your role extends beyond simply adding a Redis instance. It involves orchestrating a hierarchy of caches, each tailored to a specific purpose, to create a system that is fast, scalable, and cost-effective. We will explore how to strategically place caches at different points in the request path—from the client's browser all the way down to the physical storage layer—to progressively reduce load and latency.

The Principle of Hierarchical Caching

A multi-layer caching architecture is based on a simple but powerful idea: a request for data should traverse a series of caches, each progressively slower but larger and more general-purpose than the last. A hit in an outer, faster layer prevents a costly trip to an inner, slower layer. This creates a powerful defense-in-depth against latency and high load.

The following resource provides an excellent overview of the canonical layers in such an architecture.

[PDF] Design Patterns For Multi-Tiered Caching In High-Throughput Web ...

This paper offers a structured view of a typical multi-tiered caching system. It provides a clear block diagram and explains the distinct responsibilities of each layer.

Please read the section titled "Overview and Block Diagram". Then, review the brief descriptions in the following section, "Explanation of the Layers". Focus on understanding the role and placement of each of the four tiers described.

As you've just read, a typical high-throughput web application might employ a hierarchy like the one shown below.

This diagram illustrates a modern distributed system, highlighting multiple strategic locations for caching: on the client, at the edge (CDN), within a dedicated distributed cache service, and even inside the application servers themselves.

Let's dissect this hierarchy layer by layer, considering the design choices a Principal Engineer would make at each stage.

Anatomy of a Multi-Layer Caching Architecture

We will trace the path of a request from the user inward, examining the function and design trade-offs of each cache layer.

Layer 1: Client-Side Cache

This is the cache closest to the user, residing within their browser or mobile application.

  • Purpose: To provide an instantaneous experience for repeat views and interactions by eliminating the network entirely.
  • What to cache:
    • Static Assets: The application's UI shell (HTML, CSS, JavaScript). Service Workers are particularly powerful for this.
    • Infrequently Changing Data: User settings, profile information, or reference data that is read on every visit but rarely updated.
    • Media: Images and videos already viewed by the user.
  • Key Design Considerations:
    • Control & Invalidation: This is the hardest part. You rely on browser-implemented standards like HTTP Cache-Control headers and ETags. A misconfigured max-age can serve stale content for extended periods.
    • Storage Limits: Browser storage is limited and can be cleared by the user at any time.
    • Scope: This cache is scoped to a single user on a single device. It provides no benefit to other users.

Layer 2: Content Delivery Network (CDN)

A CDN is a geographically distributed network of proxy servers. It represents the first line of defense for your origin servers.

  • Purpose: To serve content from a location physically close to the user, drastically reducing network latency.
  • What to cache:
    • Static Media: The primary use case. Images, videos, large file downloads.
    • Public API Responses: API endpoints that return the same data for all users (e.g., a list of products in a category, public configuration data). Modern CDNs can handle this dynamic content.
  • Key Design Considerations:
    • Caching Pattern: CDNs are a classic implementation of the Read-Through pattern we discussed previously. On a miss, the CDN edge node fetches the content from your origin server, caches it, and serves it.
    • Invalidation: A critical decision. You can rely on a TTL (Time-To-Live) for eventual consistency or use explicit purging via an API call when content is updated. Purging offers stronger consistency but can be complex to manage at scale.
    • Cost: CDNs are priced based on data transfer volume and the number of requests. An overly aggressive caching strategy for large files can become expensive.

Layer 3: Distributed Application Cache

This is the primary workhorse for backend performance, typically implemented with services like Redis or Memcached. It sits between your application servers and your databases.

REST API Caching Strategies Every Developer Must Know

This short video segment neatly illustrates the journey of a request through the Browser, CDN, and Application layers, reinforcing the hierarchy we've discussed so far.

Please watch from the beginning of the segment, where the narrator explains the multi-layer approach using an e-commerce example.

  • Purpose: To shield backend databases from repetitive read queries for dynamic, personalized content.
  • What to cache:
    • Database Rows: Individual user profiles, product details (user:123, product:456).
    • Expensive Query Results: Fully composed objects, like a user's social media feed or a search result page.
    • Session Data: User login sessions.
    • Transient Data: Rate-limiting counters, job queue metadata.
  • Key Design Considerations:
    • Caching Pattern: Cache-Aside is the most common choice here due to its flexibility and resilience. The application retains full control over cache and database interactions.
    • Consistency: This is a major trade-off. Do you use a short TTL and accept some staleness, or do you implement a write-invalidate strategy for stronger consistency (at the cost of complexity)?
    • Failure: What happens if the cache cluster fails? A well-designed system using Cache-Aside can degrade gracefully by falling back to the database, albeit with increased latency.
    • Hot Keys: A single, extremely popular key (e.g., a celebrity's profile) can overwhelm a single cache node. Strategies include replicating that key across all nodes or using an additional in-process cache as a further layer of protection.

Layer 4: Deep Backend Caches (The Google-Scale Approach)

In hyperscale systems, the caching hierarchy doesn't stop at the application layer. Caching is embedded deep within the storage and infrastructure stack to optimize for cost and I/O at a massive scale. This is where a Principal Engineer's perspective becomes critical, as you must understand the performance characteristics of the entire platform, not just the application code.

The CacheSack paper from Google provides a compelling look into such a system: the Colossus Flash Cache. This is a general-purpose flash cache service sitting between compute services (like Bigtable and Spanner) and the underlying Colossus file system (which uses hard disks).

[PDF] CacheSack: Admission Optimization for Google Datacenter Flash ...

This paper describes a sophisticated caching layer that optimizes for Total Cost of Ownership (TCO), not just hit ratio. It showcases how Google thinks about caching as a cost-optimization and I/O reduction tool at the infrastructure level.

Please read the following sections to understand the architectural context and goals: Section 1 (Introduction): Pay close attention to Figure 1, which illustrates the system architecture. Note how the Flash Cache sits between clients (like Bigtable) and the disk servers. The primary goal is to reduce expensive disk reads. Section 3.4 (Colossus buffer cache): This is a crucial section. It reveals that there's another cache layer—a RAM buffer cache on the disk servers themselves. This demonstrates the multi-layer principle in action deep within the backend. Crucially, it highlights that a high flash cache hit ratio is meaningless if it's only serving data that would have been in the RAM buffer anyway. The true goal is to reduce actual disk I/O.

The CacheSack paper reveals a key insight for a Principal Engineer: a caching strategy must be aware of the layers below it. The algorithm doesn't just optimize for its own hit rate; it models the behavior of the downstream disk-server buffer cache to optimize for the true metric of success: reducing expensive disk reads and thereby lowering the total cost of ownership (TCO). This holistic, cost-aware view is a hallmark of senior-level system design.

A Holistic View: Mitigating Pitfalls

A well-designed multi-layer architecture also provides opportunities to solve common caching problems more effectively. In the last lesson, we mentioned cache penetration, where floods of requests for non-existent data bypass the cache and hammer the database. A Bloom filter can be used as a tiny, memory-efficient pre-filter to mitigate this.

This diagram shows how an application first checks a Bloom filter. If the filter definitively says the key is "not found" (left side), the application can immediately return without querying the cache or database. If the filter says the key is "found" (right side), it proceeds to the cache. This prevents queries for non-existent data from reaching expensive resources.

A Bloom filter is a probabilistic data structure that can tell you if an element might be in a set or is definitively not in the set. In our architecture, we can keep a Bloom filter in memory on the application servers that contains all the keys present in the database.

The flow for a read becomes:

  1. Check the Bloom filter for the requested key.
  2. If it returns "definitively not present", return 404 Not Found immediately.
  3. If it returns "might be present", proceed to the distributed cache (Layer 3) as normal.

This small, additional layer can absorb a massive amount of malicious or accidental traffic for non-existent keys, protecting the entire backend.

Conclusion

In this lesson, we transitioned from analyzing caching primitives to synthesizing a complete, multi-layer caching architecture. Designing such a system requires a holistic view, balancing trade-offs at every level of the stack.

Here are the key takeaways:

  • Hierarchical Design: Effective caching involves a hierarchy of layers—from the client, to CDNs, to distributed application caches, and even deep into the infrastructure—each with a specific role in reducing latency and load.
  • Layer-Specific Trade-offs: Each layer has unique characteristics regarding latency, capacity, consistency guarantees, and cost. A Principal Engineer must select the right data and strategies for each.
  • Cost-Aware Optimization: At Google's scale, caching is not just about speed; it's a critical tool for managing Total Cost of Ownership (TCO). As the CacheSack paper illustrates, the goal is to optimize for expensive resources like disk I/O, not just superficial metrics like hit ratios.
  • Defense-in-Depth: A multi-layer architecture naturally creates a defense-in-depth, not only for performance but also for reliability, with patterns like Bloom filters providing protection against attacks like cache penetration.

In our next lesson, we will focus on another crucial aspect of performance: reducing tail latency. While caching excels at improving average latency, a different set of techniques like speculative requests and request hedging are needed to tame the long tail of slow requests that can disproportionately impact user experience.