Skip to main content
Back to path

Welcome to a Grasp lesson

Create a free account to keep studying and create your own study path.

Lesson 50

Multi-Region Deployment Topologies Compared

Recommended resources

Introduction

In our last lesson, we focused on making a single service resilient to overload through graceful degradation patterns. While crucial, these techniques operate within the confines of a single service or datacenter. They cannot defend against larger-scale disruptions, such as a network outage affecting an entire metropolitan area or a natural disaster that takes a whole cloud region offline. To build truly robust, globally-available systems, we must architect for failure at a much larger blast radius.

This lesson zooms out from the single datacenter to the global stage. We will explore the fundamental strategies for deploying applications across multiple geographic regions to achieve high availability and disaster recovery. You will learn to compare the three primary multi-region topologies: active-passive, active-standby, and active-active. Understanding the trade-offs between these models—in terms of cost, complexity, and recovery guarantees—is a cornerstone of principal-level system design.

Defining the Objectives: RTO and RPO

Before we can compare deployment strategies, we must first define the goals of disaster recovery (DR). The choice of architecture is not primarily a technical decision; it's a business decision driven by two key metrics: Recovery Time Objective (RTO) and Recovery Point Objective (RPO).

Develop a disaster recovery plan for multi-region ...

Microsoft's Azure documentation provides clear, industry-standard definitions for these critical terms.

In the table provided, please review the definitions for RPO and RTO. These two metrics will form the basis for our entire discussion on DR strategies.

  • Recovery Time Objective (RTO): How quickly must the service be back online after a disaster? This represents the business's tolerance for downtime. An RTO of minutes is vastly more expensive to achieve than an RTO of hours.
  • Recovery Point Objective (RPO): How much data loss is acceptable, measured in time? This dictates the maximum tolerable duration between the last data backup/replication and the disaster event. An RPO of near-zero implies continuous, real-time data replication.

As a Principal Engineer, your role is to translate business requirements into these concrete metrics. A financial transaction system might have an RTO and RPO of seconds, while an internal analytics platform might tolerate an RTO of 24 hours and an RPO of 12 hours. These targets directly determine which of the following strategies is appropriate.

Active/Passive and Active/Standby: The Spectrum of Standby Recovery

The most common approach to multi-region disaster recovery involves one region actively serving traffic while a second region remains on standby, ready to take over. These strategies are often referred to as active-passive. However, "passive" can mean different things, leading to a spectrum of readiness, cost, and recovery speed.

This chart illustrates the spectrum of DR strategies, from the slow and cheap "Backup & Restore" to the instantaneous and expensive "Multi-site active/active." The active/passive strategies—Pilot Light and Warm Standby—occupy the middle ground.

Let's break down this part of the spectrum.

1. Cold Standby (Backup and Restore)

This is the most basic and cheapest form of active-passive recovery.

  • Primary Region: Handles 100% of traffic.
  • Secondary Region: Has no running services. The only assets in the secondary region are periodic data backups (e.g., database snapshots, object storage backups).
  • Failover Process: A disaster triggers a manual process to provision all infrastructure from scratch in the secondary region (using Infrastructure as Code), restore data from the latest backup, and redirect traffic.
  • Characteristics:
    • RTO: Hours to days.
    • RPO: Hours (dependent on backup frequency).
    • Cost: Very low.
    • Complexity: Low in concept, but the manual recovery process can be error-prone if not rigorously tested.

2. Pilot Light (A type of Active-Standby)

The term "pilot light" suggests having a small flame ready to ignite a larger furnace. This strategy improves on cold standby by having the core infrastructure in place but "switched off."

Disaster recovery options in the cloud

The AWS whitepaper on disaster recovery gives an excellent, detailed explanation of the pilot light strategy.

Please read the entire section on Pilot Light. Pay close attention to the distinction that core infrastructure is provisioned but application servers are not running. Also, note the critical role of continuous data replication, which is the key improvement over a simple backup/restore strategy.

  • Primary Region: Handles 100% of traffic.
  • Secondary Region: Core infrastructure (e.g., databases, VPCs) is running to receive replicated data. However, application servers and other compute resources are provisioned but not running (or not deployed at all, ready to be launched via IaC).
  • Failover Process: A disaster triggers a process to "turn on" and scale out the application servers, promote the replicated database to be the new primary, and redirect traffic.
  • Characteristics:
    • RTO: Tens of minutes.
    • RPO: Seconds to minutes (thanks to continuous replication).
    • Cost: Low.
    • Complexity: Moderate.

3. Warm Standby (A more active Active-Passive)

Warm standby takes the pilot light concept a step further by running a scaled-down but fully functional version of the application in the secondary region.

AWS re:Invent 2023 - Best practices for creating multi-Region architectures on AWS (ARC308)

The AWS re:Invent 2023 session on multi-region architectures provides a clear, concise comparison of these strategies.

Watch the segment from Warm Standby, where the speaker explains that unlike pilot light, a warm standby environment has a baseline of compute resources already running. Then continue through the explanation of the warm standby implementation from implementing warm standby, which covers data replication and code deployment.

  • Primary Region: Handles 100% of traffic.
  • Secondary Region: A complete, but smaller-scale, version of the production environment is running and actively processing replicated data. It can handle a small amount of traffic, often used for non-production tasks or read-only queries.
  • Failover Process: A disaster triggers an automated process to scale up the secondary region to full capacity and redirect all traffic.
  • Characteristics:
    • RTO: Minutes.
    • RPO: Seconds to minutes.
    • Cost: Moderate.
    • Complexity: Moderate to high, as you are managing two active environments, even if one is scaled down.

Active-Active: The Pinnacle of Availability

An active-active topology eliminates the concept of a "passive" region altogether. Both regions are fully functional, at full scale, and serve live production traffic simultaneously. This is the most resilient—and most complex—strategy.

This diagram shows a typical active-active architecture on AWS. Two identical, independent stacks are deployed in separate regions. A global traffic manager (Route 53) distributes user requests between them, and data is replicated globally (using DynamoDB Global Tables) to keep them in sync.
  • Regions: Both regions serve a portion of the production traffic. Routing can be based on user geography (latency-based routing) or simple weighting (e.g., 50/50 split).
  • Failover Process: There is no "failover" in the traditional sense. If one region fails, the global traffic manager automatically detects this (via health checks) and routes 100% of the traffic to the remaining healthy region. Users experience no downtime.
  • Characteristics:
    • RTO: Near-zero.
    • RPO: Near-zero.
    • Cost: Very high (you're running at least double the infrastructure).
    • Complexity: Extremely high.

The primary challenge of active-active is not infrastructure, but data. With users writing to two regions simultaneously, how do you prevent data conflicts and ensure consistency? This is a deep problem that requires sophisticated architectural patterns.

AWS re:Invent 2018: Architecture Patterns for Multi-Region Active-Active Applications (ARC209-R2)

This classic re:Invent talk from 2018 provides an outstanding breakdown of the data patterns used to manage active-active writes.

Watch the section from Active-Active Patterns. The speaker outlines three crucial patterns. Pay close attention to the trade-offs of each: Read Local, Write Global: All writes go to a single primary region. Simple, avoids conflicts, but adds latency for users far from the primary write region. Read Local, Write Partitioned: The application shards users, directing their writes to a specific "home" region. A good balance, but adds application logic. Read Local, Write Local: Users write to their closest region. Offers the best performance but creates the potential for write conflicts that the application must resolve (e.g., "last writer wins"). This is the most complex and riskiest pattern.

Achieving active-active requires not just duplicating infrastructure, but also ensuring regional independence. There should be no synchronous cross-region dependencies. A failure in one region must be completely isolated and not impact the other. This principle of "strict fault isolation" is fundamental to reliability at Google and other large-scale operators.

Conclusion

You now have a framework for comparing the primary multi-region deployment topologies. The choice is a direct function of the business requirements, quantified by RTO and RPO.

Here are the key takeaways:

  • Active-Passive (Cold Standby): Lowest cost, highest RTO/RPO. Suitable for non-critical workloads where hours of downtime are acceptable. Recovery is a manual, infrastructure-provisioning process.
  • Active-Standby (Pilot Light): A cost-effective middle ground. Infrastructure is provisioned but off, with data being replicated. RTO is in the tens of minutes.
  • Active-Passive (Warm Standby): A scaled-down but live environment runs in the secondary region. RTO is in the single-digit minutes, but at a higher cost than pilot light.
  • Active-Active: Highest availability and lowest RTO/RPO, but also the highest cost and complexity. It eliminates failover downtime but introduces significant data consistency challenges that must be solved at the application layer.

Understanding these patterns is only the first step. The real challenge lies in implementation. In our next lesson, we will dive deeper into the mechanics of failover by focusing on how to design a multi-region failover strategy, including health checks, traffic routing, and data replication considerations.