Hot-Spot Management in Distributed Systems

What is a Hotspot and Why Does it Occur?

At its core, a hotspot is a severe load imbalance. While some degree of unevenness is normal, a hotspot represents a persistent state of overload on a component that becomes the critical path for a significant portion of user requests. This leads to increased latency, higher error rates, and potentially cascading failures.

The image below illustrates this concept. In the top-right panel, a single tablet in a Bigtable cluster is overloaded, receiving 25% of the read volume. This single component becomes a bottleneck for the entire system. The goal of mitigation, shown in the other panels, is to redistribute this load more evenly across the available nodes.

Hotspots primarily arise from two sources: the way data is organized and the dynamic nature of the workload itself.

1. Data and Access Pattern Imbalances

This cause is a direct consequence of the data modeling decisions we discussed in the last lesson. If data that is frequently accessed together is not partitioned effectively, hotspots will form.

A short video from Hello Interview provides excellent examples of how a poorly chosen shard key leads to hotspots.

As the video explains, there are two common scenarios:

• Temporal Hotspot: Sharding data by a timestamp (e.g., creation_date) often creates a hotspot on the most recent shard, as new data is typically accessed most frequently. All new writes and most reads will hammer a single server.
• The "Celebrity" Problem: When sharding by an entity ID (like UserID), some entities are vastly more popular than others. All traffic for that celebrity's data—profile views, posts, comments—is directed to the single shard where their data resides, overwhelming it.

2. Workload-Induced Imbalances

Even with a perfect sharding strategy, hotspots can emerge due to the dynamic and heterogeneous nature of workloads in a shared datacenter environment. A foundational Google Research paper, "Accelerating Datacenter Services with Hotspot-Aware Placement," provides deep insights into this phenomenon.

The paper reveals that in Google's datacenters, most hotspots occur at the ToR switch uplinks, and they are not just transient spikes but can persist for hours or even days. This is a critical insight: hotspots are often an infrastructure-level problem, not just an application-level one.

Hotspot Detection

Before we can mitigate a hotspot, we must first detect it. This is fundamentally a measurement and monitoring problem. The same Google paper defines a hotspot as a network link with utilization persistently above 75%. This threshold is chosen to leave a 25% headroom for maintenance and failure mitigation.

Effective hotspot detection relies on monitoring a combination of metrics:

• Resource Utilization: High CPU utilization, memory pressure, or I/O wait on a specific node or set of nodes.
• Network Throughput: Monitoring traffic on ToR switch uplinks is crucial, as the paper highlights. A consistently high throughput on one link while others are idle is a clear sign of a hotspot.
• Request Latency: The most direct signal from an application's perspective. A sudden spike in tail latency (p95, p99) for requests associated with a particular key range or service instance strongly indicates a performance bottleneck.
• Error Rates: An overloaded service will start shedding load, leading to increased error rates for a subset of requests.

The paper quantifies the severe impact of these hotspots. For example, it notes that Colossus HDD write latencies can be inflated by 2x even at just 40% ToR utilization, and some database query latencies increase by 1.5x at 75% utilization. This underscores that even operations you might consider disk-bound or compute-bound are highly sensitive to network hotspots.

Mitigation Strategies

Mitigation strategies can be categorized into proactive (designing the system to prevent hotspots) and reactive (responding to hotspots once they are detected).

1. Mitigating Data-Driven Hotspots

These strategies address the "celebrity" and temporal hotspot problems we identified earlier. The Hello Interview video you watched covers two excellent solutions.

• Compound Shard Keys: Instead of sharding on UserID alone, you create a compound key. For a celebrity, you might append a suffix, like messi-1, messi-2, ..., messi-N, and distribute their data across N shards. A write would go to a random suffix, and a read would require querying all N shards. This trades increased read complexity for massively improved write distribution for that single hot entity.
• Dedicated Celebrity Shards: This involves reactively detecting a hot entity and migrating its data to a dedicated shard, perhaps one with more powerful hardware. The routing layer needs to be aware of this, consulting a lookup service to direct traffic for "celebrities" to the special shard, while all other traffic follows the standard hash-based routing.

2. Mitigating Workload-Driven Hotspots via Aware Scheduling

This is where infrastructure-level intelligence becomes powerful. The Google paper on hotspot-aware placement details two key modifications to Borg (Google's cluster scheduler) and Colossus (its distributed file system).

The core idea behind these modifications is a powerful load-balancing principle known as "The Power of Two Choices." A seminal survey paper on this topic explains the concept beautifully.

This simple but profound idea is exactly what Google's engineers applied to make their schedulers hotspot-aware.

The results were dramatic: a 90% reduction in the number of hot ToRs and significant reductions in storage access latency. This demonstrates that for large-scale systems, building network awareness directly into the central scheduler is a highly effective mitigation strategy.

3. Mitigating Transient Hotspots via Hedged Requests

Sometimes a server becomes slow for unpredictable reasons—a background compaction, network jitter, or interference from another task. While the scheduling strategies above handle long-term load, another technique is needed for these transient issues. This is where within-request adaptation comes in, a concept famously articulated by Jeff Dean.

Hedged requests are a powerful tool for mitigating the impact of a temporary hotspot. Instead of waiting for a slow server to respond, the client proactively "bets" that another replica might be faster, effectively routing around the transient bottleneck at the request level.

Conclusion

In this lesson, we explored the causes, detection methods, and mitigation strategies for hotspots in distributed workloads. As a future Principal Engineer, your role is not only to solve these problems reactively but to design systems that are inherently resistant to them.

Here are your key takeaways:

• Hotspot Causes: Hotspots stem from both static data distribution imbalances (e.g., bad shard keys, the "celebrity" problem) and dynamic workload imbalances (e.g., network-unaware scheduling, hardware-refresh imbalances).
• Detection: Is achieved through diligent monitoring of resource utilization (CPU, network, I/O) and, most importantly, application-level latency metrics. A persistent imbalance is the key signal.
• Mitigation Strategies: Your toolkit includes a multi-layered defense:
  • Data Layer: Employ better partitioning schemes like compound keys or isolate heavy hitters onto dedicated shards.
  • Scheduling Layer: Implement hotspot-aware placement in your schedulers, leveraging principles like the "Power of Two Choices" to proactively balance load across the cluster.
  • Client Layer: Use within-request techniques like hedged requests to tolerate transient slowness and route around temporary bottlenecks.

In our next lesson, we will broaden our focus from pure performance to overall efficiency. We will describe methods for analyzing and improving resource efficiency (CPU, memory, network) in large-scale systems. Having learned how to make systems fast and resilient, we will now turn our attention to making them cost-effective without compromising performance.