Load Balancing Strategies & Trade-offs

1. What is a Load Balancer?

Before diving into specific strategies, let's solidify our understanding of what a load balancer is and where it fits into a system's architecture. At its core, a load balancer is a reverse proxy that acts as a "traffic cop" for your backend services.

Let's watch a short video to visualize this.

As the video illustrates, load balancers are not just for distributing client traffic to web servers. They are a fundamental pattern used between any two layers of a distributed system where you have multiple instances of the downstream service (e.g., between web servers and an application cache, or between application servers and a database cluster).

The decision-making logic inside a load balancer is governed by a load balancing algorithm. Let's explore the most important ones.

2. A Framework for Algorithms: Static vs. Dynamic

Load balancing algorithms can be broadly divided into two categories. This distinction provides a useful mental model for comparing them.

To summarize:

• Static Algorithms: Distribute traffic based on a fixed, predetermined pattern. They are simple and fast but "blind" to the current load on the servers.
• Dynamic Algorithms: Monitor the servers and make real-time routing decisions based on performance metrics like connection count or response time. They are more intelligent but introduce complexity and overhead.

Now, let's dive deep into the specific algorithms you'll be expected to know in an interview. We'll use a fantastic article from AlgoMaster.io as our guide.

3. Core Load Balancing Algorithms

Let's explore the mechanics and trade-offs of the most common algorithms.

Round Robin and Weighted Round Robin

This is the simplest static algorithm.

• Round Robin: Sequentially distributes requests to servers in a circular order.
  • Pros: Extremely simple, stateless, and computationally cheap.
  • Cons: Treats all servers as equal. If one server is slower or has less capacity, it can become overloaded.
• Weighted Round Robin: Assigns a "weight" to each server, distributing a proportional number of requests based on that weight.
  • Pros: Accounts for heterogeneous server capacities (e.g., a mix of powerful and weaker machines).
  • Cons: The weights are static. It doesn't adapt to transient issues like a server slowing down due to garbage collection or a temporary spike in expensive requests.

Least Connections

This is a classic dynamic algorithm that addresses the "blindness" of Round Robin.

• Least Connections: Sends the next request to the server with the fewest active connections.
  • Pros: Naturally adaptive. A server that is slow or busy with long-running requests will have a higher connection count and will automatically receive less new traffic. This self-balancing property is very powerful.
  • Cons: Requires the load balancer to track the state (connection counts) for each backend.
• Weighted Least Connections: A hybrid approach that sends traffic to the server with the lowest ratio of connections / weight. This is often the best general-purpose choice, as it accounts for both pre-configured capacity (weight) and real-time load (connections).

This image provides a helpful visual summary of several algorithms, including the ones we've just discussed.

Hash-Based Methods and the Rise of Consistent Hashing

Sometimes, you need requests from the same user to always go to the same server. This is called session persistence or "stickiness." A simple way to achieve this is with a hash-based algorithm.

However, this simple approach has a major flaw.

This "massive reshuffling" is often called the rehashing problem. When you use a simple hash(key) % N formula, changing N (the number of servers) changes the destination for almost every key. This is catastrophic for caches (causing mass cache misses) and disruptive for user sessions.

Consistent Hashing was invented to solve this exact problem.

The basic consistent hashing algorithm has its own issues, namely that server partitions on the ring can be uneven. This is solved by using virtual nodes.

• Consistent Hashing: Maps servers and keys onto a circular hash ring. A key is assigned to the first server found moving clockwise from the key's position.
  • Pros: When a server is added or removed, only a small fraction of keys are remapped. This makes it ideal for dynamic environments like auto-scaling groups or distributed caches (e.g., Memcached).
  • Cons: More complex to implement than other algorithms.

4. Choosing the Right Algorithm: An Interview-Ready Summary

In an interview, you must be able to recommend an algorithm and justify your choice. The best choice depends entirely on the system's requirements.

Here is a summary of the trade-offs to guide your decision-making.

How to articulate this in an interview:

"For the stateless API servers, I would start with Weighted Least Connections. This algorithm is adaptive and will naturally balance the load even if some requests are more expensive than others or if we have a mix of server instance types.

However, for the distributed cache layer, we must use Consistent Hashing. This ensures that when we add or remove cache nodes, we minimize the number of cache misses by only remapping a small fraction of the keys, which is critical for maintaining performance."

Conclusion

Today we've demystified load balancing, a cornerstone of scalable system design. You've moved beyond simple definitions to understand the deep trade-offs between the most important algorithms.

Key Takeaways:

• Load balancing enables horizontal scaling by distributing traffic across a pool of servers.
• Round Robin is simple but blind to server state. Weighted Round Robin accounts for different server capacities but is still static.
• Least Connections is a dynamic algorithm that adapts to real-time load, making it a robust choice for many applications.
• Consistent Hashing is the essential solution for dynamic environments where servers are frequently added or removed, as it minimizes the disruption of remapping keys.
• Your choice of algorithm must be a deliberate trade-off, justified by the specific needs of the system components you are designing.

In our next lesson, we will explore another critical performance-enhancing technique: caching strategies. We'll discuss patterns like write-through, write-back, and write-around, and see how effective caching can drastically reduce the load on your databases and backend services.