Hello! Welcome to the first lesson in our new module, "Designing Scalable & Resilient Systems." In the previous module, we concluded by looking at how to document architectural decisions using ADRs—a crucial practice for maintaining clarity in a system's design over time.
Now, we shift from documenting decisions to making them. We'll start exploring the fundamental components and patterns that enable systems to handle growth and withstand failures. Today, we begin with a cornerstone of scalability: the load balancer. You'll learn what a load balancer does, why it's indispensable for modern applications, and, most importantly, how to compare the different algorithms it uses to distribute traffic. This analysis of trade-offs is central to the architectural mindset we're cultivating.
1. What is a Load Balancer and Why is it Essential?
Imagine you've launched a successful web application. As traffic grows, your single server starts to slow down, and eventually, it can't keep up. The obvious solution is to add more servers, but how do you direct users to them efficiently? This is the problem the load balancer solves.
At its core, a load balancer acts as a "traffic director" or, as one of our resources aptly puts it, an intelligent receptionist at a busy clinic. It sits in front of your servers and distributes incoming requests across them, ensuring no single server gets overwhelmed.
To get a clear picture of its benefits, let's watch a short introductory video.
What is a LOAD BALANCER really about?
This video, 'What is a LOAD BALANCER really about?' from ByteByteGo, provides a concise overview of a load balancer's primary functions and its impact on a system's architecture.
Please watch the first part of the video (00:00 - 01:12). Focus on the four key benefits it highlights: workload distribution, dynamic scaling, latency reduction, and enhanced availability.
As the video explained, a load balancer is crucial for:
- Scalability: It allows you to add or remove servers from your server pool (horizontal scaling) without affecting the application's availability.
- High Availability & Resilience: By performing health checks, it can detect if a server has failed and automatically stop sending traffic to it, preventing errors and downtime.
- Performance: By distributing the load, it prevents any single server from becoming a bottleneck and can even reduce latency by directing users to the closest or fastest server.
2. The Brains of the Operation: Load Balancing Algorithms
The most critical function of a load balancer is its decision-making process: how does it choose which server gets the next request? This logic is governed by a load balancing algorithm.
These algorithms can be broadly categorized:
- Static Algorithms: These distribute traffic using a fixed sequence or rule, without considering the current state of the servers. They are simple but not very adaptive.
- Dynamic Algorithms: These take the real-time performance and load of each server into account, allowing for more intelligent and adaptive traffic distribution.
Choosing the right algorithm is a key architectural decision that directly impacts your system's performance and efficiency. To explore these options in depth, we will use a detailed guide.
Design Load Balancer | System Design Interview
The article 'Design Load Balancer' from algomaster.io provides a thorough, deep dive into the most common load balancing algorithms. It's structured perfectly for understanding the trade-offs of each approach.
Please read the section '6.1 Load Balancing Algorithms'. For each algorithm discussed (Round Robin, Weighted Round Robin, Least Connections, Weighted Least Connections, and IP Hash), focus on understanding its core mechanism, its pros ('Why use it?'), and its cons ('When it falls short'). We will summarize and compare them right after.
3. Comparing the Key Algorithms
Let's synthesize the information from the reading and structure our comparison. Your experience as a developer gives you an intuitive sense for performance and state management, which will be very helpful in understanding these trade-offs.
Round Robin
- How it works: Requests are sent to servers in a simple, rotating sequence (Server 1, Server 2, Server 3, Server 1, ...).
- Pros: Extremely simple, fast, and stateless. It requires no monitoring of server load.
- Cons: It's "blind." It treats all servers as equal, which can be problematic if servers have different capacities or if some requests are more resource-intensive than others. An expensive request could be sent to a weak server, or a server struggling with garbage collection will continue to receive traffic.
- Best for: Homogeneous server pools (all servers are identical) where requests are uniform in cost.
Least Connections
- How it works: The load balancer tracks the number of active connections to each server and sends the next request to the server with the fewest active connections.
- Pros: It's a dynamic and adaptive algorithm. It naturally sends less traffic to slower or overloaded servers because their connections stay active longer. This makes it excellent for handling requests with variable processing times.
- Cons: It requires the load balancer to maintain state (the connection count for each server). In a distributed system with multiple load balancer instances, this can become complex to coordinate perfectly, though in practice, each load balancer tracking its own connections is often sufficient.
- Best for: Environments with long-lived connections or where request processing times vary significantly.
Weighted Algorithms (e.g., Weighted Round Robin, Weighted Least Connections)
- How it works: You assign a static "weight" to each server, typically based on its capacity (e.g., CPU, RAM).
- Weighted Round Robin: Sends a proportional number of requests based on weight (a server with weight 2 gets twice the traffic of a server with weight 1).
- Weighted Least Connections: A powerful hybrid. It directs traffic to the server with the lowest ratio of
active_connections / weight. This accounts for both pre-configured capacity and real-time load.
- Pros: Allows you to effectively utilize a heterogeneous server pool (mix of powerful and weaker machines). Weighted Least Connections is often considered the best of both worlds—it's both capacity-aware and adaptive.
- Cons: Weights are static and must be configured manually.
- Best for: Production environments with servers of varying capacities. Weighted Least Connections is a strong default choice for many applications.
IP Hash (or Source Hashing)
- How it works: The load balancer calculates a hash of the client's IP address to determine which server receives the request. The same IP address will always be routed to the same server.
- Pros: Provides "session persistence" or "stickiness" without needing to store session state on the load balancer itself. This is useful for legacy applications that store session data in memory on the web server.
- Cons:
- Uneven Distribution: If many users are behind a single corporate NAT, their traffic will all hit the same server, creating a hot spot.
- Not Adaptive: It doesn't account for server load.
- Fragile on Scale: If you add or remove a server from the pool, the hash mapping changes, and most clients will be re-routed to new servers, losing their sessions.
A picture can often clarify these different behaviors.
4. Making the Right Choice
As an architect, your job is to select the algorithm that best fits the system's requirements. The table below, adapted from the reading, is a great tool for this analysis.
| Algorithm | Complexity | Statefulness | Adaptiveness | Best For |
|---|---|---|---|---|
| Round Robin | Very simple | Stateless | None | Homogeneous backends, uniform requests |
| Weighted Round Robin | Simple | Stateless | None | Heterogeneous backends (known capacity differences) |
| Least Connections | Moderate | Per-node state | High | Variable request processing times, long-lived connections |
| Weighted Least Conn. | Moderate | Per-node state | High | Most production scenarios with mixed servers |
| IP Hash | Simple | Stateless | None | Basic session persistence (when cookies aren't an option) |
Recommendation: For most modern, stateless web applications, Weighted Least Connections is often the most robust choice. It gracefully handles servers of different sizes and adapts to unpredictable request loads.
Test your understanding!
You are designing the architecture for a new system. For each of the following scenarios, which load balancing algorithm would you choose, and why?
- A fleet of microservices for processing background jobs. All service instances run on identical hardware, and each job is roughly equal in computational cost.
- A real-time chat application where users maintain persistent WebSocket connections to the servers. The servers are a mix of new, powerful machines and older, less powerful ones.
- An e-commerce website that needs to support user shopping carts. Due to a legacy design constraint, the cart data is temporarily stored in the local memory of the web server that a user is connected to.
Show answer
-
Round Robin: This is the ideal scenario for Round Robin. Since all servers are identical and all requests (jobs) are uniform, there's no need for the complexity of a dynamic algorithm. Round Robin will distribute the load evenly with minimal overhead.
-
Weighted Least Connections: This is the best choice here. The "Least Connections" part is critical because WebSocket connections are long-lived, and you want to direct new users to the server that is currently handling the fewest people. The "Weighted" part is necessary because the servers have different capacities, and you want to send proportionally more users to the more powerful machines.
-
IP Hash: This algorithm provides the necessary session persistence. By routing a user's requests to the same server based on their IP address, you ensure they can access their in-memory shopping cart throughout their session. While this has drawbacks (like the NAT issue), it directly addresses the "sticky session" requirement imposed by the legacy design. (Note: In a real-world scenario, you'd likely advocate for moving session state out of the web server to a distributed cache like Redis, which would remove this constraint).
Conclusion
In this lesson, we've established the critical role of the load balancer as a gateway to building scalable and resilient systems. You've learned that its true power lies in the algorithm used to distribute traffic, and choosing the right one involves analyzing the trade-offs between simplicity, statefulness, and adaptiveness.
Key Takeaways:
- A load balancer distributes incoming traffic across multiple servers to improve scalability, availability, and performance.
- Round Robin is simple but blind to server load and capacity.
- Least Connections is a dynamic algorithm that adapts to real-time server load, making it suitable for non-uniform traffic.
- Weighted algorithms are essential for heterogeneous environments with servers of different capacities.
- IP Hash provides basic session persistence but comes with significant drawbacks regarding traffic distribution and flexibility.
- The choice of algorithm is an architectural decision that must align with your application's specific characteristics and requirements.
Preview of the Next Lesson:
While load balancers help scale our compute layer by distributing work, many systems are bottlenecked not by computation, but by data access. In our next lesson, we will tackle this by exploring how to design a caching strategy to dramatically improve system performance and reduce load on your databases.