Load Balancing Strategies: L4 vs. L7 & Client vs. Server

The Two Axes of Load Balancing

When designing a load balancing strategy, you're primarily making decisions along two orthogonal axes:

1. The "Where": Client-Side vs. Server-Side. This axis determines the location of the load balancing logic. Is it embedded within the client application itself, or is it handled by a dedicated piece of infrastructure (a proxy or appliance) sitting between the client and the servers?

2. The "What": Layer 4 vs. Layer 7. This axis defines the level of intelligence the load balancer possesses. Does it operate at the transport layer (L4), only seeing IP addresses and ports, or at the application layer (L7), with full visibility into the content of requests like HTTP headers or gRPC method calls?

Understanding the trade-offs along both axes is crucial for making informed architectural decisions.

Axis 1: Client-Side vs. Server-Side Load Balancing

This first distinction is about architectural placement. Let's examine the different models.

As the video explains, the core trade-off is between client complexity and network path efficiency.

• Server-Side (Proxy) Load Balancing: In this model, clients are simple. They send all their traffic to a single, well-known endpoint: the load balancer. This proxy is then responsible for distributing the requests to a pool of healthy backend servers. This is the traditional model.

• Client-Side Load Balancing: Here, the client itself contains the load balancing logic. The client obtains a list of available backend servers (typically from a service discovery system like DNS or a dedicated registry) and chooses which one to send a request to based on a specific policy (e.g., round-robin).

The image below illustrates these patterns, along with a popular hybrid approach.

The "Service Mesh" pattern shown above has become a dominant architecture for microservices. It uses a sidecar proxy (like Envoy or linkerd) that runs alongside each application instance. The application communicates with what it thinks is a local service (e.g., on localhost), and the sidecar transparently handles all the complex logic of service discovery, load balancing, retries, and circuit breaking. This gives you the performance benefits of client-side balancing (the proxy is local to the client) while keeping the application code itself simple, like in the server-side model.

Axis 2: Layer 4 vs. Layer 7 Load Balancing

The second major distinction concerns the network layer at which the load balancer operates. This determines how "intelligent" its routing decisions can be.

The image below provides a fantastic visual summary of the concepts discussed in the video.

Let's solidify the key differences:

• Layer 4 (Transport Layer) Load Balancer:

  • Operation: Works with TCP/UDP segments. It makes routing decisions based on the "5-tuple": source IP, source port, destination IP, destination port, and protocol.
  • Intelligence: It has no knowledge of the application-level data (e.g., HTTP). It simply forwards connections or packets. Common algorithms include Round Robin or Least Connections.
  • Pros: Extremely fast and efficient due to its simplicity.
  • Cons: Limited flexibility. Cannot make decisions based on the content of the request.

• Layer 7 (Application Layer) Load Balancer:

  • Operation: Understands application protocols like HTTP, HTTPS, and gRPC. It terminates the client's connection, inspects the application data, and then initiates a new connection to the appropriate backend.
  • Intelligence: Can perform highly sophisticated routing. For example, it can route requests with the path /api/users/* to the user service and /api/orders/* to the order service. It can also handle SSL/TLS termination, redirect traffic based on headers, or implement sticky sessions using cookies.
  • Pros: Highly flexible and powerful.
  • Cons: More computationally intensive than L4, which can introduce a small amount of latency.

The Critical Challenge for gRPC

The distinction between L4 and L7 is particularly vital for gRPC. As we learned, gRPC uses long-lived HTTP/2 connections. If you place a simple L4 load balancer in front of your gRPC services, it will see an incoming TCP connection from a client, pick a backend server, and forward the connection to it. Because the connection is persistent, all subsequent gRPC requests from that client will be funneled down that single connection to the same backend server. The load is not balanced at the request level.

To properly load balance gRPC, you need a load balancer that understands HTTP/2 and can distribute individual requests (or streams) across the backend pool. This is an L7 capability.

This brings us back to our architectural patterns. To achieve L7 load balancing for gRPC, you can use:

1. A central L7 proxy (like an Ingress Controller or a dedicated proxy).
2. Client-side load balancing, where the gRPC client itself implements L7 logic.
3. A sidecar proxy that implements L7 logic on behalf of the application.

A Google-Scale Example: Maglev and GFE

To see how these concepts are applied in a massive-scale production environment, let's examine Google's own load balancing infrastructure, which consists of multiple, tiered systems.

Google's architecture demonstrates a powerful principle: using the right type of load balancer for the right job in a layered system.

1. Maglev (L4): At the edge of Google's network, Maglev acts as a first-pass, high-performance L4 load balancer. Its job is not to understand application traffic, but to rapidly and reliably spread incoming TCP connections across a fleet of the next-layer systems: the GFEs. Its use of consistent hashing ensures that packets for the same connection consistently go to the same GFE without requiring shared state across all Maglev machines.

2. GFE (L7): The GFE is the intelligent L7 proxy. It terminates the user's connection, decrypts SSL/TLS traffic, and inspects the application-layer data (e.g., the HTTP request). Based on the URL, headers, and other L7 information, it routes the request to the correct internal Google service (Search, Maps, Gmail, or a customer's application running on GCP).

This tiered approach combines the raw speed of L4 balancing at the edge with the intelligence and flexibility of L7 balancing deeper in the stack.

gRPC's Native Solution: Client-Side Policies

Recognizing the need for L7 balancing, the gRPC framework itself provides robust, built-in support for the client-side pattern. This allows gRPC clients to perform intelligent, request-level load balancing without needing an external proxy.

As the video demonstrates, gRPC abstracts load balancing into LB Policies. When a gRPC client is configured to connect to a service, it can be provided with a list of backend addresses. The chosen LB policy then governs how the client manages connections and distributes RPCs across them.

• pick_first: This is the simplest policy. The client tries to connect to the first address in the list. If it succeeds, it uses that single connection for all RPCs. If the connection fails, it tries the next address. This is effectively a failover mechanism, not true load balancing.
• round_robin: The client connects to all available backend addresses. For each new RPC, it distributes it across the healthy connections in a round-robin fashion. This provides true request-level load balancing.

By building this logic directly into the client libraries, gRPC gives developers a powerful, out-of-the-box solution for implementing effective L7 load balancing.

Conclusion

In this lesson, we have dissected the fundamental strategies for load balancing, focusing on the trade-offs that a Principal Engineer must evaluate.

Here are the key takeaways:

• Two Core Axes: Load balancing decisions hinge on where the logic resides (client-side vs. server-side) and what information it uses (L4 vs. L7).
• Client-Side vs. Server-Side: Server-side (proxy) is the traditional approach that simplifies clients at the cost of an extra network hop. Client-side removes this hop for better performance but adds complexity to the client, a problem often solved today with the sidecar pattern.
• L4 vs. L7: L4 balancing is fast but "dumb," operating at the connection level. L7 is smarter and more flexible, operating at the request level.
• gRPC Requires L7: Due to its use of long-lived, multiplexed connections, effective load balancing for gRPC generally requires an L7-aware strategy to distribute individual requests, not just connections.
• Real-World Application: Sophisticated systems like Google's use a tiered approach, combining the strengths of L4 (Maglev) and L7 (GFE) balancers at different layers of the stack.

We've now covered how services communicate (gRPC) and how traffic is distributed among them (load balancing). But in a complex microservices architecture, is synchronous RPC always the right choice? In our next lesson, we will broaden our perspective and develop a decision framework for selecting different communication patterns, comparing synchronous RPC with asynchronous messaging and event streaming to understand when and why to use each.