Scalability Essentials: Vertical, Horizontal, Load Balancing, and Data Partitioning

What is Scalability?

In system design, scalability refers to a system's capacity to handle a growing amount of work by adding resources. A system that works for 100 users might grind to a halt with 100,000 users unless it was designed to scale. For ML applications, this could mean handling a surge in prediction requests or processing an ever-increasing volume of training data.

To get a high-level overview of this concept, let's start with a short video from ByteByteGo.

As the video explains, our goal is to design systems where the performance degrades as slowly as possible when demand increases. The two primary strategies to achieve this are vertical and horizontal scaling.

Vertical vs. Horizontal Scaling

At its core, the choice between scaling "up" and scaling "out" is one of the first major decisions in system architecture. An excellent article from Clarifai provides a detailed comparison. We'll use it to explore both approaches.

To summarize the key points:

• Vertical Scaling (Scaling Up): This involves increasing the resources of a single server—more CPU, more RAM, faster storage.

  • Pros: It's simple. The application logic doesn't need to change, and since everything is on one machine, communication between components is extremely fast. This is often a good starting point.
  • Cons: There's a hard physical limit to how much you can upgrade a single machine. High-end hardware gets exponentially expensive, and crucially, it creates a single point of failure (SPOF). If that one server goes down, your entire service is offline.

• Horizontal Scaling (Scaling Out): This involves adding more servers to your resource pool and distributing the load among them.

  • Pros: It offers near-infinite scalability, high availability, and fault tolerance. If one server fails, the others can pick up the slack. You can also scale elastically, adding or removing machines in response to demand.
  • Cons: It introduces significant complexity. Your application must be designed to run in a distributed environment, which brings challenges like network latency, data consistency, and service discovery.

In the context of ML systems, you often see a hybrid approach. A stateful component, like a database holding model metadata or user profiles, might be scaled vertically to maintain strong data consistency. Meanwhile, the stateless prediction service, which just takes in data and returns a prediction, is a perfect candidate for horizontal scaling behind a load balancer.

Load Balancing: Distributing the Workload

Horizontal scaling is powerful, but it immediately raises a new question: if you have a fleet of servers, how do client requests know which one to talk to? Sending all traffic to the first server would defeat the purpose. This is where a load balancer comes in.

A load balancer is a dedicated server or service that acts as a "traffic cop." It sits in front of your application servers and distributes incoming requests across them, ensuring no single server is overwhelmed.

Load balancers use various algorithms to decide where to send the next request. A great, concise explanation of these can be found in the following video.

The most common load balancing algorithms include:

• Round Robin: Requests are distributed sequentially across the group of servers. Simple and effective if all servers are identical.
• Least Connections: The load balancer sends the next request to the server that is currently handling the fewest active connections. This is a dynamic approach that adapts to the current server load.
• IP Hash: The client's IP address is used to calculate a hash, which determines which server receives the request. This ensures that requests from a particular user (or, more accurately, a particular IP) always go to the same server. This is useful for maintaining session state without a shared database, a concept known as session affinity or sticky sessions.

The choice of algorithm depends on the specific needs of your application. For a stateless API, Round Robin or Least Connections works well. For an application that stores session data on the server, IP Hashing is often necessary.

Data Partitioning: Scaling the Database

We've covered how to scale the application logic, but what about the data itself? Just like a single application server, a single database server can become a bottleneck. When a dataset grows too large for a single machine to handle efficiently, we need to scale the database layer. While vertical scaling is an option, for massive datasets, we must turn to horizontal scaling for databases.

This is achieved through data partitioning, also known as sharding. The core idea is to break a large database into smaller, more manageable pieces called shards, and spread them across multiple database servers.

Each shard contains a subset of the data, and collectively, the shards make up the entire dataset. When a query comes in, the system must know which shard holds the relevant data. This is determined by a shard key.

There are a few common partitioning strategies:

1. Key or Hash-based Partitioning: A shard key (e.g., user_id, product_id) is fed into a hash function. The output of the function determines which shard the data belongs to. For example, you could use hash(user_id) % N, where N is the number of shards. This generally leads to a uniform data distribution.
2. Range-based Partitioning: Data is sharded based on a range of the shard key. For example, user profiles with usernames starting A-F go to Shard 1, G-M to Shard 2, and so on. This can make range queries (e.g., "find all users from A to C") very efficient, but it can also lead to "hotspots" if data is not evenly distributed across ranges.

A significant challenge with simple hash-based sharding (mod N) is what happens when you add or remove servers. Changing N to N+1 or N-1 changes the result of the modulo operation for almost every key, forcing a massive, disruptive reshuffling of data across nearly all shards. This problem is solved by a more advanced technique called consistent hashing, which we will explore in a future lesson.

Conclusion

In this lesson, we've laid the groundwork for designing scalable systems. You've learned about the fundamental trade-offs between making servers bigger (vertical scaling) versus adding more of them (horizontal scaling), and how to manage the complexity that arises from the latter.

Here are the key takeaways:

• Scalability is a system's ability to handle growing load while maintaining performance.
• Vertical scaling is simple but limited by cost and a hardware ceiling, and it creates a single point of failure.
• Horizontal scaling offers near-limitless elasticity and fault tolerance but requires designing for a distributed environment.
• Load balancers are essential for horizontal scaling, distributing traffic across servers using algorithms like Round Robin, Least Connections, or Hashing.
• Data partitioning (sharding) is the strategy for horizontally scaling databases, splitting large datasets across multiple servers based on a shard key.

These concepts are not just theoretical; they are the bread and butter of system design interviews and the building blocks of any application meant to operate at scale.

In our next lesson, we will build directly on these ideas. We will compare monolithic vs. microservice architectures and see how horizontal scaling and load balancing are the enabling technologies behind the modern microservices paradigm.