Scaling Stateless Services & Externalizing State

What are Stateful and Stateless Architectures?

At its core, the difference between stateful and stateless comes down to one question: Does the server need to remember information about a client from one request to the next?

• A stateful service remembers client-specific data (the "state") across multiple requests. This could be a user's session ID, the items in their shopping cart, or their progress in a multi-step form. The server holds this context in its own memory.
• A stateless service treats every request as a completely independent transaction. It holds no client-specific data between requests. Any information needed to process the request must be provided by the client with the request itself, or fetched from an external storage system.

To get a more intuitive feel for this, let's watch a short video that uses some great real-world examples.

As the video illustrates, the fundamental shift is moving the "memory" of the application out of the individual application servers.

Now, let's dive into the most important consequence of this architectural choice: its impact on scalability.

Why Stateless Services Are Easier to Scale

In the last lesson, you learned that horizontal scaling (adding more servers) is the key to handling high traffic. Statelessness is the principle that makes horizontal scaling truly effective and straightforward.

To understand why, I'd like you to read a few sections from a detailed article that breaks down the differences.

Based on that reading, let's crystallize the key reasons why stateless services are superior for scaling:

1. Frictionless Horizontal Scaling & Load Balancing:

   • In a stateless architecture, every server instance is identical. Since no server holds unique session data, any server can process any incoming request.
   • This means a load balancer can use simple, efficient algorithms like Round Robin or Least Connections to distribute traffic evenly. There's no need for complex "sticky sessions" or session affinity.
   • When traffic spikes, you simply add more identical server instances to the pool. The load balancer starts sending traffic to them immediately, and the system scales seamlessly.

2. High Availability & Fault Tolerance:

   • If a stateless server crashes, no critical state is lost with it. The server was just a processor of requests.
   • The load balancer's health checks will detect the failed instance and stop routing traffic to it. Other healthy instances in the pool continue serving requests without interruption. Users are unlikely to even notice a server has failed.
   • Contrast this with a stateful server: if it fails, all the session data it held in memory (e.g., shopping carts, login status for a group of users) is lost, leading to a poor user experience.

3. Simplified Server Design & Maintenance:

   • The code for stateless services is often simpler. Developers don't need to manage session creation, expiration, or replication. The focus is purely on the business logic of processing a single request.
   • Deploying updates is much easier. You can take down instances one by one to update them without worrying about disrupting active user sessions. This is a core tenet of modern CI/CD practices.

This ideal architecture is often depicted as follows:

This diagram brings us to the next logical question: if the state isn't on the application server, where does it go?

Strategies for Externalizing State

Designing stateless services doesn't mean your application has no state—far from it. It means you externalize the state, moving it from the application server's memory to a dedicated, shared backend system that is accessible to all your application servers.

The Microsoft Azure documentation provides a concise, practical guide on this concept.

As the article suggests, there are several common solutions for externalizing state. Let's discuss the two most prevalent ones for backend services.

1. Shared Database

This is a straightforward approach where you store session-like data directly in your primary database (e.g., PostgreSQL, MySQL, or a NoSQL database like MongoDB).

• How it works: When a user logs in, you create a session record in a sessions table in your database with a unique session ID. This ID is sent to the client as a cookie. On subsequent requests, your stateless application server gets the session ID from the cookie, queries the database to retrieve the session data, validates it, and proceeds.
• Use cases: Storing user profile information, shopping cart contents, or any state that requires strong consistency and durability.
• Trade-offs:
  • Pros: Data is durable (it survives restarts) and can be part of ACID transactions. It leverages your existing database infrastructure.
  • Cons: Can be slower than in-memory options. A high volume of session reads/writes can add significant load to your primary database, potentially creating a new bottleneck.

2. Distributed Cache / In-Memory Data Store

For state that is frequently accessed and doesn't require the same level of durability as your primary data (like session data), a dedicated in-memory store like Redis or Memcached is a superior choice.

• How it works: The flow is similar to the database approach, but instead of writing to a database on disk, the session data is written to a high-speed, in-memory key-value store. Redis is particularly popular for this because it offers data structures (hashes, lists) and persistence options.
• Use cases: Caching session data, real-time leaderboards, rate limiting counters. This is the most common solution for session management in large-scale systems.
• Trade-offs:
  • Pros: Extremely fast (sub-millisecond latency) because data is served from RAM. Reduces load on your primary database.
  • Cons: By default, data is volatile (can be lost on restart, though Redis offers persistence). It adds another component to your infrastructure that you need to manage.

To understand how this works in practice, let's look at how backend caching is implemented.

A third strategy, which we'll cover in depth in a later module on authentication, involves using client-side tokens like JSON Web Tokens (JWTs). In this model, the state (e.g., user ID, roles) is packaged into a cryptographically signed token that the client stores and sends with every request. The server remains stateless because it doesn't need to store anything; it simply validates the token's signature on each call.

Conclusion

In this lesson, we established that statelessness is a foundational principle for building scalable and resilient backend systems. By designing your application servers to be stateless, you unlock the full potential of horizontal scaling.

Key Takeaways:

• Stateless services treat each request independently, holding no client state between requests. Stateful services remember client context in memory.
• Stateless architecture is far easier to scale because any server can handle any request. This simplifies load balancing, eliminates the need for sticky sessions, and dramatically improves fault tolerance.
• State is not eliminated; it is externalized to dedicated, shared systems.
• Common strategies for externalizing state include using a shared database for durable state or a distributed in-memory cache like Redis for high-performance, ephemeral state like user sessions.

Preview of the Next Lesson

We've established that to build scalable systems, we need to externalize state to dedicated data stores. This immediately raises new questions: What kind of data store should you choose? A relational database? A NoSQL database? What are the trade-offs?

In our next lesson, we will explore one of the most fundamental concepts in distributed systems that helps answer these questions: the CAP Theorem. It describes an unavoidable trade-off between Consistency, Availability, and Partition Tolerance that every distributed data store must make. Understanding this will be critical to making informed architectural decisions.