Hello! Welcome to the fifth lesson in our module on the foundations of distributed systems.
In our last two lessons, we delved into the intricacies of time and ordering, learning how Lamport timestamps establish a partial order and how vector clocks give us the power to determine the full causal relationship between events. These tools are crucial for understanding what happened in a system and in what order.
Now, we're going to build on that by learning the language to describe what a system guarantees about its behavior. This lesson addresses the learning outcome: Distinguish between safety and liveness properties in system guarantees.
Understanding this distinction is fundamental. Almost every major design decision in a distributed system involves a trade-off between these two types of properties. This framework will give you a precise vocabulary to analyze and discuss these trade-offs, moving beyond vague terms like "reliability" or "performance."
Defining Safety and Liveness
At a high level, the distinction is quite intuitive:
- Safety: "Nothing bad ever happens."
- Liveness: "Something good eventually happens."
While these informal definitions are a great starting point, the formal definitions are more precise and useful. Let's read a clear explanation from a summary of the book Designing Data-Intensive Applications.
Chapter 8 - The Trouble with Distributed Systems
The following text defines safety and liveness properties, first informally and then with more precise, practical definitions. It uses the example of a fencing token service to make the concepts concrete.
Read the subsections titled 'Correctness of an algorithm' and 'Safety and liveness'. Focus on the distinction between when a violation can be identified for each property.
As you just read, the key difference lies in how we can observe a violation:
-
A safety property is violated at a specific moment in time. If a system promises unique transaction IDs (a safety property) and it produces a duplicate, we can point to that exact operation and say, "The bad thing happened right here." The damage is done and cannot be undone.
-
A liveness property, on the other hand, can only be violated over an infinite execution. If a system promises that "every request will eventually get a response," and a request hasn't received one after five minutes, has the property been violated? We don't know. The response might arrive in the next millisecond. We can only claim a liveness failure if we know the response will never arrive.
Let's look at some examples relevant to your work as a backend engineer:
| Property Type | Example | Why? |
|---|---|---|
| Safety | In a financial ledger, the sum of debits and credits in a transaction must be zero. | If a transaction is created where sum(debits) != sum(credits), a "bad thing" (money created/lost) has occurred. |
| Safety | A database with UNIQUE constraints will never allow two rows to have the same value in a constrained column. |
If a duplicate value is inserted, the integrity of the data is compromised at that specific point in time. |
| Liveness | A consumer in a Kafka-based system will eventually process a message that has been published to a topic. | The processing may be delayed, but we expect it to "eventually" happen. The system is stuck if it never does. |
| Liveness | In a system with leader election, if the current leader fails, a new leader will eventually be elected. | The election might take time, but the system is unhealthy if it gets stuck without a leader indefinitely. |
The Inherent Trade-Off
A system can be perfectly safe by doing nothing. Imagine a traffic light that is always red. It's perfectly safe—it will never cause an accident. However, it has zero liveness—no car will ever get through the intersection.
This simple analogy illustrates one of the most fundamental tensions in distributed systems design. To make progress (liveness), we often have to take actions that carry some risk of violating safety. Conversely, making a system perfectly safe can sometimes bring it to a halt.
Let's hear Martin Fowler discuss this exact point.
The Patterns Of Distributed Systems With Martin Fowler
In this short clip, Martin Fowler frames the safety vs. liveness discussion as a classic, unavoidable trade-off in any concurrent or distributed system.
Watch from 13:01 to 13:51. Pay attention to how he defines the trade-off and why it's more acute in distributed systems due to partial failures.
As Fowler notes, this trade-off is particularly challenging in distributed systems because of partial failures. A single-process application typically "all falls over or none of it does." In a distributed system, one node can fail while others continue, leading to complex scenarios where maintaining both safety and liveness is difficult.
Case Study: Safety and Liveness in Consensus
The consensus problem, which is about getting multiple nodes to agree on something, is a perfect real-world illustration of this trade-off.
- Safety in Consensus: All nodes that decide on a value must decide on the same value. Two nodes deciding on conflicting values would be a "bad thing."
- Liveness in Consensus: Every node that does not crash eventually decides on a value. The process must eventually terminate and make progress.
The famous FLP Impossibility Result proves that in a fully asynchronous system (where messages can be arbitrarily delayed) with even one faulty process, there is no consensus algorithm that can guarantee both safety and liveness.
In practice, algorithms like Raft and Paxos make a crucial compromise. They always guarantee safety. They will never allow a state of disagreement. However, to guarantee liveness (that they will eventually make progress), they must rely on a partially synchronous model—they use timeouts to detect failures and drive the system forward. If the network is consistently unstable and timeouts are not reliable, the system might not make progress, but it will remain safe.
Martin Kleppmann explains this distinction very clearly.
Distributed Systems 6.1: Consensus
Here, Martin Kleppmann discusses consensus algorithms in the context of different system models. He explicitly points out how safety and liveness are treated differently.
Watch from 08:20 to 09:26. Note his key point: the correctness (safety) of the algorithm doesn't depend on timing, but the liveness (making progress) does.
This is a profound point: a well-designed distributed algorithm prioritizes safety above all else. Liveness may be conditional (e.g., "if the network stabilizes"), but safety must be absolute.
Test your understanding!
Classify each of the following system guarantees as either a safety or a liveness property.
- In a distributed lock service, at most one client can hold the lock at any given time.
- A web server will eventually respond to every HTTP request it receives.
- In an eventually consistent database, all replicas of a data item will eventually hold the same value.
- A transaction in a database is atomic (i.e., it either commits fully or aborts fully, never leaving the database in a partially-updated state).
- If a client submits a message to a queue, it will not be permanently lost before being processed.
Show answer
- Safety. "At most one client" is a state that must never be violated. If two clients hold the lock simultaneously, something bad has happened.
- Liveness. "Eventually respond" is the key phrase. The system is live if it makes progress on requests.
- Liveness. "Eventually hold the same value" implies that convergence is a future goal. The system can be in a state of divergence temporarily.
- Safety. Atomicity prevents the "bad thing" of a partially applied transaction, which would corrupt the system's state.
- This is subtle and can be argued both ways, but is most often treated as a safety property. It can be framed as "nothing bad happens" (a message is not lost). You can often identify the point where the message was dropped. Some might argue it's liveness (the message eventually gets processed), but the guarantee against permanent loss is a stronger, safety-like assertion about the system's integrity. This ambiguity shows how precise definitions are important!
Why This Matters in Practice: Finding Flaws
This distinction isn't just academic. When designing and verifying complex systems, thinking in terms of safety and liveness helps uncover subtle bugs. Safety properties are often easier to check ("Is there a duplicate primary key right now?"), but liveness properties often uncover more complex design flaws like deadlocks, livelocks, or logic cycles where the system gets stuck.
This final resource gives a glimpse into how these concepts are used in formal verification to build more robust systems.
The importance of liveness properties (with TLA+ Part 1)
This blog post, aimed at engineers using formal methods, introduces safety and liveness from a practical standpoint. It reinforces the definitions and explains why liveness properties are especially good at finding tricky bugs and design flaws.
Read the first two sections, 'Liveness properties, a surprisingly useful (and necessary) tool' and the definitions that follow. You can ignore the TLA+ specific syntax; focus on the concepts and the reasoning for why liveness is so important for finding bugs.
Conclusion
In this lesson, we've established a formal vocabulary for discussing the guarantees that a distributed system provides. This framework is essential for reasoning about complex system behaviors and their trade-offs.
Key Takeaways:
- Safety properties state that nothing bad ever happens. They are about correctness and integrity. A violation is a specific, observable event (e.g., duplicate data, inconsistent state).
- Liveness properties state that something good eventually happens. They are about progress and termination. A failure is the indefinite postponement of a desired outcome (e.g., a request is never answered, a leader is never elected).
- There is a fundamental trade-off between safety and liveness. Many distributed algorithms, like those for consensus, prioritize absolute safety while providing conditional liveness (i.e., the system will make progress if the network is stable enough).
- This vocabulary is a critical tool for system design, allowing you to precisely articulate the guarantees your system provides and to analyze potential failure modes.
Preview of the next lesson:
Now that we have the concepts of safety and liveness, we are perfectly equipped to tackle one of the most famous principles in distributed systems: the CAP theorem. We will see that the theorem is, in essence, a specific statement about the trade-off between a safety property (Consistency) and a liveness property (Availability) in the face of network partitions.