Understanding Distributed System Failures

Hello! Welcome to the final lesson of our first module, "Foundations of Distributed Computing."

In our previous lessons, we've established the fundamental trade-offs in distributed systems (CAP and PACELC theorems) and explored how consensus algorithms like Raft and Paxos enable a group of servers to agree on a shared state, providing the strong consistency and fault tolerance required for critical applications.

We've talked a lot about "tolerating failures," but we haven't formally defined what those failures are. A server crashing is very different from a server being hacked and maliciously sending false data. The strategies to deal with them are also vastly different.

Today, we will build a precise vocabulary for discussing system failures. This is the final piece of our foundational knowledge before we move on to building systems with specific communication patterns.

Learning Outcome:

Define network partitions, node failures, and Byzantine failures, providing concrete examples of how each could manifest in a high-stakes financial system.

We will dissect these three major classes of failures, using your experience with financial platforms to ground the concepts in concrete, high-stakes scenarios.

1. A Taxonomy of Failures

In a distributed system, things can go wrong in two main places: with the individual computers (nodes) or with the communication links between them (network). This distinction is a useful starting point for classifying failures.

Let's begin with a video that introduces these broad categories and drills down into the most common failure types.

To build on the video's explanation, let's formally define these failure types. We'll use a more detailed classification that distinguishes between how a node stops working.

Now, let's synthesize this information and apply it to a financial context.

Node Failures (Crash/Fail-Stop)

This is the most common and intuitive type of failure. A component simply stops working.

• Definition: A node failure occurs when a process or server in the system halts. This can be a crash failure (it stops silently) or a fail-stop failure (it stops and its failure is detectable by other nodes, often via missed heartbeats).
• Financial System Example: Imagine the FX trading platform you built at Revolut. A specific microservice responsible for pre-trade credit checks experiences an unhandled exception (e.g., NullPointerException or OutOfMemoryError) and crashes. Other services that depend on it start experiencing timeouts. A monitoring system or service orchestrator (like Kubernetes) detects the missed health checks, declares the node dead, and redirects traffic to a replica instance. This is a classic example of tolerating a crash failure through redundancy and failover, as described in the video.

Network Partitions

This failure is more subtle. The nodes are fine, but they can't talk to each other.

• Definition: A network partition occurs when a communication breakdown splits the system into two or more groups of nodes that cannot communicate with each other. The nodes within each partition are still running but are isolated from the other partitions.
• Financial System Example: Consider the global clearing and settlement system you worked on. Let's say it operates across data centers in London and New York. A failure in the transatlantic network cable (the "backhoe fade" scenario from the video) severs the link between the two data centers.
  • The London partition can communicate with European banks, and the New York partition can communicate with US banks. Both partitions are "up," but they are blind to each other.
  • This creates a "split-brain" scenario. If both partitions were allowed to continue processing settlements, their ledgers would diverge, creating a reconciliation nightmare.
  • As we discussed in the lesson on the CAP theorem, the system must now choose. A CP (Consistency/Partition Tolerance) system would likely have one partition (e.g., the one without the primary leader) halt write operations to prevent divergence, sacrificing availability in that region to maintain global consistency.

2. The Ultimate Challenge: Byzantine Failures

Node crashes and network partitions are challenging but predictable. The nodes either work correctly or they stop. Byzantine failures break this assumption.

• Definition: A Byzantine failure occurs when a node continues to run but behaves incorrectly or maliciously. It can send corrupted data, send conflicting information to different peers, or arbitrarily deviate from the protocol. It is the most general and difficult class of failure to handle.

The name comes from a famous thought experiment.

As the video explains, you need at least 3f + 1 total nodes to guarantee agreement if f nodes are Byzantine. This means to tolerate just one malicious node, you need a system of at least four nodes. This high overhead is one reason why true Byzantine Fault Tolerance (BFT) is not used unless absolutely necessary.

Now, let's see how this abstract problem manifests in the real world.

Building on this, here are two concrete examples in a high-stakes financial context:

• Example 1: Non-Malicious Byzantine Failure (Software Bug)
  In a high-frequency trading system, a price aggregator service collects quotes from multiple exchanges. A subtle floating-point precision bug is triggered only when processing quotes for the USD/JPY pair during extreme market volatility. This causes the service to send a slightly incorrect, but still plausible, price to the algorithmic trading engine, while simultaneously reporting the correct price to the compliance and logging service. The node hasn't crashed; it appears healthy but is corrupting state by sending conflicting information, a classic Byzantine behavior.

• Example 2: Malicious Byzantine Failure (Attack)
  Consider a decentralized finance (DeFi) protocol or a consortium blockchain for interbank settlements, which relies on a set of validators to approve transactions. An attacker gains control of one of the validator nodes (f=1). To approve a large transfer, 2/3 of the validators must sign off. Suppose there are four validators in total (3f+1 = 4). The attacker wants to execute a fraudulent transfer. The malicious node sends an "APPROVE" message to one honest validator and a "REJECT" message to the other two. From the perspective of the honest nodes, there is no clear consensus, and they cannot determine who is lying. The system may stall or, in a poorly designed protocol, reach an inconsistent state. This is where BFT consensus algorithms (like PBFT) are critical.

3. Synthesis and Relation to Consensus

We can summarize the failure models in a table:

This brings us full circle to our last lesson. The consensus algorithms we discussed are designed for specific failure models:

• Raft and Paxos are designed to handle crash failures and network partitions. They assume that nodes are honest—they follow the protocol correctly and only fail by stopping. They cannot handle Byzantine failures.
• Byzantine Fault Tolerant (BFT) algorithms, such as Practical Byzantine Fault Tolerance (PBFT) or the algorithms used in many cryptocurrencies, are explicitly designed to handle Byzantine failures in addition to crashes and partitions.

The choice of which consensus protocol to use depends entirely on your system's threat model. For internal systems where you control all the nodes (like most services within a company like Revolut), assuming non-Byzantine failures is usually a safe and efficient trade-off. For systems involving multiple untrusting parties (like a public blockchain or a consortium of banks), BFT is often a requirement.

Conclusion

This lesson concludes our first module on the foundations of distributed computing. We have moved from identifying distributed characteristics to understanding the core trade-offs (CAP/PACELC), the solution to consistency (consensus), and finally, the types of failures these solutions address.

Key Takeaways:

• Node Failures occur when a component stops working. They are typically handled with redundancy and failover mechanisms.
• Network Partitions split a system into isolated subgroups, risking a "split-brain" scenario. They force a trade-off between consistency and availability.
• Byzantine Failures are the most severe, where a node behaves arbitrarily or maliciously. Tolerating them requires specialized BFT algorithms and significant overhead (3f+1 nodes to tolerate f failures).
• The choice of a fault-tolerance strategy, particularly the consensus algorithm, is dictated by the failure model you assume for your system.

Preview of the Next Module:

With this foundational theory in place, we are now ready to start building. In the next module, "Distributed Communication Patterns," we will get practical. Our first lesson will focus on implementing the most common pattern of all: the synchronous request-response pattern, and how to make it robust against the very failures we've just discussed.