Skip to main content
Back to path

Welcome to a Grasp lesson

Create a free account to keep studying and create your own study path.

Lesson 65

2PC Failure and Blocking Scenarios

Recommended resources

Hello! Welcome back.

In our last lesson, we established the blueprint for the Two-Phase Commit (2PC) protocol, seeing how its prepare and commit phases work together to achieve atomic transactions across multiple database nodes. We ended on a critical observation: 2PC is a blocking protocol, and its greatest weakness is the potential for the system to grind to a halt if the coordinator fails at the wrong time.

Today, we're going to put that claim under the microscope. Your learning outcome is to analyze failure and blocking scenarios in the 2PC protocol. We will move beyond the "happy path" and dissect what happens when participants or the coordinator crash at various points in the protocol. Understanding these failure modes is crucial for any systems engineer deciding whether 2PC is the right tool for a job, as it forces a direct confrontation with the trade-offs between consistency and availability.

A Systematic Look at Failure

To analyze failures methodically, we need to consider two main actors: the participants (which the CMU literature calls Resource Managers or RMs) and the coordinator (the Transaction Manager or TM). The impact of a crash depends heavily on who crashes and when.

Let's start with a structured overview from the CMU course notes, which neatly categorize the primary failure cases.

Lecture #12: Consensus on Transaction Commit

These lecture notes provide a concise, structured list of failure cases and their outcomes in a classic 2PC implementation. This will give us a framework for our analysis.

Please read the subsection '3.1.3 Failure Cases' on page 2. As you read, notice the distinction between a participant (RM) failure and a coordinator (TM) failure, and how the protocol's state determines the outcome.

This reading lays out the rules of recovery. Let's expand on them.

Participant Failures

A participant crash is the less severe of the two failure types, but it can still be problematic.

  1. Crash before voting: If a participant crashes before it has logged a READY state and sent its VOTE-COMMIT, the coordinator will simply time out waiting for its vote. The coordinator will then decide to abort the entire transaction. When the participant recovers, it will see no READY record in its log and will also abort. This is a clean, non-blocking failure.

  2. Crash after voting VOTE-COMMIT: This is the scenario we hinted at in the last lesson. The participant has made a promise to commit and logged its READY state. If it crashes now, upon recovery it finds itself in a state of uncertainty. It knows it promised to be ready, but it doesn't know the coordinator's final decision.

    • It is blocked. It cannot unilaterally abort (as that would break its promise) or commit (as another participant might have voted to abort).
    • It must hold its locks and wait until it can re-contact the coordinator to learn the final outcome. If the coordinator is also down, the participant remains blocked indefinitely.

Coordinator Failures: The Single Point of Failure

The coordinator is the Achilles' heel of 2PC. Its failure can bring the entire transaction to a standstill.

Distributed Systems 7.1: Two-phase commit

The reading gave us a list of failure scenarios. Now, let's focus on the most problematic one: the coordinator crashing at just the wrong moment. Martin Kleppmann provides an excellent explanation of why this leads to a 'blocked' state.

Watch the segment from 09:54 to 12:39. Pay close attention to the state of the participants after they have voted 'yes' but before the coordinator makes a decision. Why can't they just decide to abort on their own after a timeout?

As the video explains, the critical failure occurs when the coordinator crashes after sending PREPARE but before logging and broadcasting its final decision.

In this situation, any participant that voted VOTE-COMMIT is now blocked. They are stuck in the READY state, holding locks on resources, with no way to find out the transaction's outcome.

Why can't the participants just talk to each other and come to a decision? Imagine a transaction with coordinator C and participants P1, P2, and P3.

  • C sends PREPARE to all three.
  • P1 and P2 vote VOTE-COMMIT.
  • P3 votes VOTE-ABORT.
  • C receives all votes and is about to decide to ABORT, but it crashes before it can tell anyone.

Now, P1 and P2 are blocked. If they were to communicate only with each other, they would both see that the other is ready to commit. If they decided to go ahead and commit, they would violate atomicity, because P3 has already decided to abort. This is why they are forced to wait for the original coordinator—it is the only entity that had the complete picture.

The following reading reinforces this exact point.

Lecture #12: Consensus on Transaction Commit

To reinforce what the video explained, these notes explicitly describe why the coordinator is a single point of failure and how its crash paralyzes the system.

Read the short subsection '3.1.4 Transaction Manager is a single point of failure' on page 3. It concisely explains why participants are 'stuck' and unable to make progress without the coordinator.

The Full Picture of Recovery and Blocking

Let's watch one more video segment that provides a conversational walkthrough of these scenarios, helping to tie all the pieces together.

#23 - Distributed Transactional Databases (CMU Intro to Database Systems)

Professor Andy Pavlo from CMU also provides a great walkthrough of these failure scenarios in a lecture setting. This will help solidify your understanding of how recovery works from both perspectives.

Watch the segment from 00:55:07 to 00:58:40. He discusses recovery from the perspective of both the participant and the coordinator and what happens in each crash scenario.

This walkthrough highlights the importance of the write-ahead log. In every case where recovery is possible, it's because a node can look at its durable log to determine its state before the crash (READY, COMMIT, ABORT) and act accordingly. The blocking scenario is so pernicious precisely because the log contains a question (READY) but not the answer.

Test your understanding!

A distributed transaction involves a coordinator (C) and three participants (P1, P2, P3).

  1. C sends PREPARE to all three.
  2. P1 and P2 receive the message, log READY, and send VOTE-COMMIT back to C.
  3. The network connection to P3 is lost, so P3 never receives the PREPARE message.
  4. C receives votes from P1 and P2, but then its machine suffers a power failure before it can time out waiting for P3 or log a decision.

What is the state of the system, and what happens next?

  • What is the state of P1 and P2? Can they do anything?
  • What is the state of P3?
  • What will be the eventual outcome of this transaction, and why?
Show answer
  • State of P1 and P2: They are blocked. They are in the READY state, holding locks on their resources. They cannot proceed because they are waiting for a final decision from the now-crashed coordinator.
  • State of P3: P3 is completely unaware of the commit attempt. It holds no locks and is operating normally. Its state is consistent with an aborted transaction.
  • Eventual Outcome: The transaction must eventually abort. When the coordinator C recovers, it will check its log. It will see that it initiated a transaction but will find no logged decision of COMMIT. A recovering coordinator that has not logged a COMMIT decision must decide to abort to be safe, as it cannot be sure that all participants were ready. It will then send an ABORT message to P1, P2, and P3, allowing P1 and P2 to unblock and roll back.

Conclusion

Today we explored the fragility of the Two-Phase Commit protocol. While it correctly guarantees atomicity, it does so at a potentially high cost to availability.

Key Takeaways:

  • Participant Failures: A participant crashing before voting leads to a clean abort. A participant crashing after voting COMMIT becomes blocked until it can learn the final outcome from the coordinator.
  • Coordinator Failures: The coordinator is a single point of failure. If it crashes after participants have entered the READY state but before it has communicated a final decision, those participants are blocked.
  • Blocking: This is the critical weakness of 2PC. Blocked participants hold locks and cannot proceed, which can cause cascading failures and reduce system availability. The system must wait for the coordinator to recover. If the coordinator is permanently lost, manual intervention is required.
  • The Role of Logs: Recovery from non-blocking failures relies entirely on nodes having durably logged their state (READY, COMMIT, ABORT) before sending messages.

The blocking behavior of 2PC is a classic example of a system prioritizing Consistency over Availability when a Partition occurs (in this case, the coordinator becoming unreachable). This trade-off is fundamental to distributed systems.

Preview of the Next Lesson

Our analysis of 2PC's blocking behavior provides the perfect motivation for our next topic. We will step back from specific protocols and look at a foundational principle of distributed systems. In the next lesson, we will explain the trade-offs defined by the CAP theorem (Consistency, Availability, Partition Tolerance). You will see how the dilemma faced by 2PC's designers is a specific instance of a much more general and unavoidable choice that all distributed systems must make.