Hello! Welcome back to your course on distributed systems design.
In our first lesson, we explored the foundational challenges of distributed systems: partial failures, unreliable networks, and unbounded latency. The common thread was uncertainty—a node in a distributed system can never be entirely sure about the state of other nodes.
Today, we'll confront another fundamental source of uncertainty: the lack of a single, shared source of truth for time. This lesson directly addresses the learning outcome: describing the implications of the lack of a global clock and synchronized time. We will investigate why physical clocks are unreliable, how we attempt to synchronize them, and what this means for you as an engineer designing and debugging distributed applications.
Why Time is Tricky in Distributed Systems
In your experience developing backend services, you've certainly relied on timestamps for logging, performance monitoring, cache invalidation, and more. On a single machine, time is straightforward. In a distributed system, it's a source of profound complexity. Each computer has its own physical clock, and these clocks are never perfectly aligned.
Let's start with a short video that poses a real-world puzzle caused by a time-related anomaly and outlines the many ways we use time in software systems.
Distributed Systems 3.1: Physical time
This segment from Martin Kleppmann's lecture series introduces the importance of time and presents a mysterious, widespread server failure that we will solve by the end of this lesson.
Watch from the beginning to 04:41. Pay attention to the variety of applications for time, from caching (like DNS TTLs) to security (like TLS certificate expiry).
The core problem is this: if every machine has a slightly different idea of the current time, how can we establish a definitive order of events? Imagine debugging an issue where a log message on server A has a timestamp of 10:00:01.500 and a related message on server B has a timestamp of 10:00:01.490. Did the event on server B really happen first? You can't be sure.
This discrepancy arises from two related phenomena: clock drift and clock skew.
Every computer clock, based on a vibrating quartz crystal, has some drift. Over time, these small differences accumulate, leading to significant skew between nodes in a cluster.
Attempting to Tame Time: The Network Time Protocol (NTP)
Since individual clocks are unreliable, the natural next step is to synchronize them with a more accurate time source. This is the job of the Network Time Protocol (NTP), a protocol that's almost certainly running on every server you've ever managed.
NTP organizes servers into a hierarchy of strata, where stratum 0 devices are high-precision sources like atomic clocks or GPS receivers. Servers in lower strata synchronize with servers in higher strata.
To understand how this works and its limitations, let's turn back to Martin Kleppmann.
Distributed Systems 3.2: Clock synchronisation
This video explains the principles of NTP, including its hierarchical structure and the clever four-timestamp method it uses to estimate clock skew despite network latency.
Watch from the beginning to 07:33. Focus on the high-level strategy (strata, querying multiple servers) and the logic behind the four-timestamp exchange. You don't need to memorize the formula, but you should understand how it attempts to isolate network delay from clock skew.
As you saw, NTP estimates the round-trip network delay and assumes that the latency is symmetrical (request time ≈ response time). It then calculates the clock skew and adjusts the local clock.
However, this is just an estimate. Asymmetric routing paths or network congestion can violate the assumption of symmetric latency, introducing errors. While NTP can reduce skew to a few milliseconds on a good network, it cannot eliminate it. This residual uncertainty means you still can't use timestamps to definitively order events that happen very close together on different machines.
The Perils of Time-of-Day Clocks
NTP's attempt to discipline clocks introduces its own set of problems, particularly for developers. Depending on the size of the skew, an NTP client will either slew the clock (speeding it up or slowing it down slightly) or step it (jumping the time forward or backward). These jumps can have bizarre consequences.
This brings us back to the puzzle from the beginning of the lesson. The culprit behind the massive 2012 outage was an even stranger source of time jumps: the leap second.
Distributed Systems 3.1: Physical time
Let's solve the puzzle. This video explains what leap seconds are, why they exist, and how a bug in handling a leap second brought down systems across the internet. It also discusses the pragmatic solution of 'leap second smearing'.
Watch from 14:10 to 20:47. This is a fantastic case study on how a seemingly obscure detail about time can have catastrophic effects in distributed systems.
The combination of NTP stepping and leap seconds makes one thing clear: the time reported by a standard system clock can suddenly jump, and it doesn't even flow at a constant rate. This has a critical implication for programming.
The Developer's Toolkit: Monotonic vs. Time-of-Day Clocks
If you need to measure an elapsed time interval—for a timeout, a retry delay, or performance profiling—using a standard "wall-clock" is dangerous. If the clock jumps backward during the interval, you could get a negative duration!
To solve this, modern operating systems and programming languages provide two types of clocks. You've likely encountered their equivalents in Python (time.time() vs. time.monotonic()) or Go.
-
Time-of-Day Clock (e.g.,
System.currentTimeMillis()in Java):- Represents calendar time (wall-clock time).
- Synchronized by NTP; subject to stepping and leap seconds.
- Use for: Timestamping events for human-readable logs; situations requiring a common point of reference in calendar time (e.g., checking a certificate's expiry date).
-
Monotonic Clock (e.g.,
System.nanoTime()in Java):- Guaranteed to always move forward.
- Starts from an arbitrary point (like system boot), so its absolute value is meaningless.
- Not affected by time-of-day clock jumps.
- Use for: Measuring elapsed time intervals on a single node.
The distinction is crucial for writing robust code.
Distributed Systems 3.2: Clock synchronisation
This final video segment explicitly contrasts these two types of clocks with a concrete code example, showing why using a time-of-day clock to measure duration is a bug waiting to happen.
Watch from 11:09 to 15:55. This directly translates the theoretical problems we've discussed into a practical programming pattern.
Test your understanding!
You are implementing a client for a REST API. You need to enforce a 500ms timeout on requests. Your programming language provides get_current_time(), which returns the Unix timestamp, and get_monotonic_time(), which returns nanoseconds since boot. Which function should you use to implement the timeout logic, and why? What could go wrong if you use the other function?
Show answer
You should use get_monotonic_time().
The logic would be:
start = get_monotonic_time()- Initiate the network request.
- In a loop, check if
get_monotonic_time() - start > 500_000_000. If so, abort the request and raise a timeout error.
If you used get_current_time(), your timeout logic could fail spectacularly. For example, if an NTP client steps the system clock backward by 1 second while the request is in flight, get_current_time() would suddenly return a smaller value. Your timeout calculation might never exceed 500ms, and the client could hang indefinitely waiting for a response that never comes. Conversely, a forward jump could cause a premature timeout.
Implications for Distributed System Design
The unreliability of physical clocks has profound consequences for system design, especially when dealing with data.
Chapter 8 - The Trouble with Distributed Systems
To consolidate these ideas, please read a few sections from a summary of the 'Designing Data-Intensive Applications' chapter on this topic. It reinforces the clock types and explores the design implications.
Read the sections 'Monotonic vs Time-of-Day Clocks', 'Relying on Synchronized Clocks', and 'Clock readings have a confidence interval'. Focus on how these clock issues affect real-world strategies like 'Last Write Wins'.
As the reading highlights, a common strategy for conflict resolution in distributed databases (like some configurations of Cassandra or Riak) is Last Write Wins (LWW). When two clients write to the same key concurrently, the write with the later timestamp "wins". This approach is simple but dangerous. If the clients' clocks are skewed, a causally earlier write might have a later timestamp, incorrectly overwriting a newer value and leading to silent data loss.
Recognizing that timestamps are inherently uncertain, some advanced systems like Google's Spanner use a TrueTime API. Instead of returning a single timestamp, the API returns a confidence interval [earliest, latest]. This allows the system to make definitive statements about causality. If transaction A's interval [A_earliest, A_latest] is entirely before transaction B's interval [B_earliest, B_latest], the system knows for sure that A happened before B. This requires specialized hardware (atomic clocks in each datacenter) to keep the uncertainty interval small (a few milliseconds), but it's a powerful approach to building globally consistent systems.
Conclusion
This lesson explored the deep complexities of time in distributed systems. Here are the key takeaways:
- There is no global clock: Each machine has its own physical clock, which drifts at its own rate, leading to clock skew.
- Synchronization is imperfect: NTP is used to synchronize clocks, but it's limited by network latency and can't eliminate skew. Furthermore, NTP's corrective "stepping" and astronomical "leap seconds" mean wall-clock time can jump forwards or backwards.
- Use the right clock for the job: As a developer, you must distinguish between time-of-day clocks (for absolute timestamps) and monotonic clocks (for measuring time intervals on a single node). Using the wrong one leads to subtle, hard-to-debug bugs.
- Timestamps don't equal truth: Relying on physical timestamps to determine the causal order of events between nodes is fundamentally unreliable and can lead to data loss in strategies like Last Write Wins.
We've established that physical time is too messy to be the foundation for ordering in a distributed system. So, what's the alternative?
Preview of the next lesson: We will introduce the concept of logical clocks. Instead of measuring seconds, logical clocks track causality itself by counting events. We will start with Lamport timestamps, a simple yet powerful algorithm for establishing a partial ordering of events across a distributed system, bringing a new kind of order to the chaos.