Recommended resources
The freely available resources this lesson will direct you to.
Hello again. Last lesson, you saw why a bucket value describes a file’s partition, not its S3 directory or its membership in a snapshot. Before that, you traced a snapshot through manifest lists and manifests to its files. The question now is: how does Iceberg make a new version of that whole structure current without exposing a half-finished update?
This begins the module on concurrent commits and backup consistency. In about 40 minutes, you’ll learn what a writer publishes, what happens when two writers race, and why a backup can keep reading an older version after a new snapshot is committed.
The commit changes what the table points to
An Iceberg table’s metadata JSON file describes a table version: among other things, it records snapshot information and identifies the current snapshot. The catalog provides the location of the current metadata file. In the setup you are investigating, that catalog is HMS and the file may be fetched from S3.
The key design choice is that a writer does not edit the current metadata file in place. It writes new objects—including any needed data files, manifests, manifest list, and table metadata file—while the catalog still points to the old metadata. Those new objects can physically exist without yet being part of the current table state. The commit publishes the new state by changing the catalog’s metadata pointer.
In the diagram, the two metadata files are not two halves of one file being rewritten. They are distinct versions. The right-hand version can include an older snapshot alongside a new one, reusing objects from the older file graph. The catalog’s current pointer determines which metadata version a fresh table load sees.
Read the specification’s short explanation of the publication rule:
Apache Iceberg’s specification explains why writing a new metadata file and publishing its location are separate acts.
In “Overview,” read table state. Then, in “Optimistic Concurrency,” read the commit model. Focus on the difference between creating a candidate metadata file and making it current, and on what a reader keeps using until it refreshes.
Immutability makes the reader’s view stable; the catalog change makes the writer’s update visible. If a reader has already loaded a particular metadata file and follows the snapshot referenced there, another writer does not alter that file beneath it. A fresh reader—or a reader that refreshes—can discover a newer metadata location. This stability of references is separate from how long physical files will be retained; later lessons will address cleanup.
Why the pointer change must be conditional
Imagine the catalog currently points to metadata file M0. Two writers, A and B, both load M0. Each prepares a valid-looking replacement:
- A writes candidate metadata file
MA, based onM0. - B writes candidate metadata file
MB, also based onM0. - A publishes
MA. - B tries to publish
MB.
If B simply overwrote the catalog location, B’s metadata could omit A’s committed update. The two files are individually well formed, but B built its version from a state in which A’s change did not exist. Immutable files alone do not prevent a lost update.
Instead, B’s publication must mean: make MB current only if M0 is still current. A successful publication is the commit’s linearization point: the moment the new table version becomes current. Since A has already committed, B’s condition fails. B must load the newer state and decide whether its operation can be reapplied.
This resembles AtomicReference.compareAndSet(expected, replacement) in Java at the level of the condition. It is not a JVM reference, though: the check and publication need a guarantee from the catalog’s implementation that works across processes. A database-backed catalog might use a conditional update. A catalog implementation backed by HMS may use a metastore lock to serialize committers and verify the expected metadata location while holding that lock. An unguarded Thrift read followed by an unguarded update would not provide the same guarantee.
The Iceberg specification describes the metastore-pointer approach without prescribing one universal catalog mechanism:
The specification uses a metastore table to show how a newly written metadata file becomes the current version without losing a concurrent commit.
Find “Metastore Tables” beneath “Commit Conflict Resolution and Retry.” Read the metastore procedure, from its opening paragraph through the numbered commit steps, stopping before “Notes.” Track three distinct locations or roles: the base metadata file, the newly written candidate file, and the pointer stored by the metastore. Notice what the failed writer must do when that pointer has changed.
“Atomic swap” describes the effect at the table boundary, not an instruction to rename an S3 object or rewrite an existing metadata JSON file. The exact conditional-update or locking primitive is catalog-specific. When reading repository code, it matters whether a call is merely reading HMS’s current metadata_location or is part of a catalog operation that safely commits a replacement.
A failed commit is not necessarily a failed operation
After B loses the race, its candidate metadata must not become current unchanged. But B’s intended work might still be valid. Iceberg uses optimistic concurrency: writers do substantial work without holding a table-wide lock throughout that work, then validate and publish against the current version. If publication fails, they check whether the operation’s assumptions still hold before rebuilding its metadata on the new base.
For example, suppose A and B each append different files. After A commits, B can generally prepare a new metadata version that includes the current table state and B’s appended files. B need not assume its first candidate metadata file somehow acquired A’s changes.
A file rewrite has a stronger precondition. If B compacts two source files, but A’s intervening commit removes one of those files, B cannot safely publish a replacement that assumes both sources are still present. The retry must validate the sources; if that condition fails, the operation cannot simply be rebased.
Apache Iceberg’s Reliability guide gives a compact account of retries and a concrete file-rewrite example.
In “Concurrent write operations,” read concurrent commits. Continue through “Cost of retries” and “Retry validation,” ending at the compaction example. Focus on which prepared work may be reused and which assumptions must be checked again.
A failed pointer publication can leave candidate files in storage even though no current snapshot references them. Their physical existence is not proof that the write committed. Conversely, success is not determined by whether a writer finished uploading its data files; it is determined by whether its new metadata version was safely published as current.
What this means for a backup reader
A backup is a reader in this commit protocol, not a competing table writer. Suppose it loads metadata file M0 and selects snapshot S0. While it fetches that snapshot’s manifest list and files, a writer may commit new metadata file M1 with a new current snapshot.
The backup does not automatically switch snapshots: M0 and S0 remain the identifiers it selected. It can, however, create an inconsistent result if its own code later rereads HMS for the latest metadata location, takes a snapshot ID from M1, and combines that with references collected from M0. The commit protocol gives the backup a stable version to use; it does not make a sequence of independent “get latest” calls refer to the same version.
That is why the boundary between loading the current table state once and following references within that loaded state will matter when you inspect the service. There is also a separate lifetime question: files belonging to an older snapshot may eventually be removed through expiration and cleanup. A stable metadata reference does not, by itself, promise that every referenced S3 object will remain available indefinitely.
Key takeaways
An Iceberg writer prepares immutable files before commit, then makes a new table metadata file current through a catalog-protected pointer change. The change must verify that the writer’s base version is still current—whether through a conditional update or an appropriate locking mechanism—so simultaneous commits cannot silently overwrite one another. A losing writer reloads, validates its assumptions, and retries only when its operation remains safe.
Next, you’ll put a backup and a new snapshot commit on the same timeline. The goal is to identify exactly which reads stay pinned to one version and which rereads could accidentally mix versions.