Tracing HMS Catalog Entries to Immutable Table Metadata Versions

Hello again. Last time, we separated the table name in HMS from the files stored in S3. Now we will follow the first actual load: starting with an HMS table record, finding its metadata_location, and reading one particular table metadata JSON file.

This is a useful stopping point for your backup-service investigation. Before asking which snapshot or files the service backs up, you need to know which metadata version supplied its view of the table.

The catalog lookup selects a file

Suppose the service needs to load analytics.trades. An HMS Thrift call returns a table record containing, among other things, a map of table parameters. For an Iceberg table managed through HMS, the parameter to look for is metadata_location. Its value is a file location, not the JSON contents:

HMS table: analytics.trades
parameter: metadata_location
value: s3://lake/warehouse/analytics/trades/metadata/00008-a1b2.metadata.json

The service must then read that exact object from storage. HMS resolves the name; the metadata JSON supplies the table state. This differs from reading the table’s general location, which identifies a base storage location but does not select a metadata version.

A first-pass trace in the repository might look like this pseudocode:

HmsTable table = hmsClient.loadTable("analytics", "trades");
String metadataLocation = table.parameters().get("metadata_location");

byte[] json = objectStore.read(metadataLocation);
TableMetadata metadata = deserialize(json);

Your repository may use different methods, URI schemes, or deserialization classes. The invariant to look for is simpler: which value from the HMS response becomes the location of the JSON read? If metadata_location is missing, guessing the newest-looking JSON filename or listing an S3 prefix is not an equivalent load.

What the selected JSON tells you

The location above selects one table metadata version. That JSON contains table-wide information, including format and schema information, partition specifications, properties, and snapshot information. Here is a deliberately incomplete illustration of fields you might inspect:

{
  "format-version": 2,
  "table-uuid": "example-table-uuid",
  "location": "s3://lake/warehouse/analytics/trades",
  "current-schema-id": 7,
  "default-spec-id": 0,
  "last-updated-ms": 1730000000000,
  "current-snapshot-id": 912345,
  "snapshots": [
    {
      "snapshot-id": 912345,
      "manifest-list": "s3://lake/warehouse/analytics/trades/metadata/snap-912345.avro"
    }
  ]
}

This is a selected-fields illustration, not a complete valid metadata document. Notice the two different locations: HMS’s metadata_location identifies this JSON file, while the JSON’s location is the table’s base location. Substituting one for the other would change what the service reads.

Also notice that current-snapshot-id is a value inside the selected metadata version. It is not a separate “latest snapshot” value to discover by scanning S3. A metadata file can contain multiple retained snapshot records, but its current-snapshot ID selects the current one for that version. A table can also have no current snapshot, so code should not assume the field always identifies a populated table.

An important distinction follows: a metadata version is not a snapshot. The JSON version describes the table more broadly. For instance, changing a table property or schema can require a new metadata JSON file without creating a new data snapshot. Conversely, a metadata version that adds a snapshot can retain records of earlier snapshots. We will examine snapshot records themselves in the next module; today, the goal is to know which JSON supplied them.

Why capturing the location gives you a stable metadata view

Iceberg does not normally update the selected JSON object in place when table state changes. A change produces a new metadata file; the catalog can then point to the new file. The old file may still exist.

Take the earlier example. At the moment of the HMS read, metadata_location is:

s3://lake/warehouse/analytics/trades/metadata/00008-a1b2.metadata.json

Later, a successful table update might cause HMS to point to:

s3://lake/warehouse/analytics/trades/metadata/00009-c3d4.metadata.json

If the backup service saved the first location and reads 00008-a1b2.metadata.json, it reads the version it selected—even if a later HMS lookup would return 00009-c3d4.metadata.json. Reading the old JSON does not silently turn it into the new JSON.

For a short visual account, revisit just the update portion of the architecture video from the previous lesson:

Stable view needs one qualification: pinning a location means later commits do not change the contents you selected. It does not, by itself, guarantee that an old object remains available forever; retention and cleanup matter for backups and will be addressed later. Nor does it help if the service reads HMS a second time and accidentally combines information obtained from two different metadata versions.

A precise trace to look for in the service

When you inspect the unfamiliar repository, try following a single table load without yet chasing every manifest or data-file fetch. Record four concrete facts:

1. HMS response: Which Thrift call returns the table, and where in its returned record are the parameters read?
2. Captured value: Is metadata_location extracted and retained as a specific string or URI? Distinguish it from the table base location and any storage-descriptor location.
3. Object read: Which method fetches the JSON at that captured location? Identify where the HMS lookup ends and the S3 or file-I/O read begins.
4. Parsed state: Which class receives the JSON, and where does code read its current-snapshot ID or snapshot records?

For example, if you find two methods that each load analytics.trades from HMS, mark both. Do not assume they return the same pointer at different times. Equally, if a method receives a Snapshot object, find the metadata JSON from which that object was deserialized. Its class name alone does not establish which table version it belongs to.

At the end of this trace, you should be able to write a concrete statement such as:

“This invocation read metadata_location from the HMS table parameters, fetched the JSON at that exact URI, and obtained snapshot ID 912345 from that JSON.”

That is more informative than “the service loaded the latest Iceberg table.” It identifies the version the service actually observed and gives you a basis for checking later reads.

Key takeaways

HMS resolves an Iceberg table name to a metadata_location; the service then reads the JSON file at that exact location. The JSON is one immutable metadata version containing table configuration and snapshot information. Its base location is not a replacement for the HMS pointer, and a metadata version is not the same thing as a snapshot.

Next, we will classify the JSON, manifest lists, manifests, data files, and delete files by what each contains and whether it changes in place. That will make the rest of the backup’s object reads easier to recognize.