Understanding Bronze, Silver, and Gold Tables

What is the Medallion Architecture?

The medallion architecture is a data design pattern that organizes data into three distinct layers based on their quality and structure: Bronze, Silver, and Gold. It is a multi-hop architecture, meaning data "hops" from one layer to the next, becoming progressively more refined and valuable at each stage.

The core idea is to incrementally improve data quality. You start with raw data, clean it up, and then aggregate it for business use. Given your physics background, you can think of this as analogous to processing experimental data:

• The Bronze layer is like the raw, uncalibrated output from a detector.
• The Silver layer is where you apply calibrations, remove noise, and reconstruct meaningful events.
• The Gold layer contains the final, aggregated physics results used for analysis and publication.

To get a quick overview of this concept, please watch the following short video.

Now, let's dive into the specifics of each layer. We will use the official Microsoft Azure Databricks documentation as our primary guide.

The Bronze Layer: Raw Data Ingestion

The journey begins at the Bronze layer. This is the first stop for data entering your lakehouse.

• Purpose: To ingest and store data from source systems in its original, raw format.
• Characteristics:
  • Raw and Unaltered: Data is kept as-is, with no transformations or cleansing. This preserves the full history and fidelity of the source.
  • Append-Only: New data is added incrementally, building a historical archive over time.
  • Source of Truth: It serves as the single source of truth. If there's an error in a downstream pipeline, you can always reprocess from the raw Bronze data without having to re-ingest from the source system.
• Users: Primarily data engineers and operations teams who manage the ingestion process.

To understand the role and characteristics of this layer in more detail, please read the following section from the documentation.

The Silver Layer: Cleansed and Conformed Data

Once data lands in the Bronze layer, the next step is to move it to the Silver layer. This is where the first significant transformations happen.

• Purpose: To provide a validated, enriched, and queryable view of the data.
• Characteristics:
  • Cleaned and Validated: Data from the Bronze layer is processed to handle issues like null values, incorrect data types, and inconsistencies.
  • Enriched and Conformed: Data from different sources is often joined together to create a more complete view. For example, you might join customer data with transaction data.
  • Structured for Analysis: While still granular, the data is modeled into a more consistent and accessible format, often involving parsing nested structures like JSON.
• Users: Data engineers who build the transformations, and data scientists or analysts who need access to detailed, clean data for exploratory analysis or feature engineering.

Read the following section to learn about the typical operations performed in the Silver layer.

The Gold Layer: Business-Ready Analytics

The final stage is the Gold layer. This layer is all about delivering value to the business.

• Purpose: To provide highly refined, aggregated data tables that are optimized for business intelligence (BI) and reporting.
• Characteristics:
  • Aggregated: Data is often summarized to key business metrics (e.g., daily sales, monthly active users).
  • Business-Oriented: Tables are organized around business domains (e.g., finance, marketing, sales) and use business-friendly naming conventions.
  • Optimized for Performance: Gold tables are designed for fast, efficient querying by analytics tools and dashboards.
• Users: Business analysts, BI developers, executives, and machine learning applications that consume final, aggregated features.

Let's read about this final, value-driven layer.

Putting It All Together

To help you visualize the entire flow and the distinct purpose of each layer, here is a helpful infographic and a summary table.

The following table from the Microsoft documentation provides a concise comparison of what happens in each layer and who the intended users are.

When implemented in a DLT pipeline, this architecture creates a clear, manageable flow of data.

Conclusion

In this lesson, you've learned about the medallion architecture, a foundational data design pattern for building robust and scalable data pipelines in the Databricks Lakehouse.

Key Takeaways

• Medallion Architecture: A multi-hop pattern that progressively improves data quality across three layers: Bronze, Silver, and Gold.
• Bronze Layer: The landing zone for raw, untransformed data. It serves as a historical archive and enables pipeline reprocessing.
• Silver Layer: Contains cleansed, validated, and conformed data. This is where data quality is enforced, and datasets are enriched and structured for analysis.
• Gold Layer: Holds highly refined, business-level aggregated data. These tables are optimized for BI, reporting, and analytics.
• Value Proposition: As data moves from Bronze to Gold, its quality, reliability, and business value increase, while latency also typically increases due to the processing involved.

Next Steps

Now that you understand the "what" and "why" of the medallion architecture, it's time to put it into practice. In the next lesson, you will implement a DLT pipeline using SQL syntax (CREATE LIVE TABLE), creating your own Bronze, Silver, and Gold tables to transform raw data into business-ready insights. This will be your first hands-on experience building a declarative pipeline.