Medallion Data Layering Explained

Hello! Welcome to our next lesson.

In our last session, we focused on idempotency, establishing how to load data reliably and prevent duplicates when tasks are re-run. This is a crucial technique for ensuring data integrity at the point of ingestion. Now that we know how to load data reliably, the next logical question is where should this data land, and how should we organize it as it gets refined?

This lesson introduces the medallion architecture, a widely adopted design pattern for structuring data in a data pipeline. You'll learn how this pattern uses a series of layers—named Bronze, Silver, and Gold—to progressively improve the quality and usability of your data. This architectural concept will serve as the blueprint for organizing the data within your PostgreSQL database, providing a clear path from raw ingestion to the final analytics dashboard and NLP-ready datasets you aim to build.

By the end of this lesson, you will be able to define the medallion (bronze, silver, gold) data layering pattern and explain its central purpose in creating trustworthy and scalable data assets.

What is the Medallion Architecture?

At its core, the medallion architecture is a data engineering pattern for logically organizing data into layers of increasing quality. It provides a structured approach to transform raw, and often messy, data into clean, reliable, and valuable assets ready for consumption.

To get a quick overview of this concept and see where it fits into the larger data engineering landscape, let's start with a short video.

The Medallion Data Architecture (Pros & Cons)

This video from Kahan Data Solutions provides a clear, high-level introduction to the medallion architecture.

Please watch the first two minutes. Pay attention to how the presenter defines the three layers and then places the entire architecture within the context of turning raw sources into insights. This "central hub" concept is precisely what we are designing.

As the video explained, the process involves moving data through three stages:

Bronze: A landing zone for raw, untransformed data.
Silver: Where data is cleaned, conformed, and integrated.
Gold: The final presentation layer, containing aggregated, business-ready data.

This diagram illustrates the fundamental flow of the medallion architecture, moving data from various sources through the Bronze, Silver, and Gold layers to serve end-user applications like Business Intelligence and Machine Learning.

While the "medallion" terminology was popularized by Databricks, the underlying principle of layered data curation has been a best practice in data warehousing for decades. It's a time-tested approach to data management, repackaged for the modern data stack.

The "Why": Purpose and Benefits of Layering

Before we examine each layer in detail, it's important to understand why this layered approach is so valuable. Simply dumping all data into one place and transforming it on the fly can quickly lead to chaos. The medallion architecture brings order by establishing a clear data flow with specific goals at each stage.

What Is Medallion Architecture? Bronze, Silver & Gold Explained

This article from Clarifai clearly outlines the core motivations behind adopting a layered data architecture.

Please read the section under the heading "What Is Medallion Architecture?". Focus on the bulleted list explaining the core needs that this pattern addresses.

As the article highlights, the key benefits are:

Trust and Quality: It provides a systematic way to build trust. End users of the Gold layer know the data has been rigorously cleaned and validated.
Modularity and Traceability: Each layer has a distinct responsibility. If an issue is found in a final report (Gold), you can trace the data's lineage back through the Silver and Bronze layers to diagnose the root cause.
Reproducibility and Reusability: The Bronze layer acts as a permanent, raw archive. If you discover a bug in your transformation logic, you don't need to re-ingest data from the source systems. You can simply fix the logic and re-process the raw data from the Bronze layer to rebuild the Silver and Gold layers. This is a powerful feature for reliability.
Compliance and Auditability: Storing the untouched source data in the Bronze layer provides a full-fidelity audit trail, which is crucial in many industries.

Now, let's explore what happens within each of these layers.

A more detailed view of the Medallion Architecture, showing the progression from raw ingestion to business-level aggregates, and the various technologies involved at each stage of the data lifecycle.

The Layers in Detail

1. The Bronze Layer: Raw Ingestion and History

The Bronze layer is the foundation of the architecture. Its primary purpose is to ingest data from your various sources and store it in its most raw, unaltered state.

What it is: A landing zone or staging area. For your project, this will be a dedicated bronze schema in your PostgreSQL database containing tables that mirror the structure of your source data (from APIs, CSVs, and Kafka topics).
State of Data: Raw, unvalidated, and untransformed. It should be an exact copy of the source. Think of it as your data's historical archive.
Key Operations: Data is loaded here using the idempotent upsert patterns we discussed in the previous lesson (INSERT ... ON CONFLICT). This ensures that re-running your ingestion scripts won't create duplicate raw records. You also capture metadata like ingested_at timestamps.

The main principle of the Bronze layer is to get the data in and preserve it. You are not concerned with its quality or structure yet; you are simply securing it for future processing.

2. The Silver Layer: Filtered, Cleaned, and Augmented

The Silver layer is where the real transformation begins. Here, you take the raw data from the Bronze layer and turn it into a reliable, queryable, and consistent set of tables.

What it is: The "conformed" or "enriched" layer. For you, this will be a silver schema in PostgreSQL, populated by dbt models that read from the bronze schema.
State of Data: Cleaned (handling nulls, fixing inconsistencies), validated (enforcing data types, checking constraints), and integrated (joining data from different sources).
Key Operations: Your advanced SQL and dbt skills will be heavily utilized here. You'll perform tasks like:
- Casting data types (e.g., '2024-01-01' as text to a DATE).
- Renaming cryptic column names (e.g., user_ts to user_registration_timestamp).
- Joining different datasets (e.g., enriching user activity data with user profile information).
- Filtering out records that don't meet quality standards.

The output of the Silver layer is a set of well-structured, trustworthy tables that represent a "single source of truth." Data scientists and analysts can confidently use this layer for exploratory analysis.

3. The Gold Layer: Business-Level Aggregates

The Gold layer is the final, highly-refined presentation layer. It's built on top of the Silver layer and is optimized for specific business use cases, like analytics and machine learning.

What it is: The "presentation" or "data mart" layer. This will be a gold schema in your PostgreSQL database, also built with dbt models that read from the silver schema.
State of Data: Aggregated, feature-rich, and optimized for performance. It's not a normalized database; it's designed for fast reads by downstream applications.
Key Operations: This is where you create datasets tailored for your two end goals:
- For your analytics dashboard: You will create aggregated tables (e.g., daily active users, monthly summaries) that your BI tool can query directly and efficiently. This is often where you build fact and dimension tables.
- For your NLP service: You will create a model that prepares text data specifically for the third-party AI. This could involve concatenating fields, performing final cleaning or PII redaction, and structuring the output in the exact format the API expects.

The Gold layer transforms trusted data into valuable, actionable insights.

Key Takeaways

The Medallion Architecture is a design pattern that organizes data into Bronze (raw), Silver (cleaned), and Gold (business-ready) layers.
Its primary purpose is to progressively improve data quality, trust, and usability as data flows through the pipeline.
Bronze Layer: The raw data landing zone. You ingest data "as-is" for audit and reprocessing purposes. This is where your idempotent loading scripts will write to.
Silver Layer: The transformation and cleaning hub. Here you use tools like dbt and SQL to create a validated, unified source of truth.
Gold Layer: The presentation layer. Data is aggregated and modeled for specific end-uses, such as BI dashboards and ML models.

This pattern provides a robust and scalable mental model for structuring your data pipeline, ensuring that by the time data reaches your end users, it is both reliable and valuable.

Next time

We have now covered all the foundational concepts of our data pipeline: the ELT approach, batch vs. streaming processing, the principle of idempotency, and the medallion architecture for data organization. In our next lesson, we will synthesize all of this knowledge to outline the complete target architecture for your project, mapping each concept and processing stage to your chosen technology stack of PostgreSQL, Airflow, dbt, and Kafka.

Recommended resources