Unity Catalog: Data Lineage UI

1. What is Data Lineage and Why is it Important?

Data lineage is the story of your data's lifecycle. It answers fundamental questions:

• Where did this data come from? (Upstream sources)
• What transformations has it undergone? (Jobs, notebooks, queries)
• Where is this data being used? (Downstream tables, views, dashboards)

To see these benefits in action, let's watch a brief introductory video from Databricks.

As you can see, lineage isn't just an audit feature; it's a critical tool for data engineers, analysts, and scientists to understand, trust, and debug their data pipelines.

2. How Unity Catalog Captures Lineage

One of the most powerful features of Unity Catalog is that it captures lineage automatically at runtime for any query executed via the Databricks SQL or DataFrame interfaces. This means that as you run your notebooks, jobs, and DLT pipelines, Unity Catalog is building a map of your data flows in the background.

However, for this automatic capture to work, certain conditions must be met, and it's also important to be aware of its limitations. These are common topics for the certification exam.

3. Hands-On: Exploring Lineage in the UI

Now for the most important part: learning by doing. We'll explore the lineage UI in Catalog Explorer. The interface provides an interactive graph to visualize data flows at the table, view, dashboard, notebook, and job levels.

Exploring the Lineage Graph

The lineage graph is your primary tool for tracing data. Let's walk through how to use it.

The key components you'll interact with are:

• Upstream vs. Downstream: Upstream shows where the data came from; downstream shows where it's going.
• Nodes: These represent data assets (tables, views) or workloads (notebooks, jobs).
• Column-Level Lineage: You can click on a column to see its specific end-to-end journey.

Practical Exercise: Generate and Trace Your Own Lineage

Let's create a simple data pipeline using SQL to generate some lineage data that you can then explore.

Instructions:

1. In a new SQL notebook in your Databricks workspace, make sure you have selected a catalog and schema where you have USE CATALOG, USE SCHEMA, and CREATE TABLE permissions.

2. Step 1: Create a Raw Data Table. This will be our source.

   CREATE OR REPLACE TABLE sales_raw (
     sales_id INT,
     customer_name STRING,
     amount DECIMAL(10, 2),
     sale_date DATE
   );
   
   INSERT INTO sales_raw VALUES
     (1, 'Alice', 100.00, '2023-11-01'),
     (2, 'Bob', 250.50, '2023-11-02'),
     (3, 'Alice', 75.25, '2023-11-05');
   

3. Step 2: Create a Transformed Table. This table will be derived from our raw table.

   CREATE OR REPLACE TABLE sales_transformed AS
   SELECT
     sales_id,
     customer_name,
     amount * 1.08 AS amount_with_tax, -- Add 8% tax
     sale_date,
     YEAR(sale_date) as sale_year
   FROM sales_raw;
   

4. Step 3: Create a Report View. This view aggregates data from the transformed table.

   CREATE OR REPLACE VIEW customer_sales_report AS
   SELECT
     customer_name,
     SUM(amount_with_tax) as total_sales
   FROM sales_transformed
   GROUP BY customer_name;
   

5. Step 4: Explore the Lineage. Now, navigate to Catalog Explorer (Data icon in the left sidebar).

   • Find your sales_transformed table.
   • Click on the Lineage tab.
   • Click See Lineage Graph.

Your Tasks:

• Identify the upstream table and downstream view connected to sales_transformed.
• Expand the columns for each table in the graph. Click on the amount_with_tax column in sales_transformed. Which column in the sales_raw table is its ancestor?
• Close the graph and, still on the Lineage page, click the Notebooks tab. You should see a link to the notebook you just used to create the table. This is how you trace back to the exact code that produced the data.

4. Lineage and Permissions

Just like everything else in Unity Catalog, lineage is governed by permissions. Your ability to see assets in the lineage graph depends on your privileges.

In short, if you don't have at least BROWSE or SELECT privilege on a table, you won't be able to see its details in the lineage graph. The object will appear as a "masked" node, protecting sensitive information about its structure or existence. This ensures that data discovery through lineage still respects the security boundaries you've established.

Conclusion

In this lesson, we've explored how Unity Catalog provides powerful, automated data lineage capabilities. This feature is fundamental to building a trustworthy and governable data lakehouse.

Key Takeaways

• Data Lineage tracks the entire lifecycle of data, from its source to its consumption.
• It is critical for impact analysis, root cause analysis of data quality issues, and regulatory compliance.
• Unity Catalog automatically captures lineage for table and column-level operations performed through Spark SQL and DataFrames.
• You can visualize and interact with data lineage through the Lineage Graph in Catalog Explorer.
• The graph shows relationships between tables, views, notebooks, jobs, and dashboards.
• Access to view lineage information is governed by the standard Unity Catalog permission model.

Next Steps

Now that we can secure our data and trace its lineage, the final piece of the governance puzzle is making our data discoverable. In our next and final lesson of this module, we will learn how to use Unity Catalog to discover and search for data assets across the organization.