DLT Pipeline Configuration

Understanding Target Schema and Storage Location

When a DLT pipeline runs, it creates tables and stores their underlying data files. These two aspects are managed through two key settings:

1. Target Schema: This is the logical identifier for your tables within Databricks. It specifies the database (or in Unity Catalog, the catalog and schema) where your tables will be registered. Once registered, they become discoverable and can be queried by anyone with the correct permissions, just like any other table.

2. Storage Location: This is the physical path in cloud storage (like Amazon S3 or Azure Data Lake Storage) where the actual data files (in Delta format) and pipeline metadata are saved.

While related, they serve different purposes. The target schema provides a user-friendly name for querying, while the storage location points to the data's physical home.

Configuring the Default Destination in the UI

The most straightforward way to manage your pipeline's destination is by setting a default location in the pipeline's configuration UI. Let's walk through where to find these settings.

As you read, the Destination setting is key. You can specify a target schema (or database) where your pipeline will publish its tables. If you leave the Storage location blank, Databricks automatically creates a directory for your pipeline's data and metadata.

Let's see a quick demonstration of this in action.

The interface has been updated since that video was made, especially with the deep integration of Unity Catalog. Here is what the modern UI for configuring the destination looks like:

As you can see, you now explicitly choose whether to publish your tables to the older Hive Metastore or to the modern Unity Catalog. For the certification and modern data engineering practices, Unity Catalog is the recommended choice. When you select it, you must provide:

• Catalog: The top-level container in Unity Catalog's three-level namespace.
• Target schema: The schema (or database) within that catalog.

Any tables created by this pipeline (e.g., CREATE LIVE TABLE my_bronze_table) will, by default, be created as your_catalog.your_schema.my_bronze_table.

Advanced Control: Overriding the Default Destination in Code

Setting a default destination is convenient, but what if you want to publish different tables from the same pipeline to different locations? For example, you might process Bronze and Silver tables in a development schema but want to publish the final Gold table to a separate, governed production schema.

DLT allows you to override the pipeline's default destination directly in your code. The official Databricks documentation explains this feature well.

Let's summarize the key techniques from the documentation:

1. Using Fully-Qualified Names

This is the most direct and explicit method. You can specify the full three-level namespace name (catalog.schema.table) when defining a table. This overrides any default pipeline settings for that specific table.

SQL Example:

-- This table will be created in the 'prod_analytics' catalog and 'gold' schema,
-- regardless of the pipeline's default destination settings.
CREATE OR REPLACE LIVE TABLE prod_analytics.gold.daily_sales_summary
COMMENT "Daily aggregated sales summary"
AS SELECT
  transaction_date,
  SUM(sales_amount) as total_sales
FROM LIVE.silver_sales
GROUP BY transaction_date;

Python Example:

import dlt




# This materialized view will be created in 'prod_analytics.gold',
# overriding the pipeline's default.
@dlt.table(
  name="prod_analytics.gold.daily_sales_summary",
  comment="Daily aggregated sales summary"
)
def daily_sales_summary():
  return (
    dlt.read("silver_sales")
      .groupBy("transaction_date")
      .agg(F.sum("sales_amount").alias("total_sales"))
  )

This method gives you granular control and makes your code's intentions very clear.

2. Using USE CATALOG and USE SCHEMA

In SQL, you can use these commands at the top of a notebook. They change the default context for all subsequent DLT table definitions within that specific notebook.

USE CATALOG prod_analytics;
USE SCHEMA gold;

-- Because of the USE commands above, this table will be created as
-- 'prod_analytics.gold.daily_sales_summary' without needing the full name.
CREATE OR REPLACE LIVE TABLE daily_sales_summary
AS SELECT ...

This can be useful for organizing code but is less explicit than using fully-qualified names for each table.

Let's watch a video that demonstrates this powerful override capability.

Writing to External Locations with Sinks

Finally, sometimes you may need to write the output of a pipeline to a standard Delta table that is not managed by DLT. This is useful for integrating with other processes that expect a regular Delta table. DLT provides a mechanism called a sink for this purpose.

A sink allows you to take the final output of a transformation and write it to a specified external Delta table.

Using a sink provides ultimate control over the output, effectively detaching it from the DLT pipeline's own managed tables and placing it in a location of your choice.

Conclusion

You now have a comprehensive understanding of how to control where your DLT pipeline's data is stored, both logically and physically.

Key Takeaways

• Target Schema vs. Storage Location: The target schema is the logical address in the metastore (e.g., my_catalog.my_schema), while the storage location is the physical path in cloud storage.
• UI Configuration: You can set a default Catalog and Target schema for your entire pipeline in the DLT settings UI. This is the primary destination for all tables created by the pipeline.
• Code Overrides: For granular control, you can override the default destination for individual tables using fully-qualified three-part names (catalog.schema.table) in your SQL or Python code.
• Sinks: Use a sink to write a pipeline's output to a standard, external Delta table, giving you full control over the final destination.

Next Steps

You've learned how to configure when a pipeline runs and where it stores its data. The next logical step is to understand what happens during a run. In our next lesson, we will focus on monitoring pipeline runs and interpreting the DLT event log for troubleshooting, which is essential for maintaining healthy, production-grade data pipelines.