Building a DLT Pipeline: Python Decorators & SQL

1. From Imperative to Declarative: What is DLT?

Before we write any code, it's crucial to understand the conceptual shift DLT introduces. Instead of writing imperative code that tells Spark how to execute a pipeline step-by-step, you write declarative code that describes the what—the desired end state of your tables. DLT then takes care of the "how" for you: orchestration, error handling, scaling, and more.

This is a significant evolution from traditional ETL/ELT development, and it's the primary reason DLT can dramatically accelerate development and improve pipeline reliability.

As the video highlighted, DLT pipelines are composed of two main types of datasets:

• Streaming Tables: These are designed for append-only, incremental data processing. They are the standard choice for ingesting data from streaming sources or incrementally transforming data that arrives over time.
• Materialized Views (MVs): These are tables where the results are precomputed as a snapshot. They are recomputed entirely during each pipeline update. They are ideal for aggregations, reference data, or transformations where you need to reprocess the entire source dataset.

2. The Language of DLT: SQL and Python Syntax

DLT provides two equivalent syntaxes—SQL and Python—for defining pipelines. Your choice depends on your team's preference and the specific task at hand. You can even mix both languages within the same pipeline.

Let's explore how to define a simple two-step pipeline: first, a streaming table to ingest raw data, and second, a materialized view to create an aggregate.

SQL Syntax

In SQL, you use variants of familiar DDL statements like CREATE STREAMING TABLE and CREATE MATERIALIZED VIEW.

Here’s a summary of the core SQL patterns:

To create a streaming table for ingestion:

CREATE OR REFRESH STREAMING TABLE raw_events
COMMENT "Raw events data from cloud storage"
AS SELECT * FROM STREAM read_files('s3://my-bucket/raw/events/', format => 'json');

• STREAMING TABLE: Declares an incrementally updated table.
• STREAM: The STREAM keyword on the read_files function tells DLT to process the source incrementally.

To create a materialized view for aggregation:

CREATE OR REFRESH MATERIALIZED VIEW daily_event_counts
COMMENT "Daily count of events"
AS SELECT
  CAST(event_timestamp AS DATE) as event_date,
  COUNT(*) as total_events
FROM LIVE.raw_events
GROUP BY event_date;

• MATERIALIZED VIEW: Declares a table that will be fully recomputed on each update.
• LIVE: This special schema name is used to refer to other tables within the same DLT pipeline, allowing DLT to automatically infer the dependency graph.

Python Syntax

In Python, you define DLT tables as functions that return a Spark DataFrame. These functions are decorated with @dlt.table to register them as part of the pipeline.

Let's recap the Python patterns from the video, which are also reflected in the Databricks tutorial we reviewed earlier.

To create a streaming table for ingestion:

import dlt
from pyspark.sql.functions import *

@dlt.table(
  name="raw_events",
  comment="Raw events data from cloud storage"
)
def raw_events_ingest():
  return (
    spark.readStream.format("cloudFiles")
      .option("cloudFiles.format", "json")
      .load("/path/to/raw/events/")
  )

• @dlt.table: The decorator that turns a Python function into a DLT table.
• The function returns a streaming DataFrame (readStream), so DLT creates a streaming table.

To create a materialized view for aggregation:

@dlt.table(
  name="daily_event_counts",
  comment="Daily count of events"
)
def daily_event_counts_agg():



  # dlt.read() reads from another table in the same pipeline
  return (
    dlt.read("raw_events")
      .groupBy(col("event_timestamp").cast("date").alias("event_date"))
      .agg(count("*").alias("total_events"))
  )

• The function returns a batch DataFrame (derived from dlt.read(), which is a batch read), so DLT creates a materialized view.
• dlt.read("raw_events"): This is the Python equivalent of LIVE.raw_events in SQL. It creates the dependency link between the tables.

3. Creating and Running a DLT Pipeline

Now that we understand the syntax, let's see how you assemble these code definitions into a running pipeline in the Databricks workspace. The process involves configuring a pipeline in the UI, pointing it to your source code (notebooks), and starting an update.

The real magic of DLT becomes apparent when you run the pipeline. DLT analyzes your code, automatically builds the dependency graph (DAG) of your tables, and orchestrates the execution.

As you saw in the video, you don't define tasks or dependencies manually as you would in tools like Airflow or Databricks Workflows. You simply declare your tables, and DLT builds and executes the pipeline for you. This declarative approach is the cornerstone of DLT's power and simplicity.

Conclusion

In this lesson, you've learned the fundamentals of implementing data pipelines with Delta Live Tables. We've moved from the imperative world of manual stream management to the declarative world of automated pipeline orchestration.

Key Takeaways:

• Declarative Framework: DLT lets you define the what (your target tables), not the how (the execution steps). It automatically manages dependencies, infrastructure, and retries.
• Two Core Abstractions: Pipelines are built from Streaming Tables for incremental processing and Materialized Views for full-snapshot recomputations.
• Dual Syntax: You can define pipelines using both SQL (CREATE STREAMING TABLE) and Python (@dlt.table decorator), and even mix them.
• Automated Orchestration: By referencing other tables using LIVE (SQL) or dlt.read() (Python), DLT automatically builds and visualizes the pipeline's execution graph (DAG).

Preview of the Next Lesson:

Today, we learned the syntax to create streaming tables and materialized views. In the next lesson, "Create streaming tables and materialized views in DLT and compare their behavior," we will take a deeper dive. We'll explore the specific use cases, performance characteristics, and trade-offs of each, so you can confidently choose the right tool for any transformation task in your DLT pipelines.