Triggered vs. Continuous Pipelines

Execution Modes: The Core of Pipeline Scheduling

In Delta Live Tables, the execution mode determines the lifecycle of the compute resources (the cluster) and how often your data is processed. You have two primary choices: Triggered and Continuous.

Let's start with a clear, high-level definition of each mode from an article written to help learners like you prepare for the certification exam.

As you just read, the choice is fundamentally about whether the pipeline runs constantly or in discrete batches. Let's formalize this.

• Continuous Mode: The pipeline runs constantly. The cluster stays active to process new data as soon as it arrives from the source. This is ideal for streaming use cases where low-latency, near-real-time data is a priority.

• Triggered Mode: The pipeline starts a cluster, processes all data available at that moment, and then terminates the cluster once the update is complete. This is best for batch workloads where data freshness is less critical (e.g., hourly or daily updates).

The Fundamental Trade-Off: Cost vs. Latency

The decision between triggered and continuous execution boils down to balancing business needs with resource costs. The official Databricks documentation provides a detailed comparison that is essential for understanding this trade-off.

From this reading, we can summarize the core trade-off:

This is a classic engineering trade-off. If your goal is to power a live dashboard monitoring sensor data, the higher cost of a continuous pipeline is justified by the need for minimal latency. Conversely, if you're updating a summary table for a daily business report, a triggered pipeline that runs once a night is far more cost-effective.

How DLT Makes This Possible

You might wonder how DLT can so easily switch between these two very different execution patterns without requiring you to rewrite your code. The answer lies in its declarative nature and its foundation on Spark's Structured Streaming engine.

In a declarative framework like DLT, you define the desired end state of your data—the "what"—and the engine figures out the most efficient way to achieve it—the "how". This separation of logic from execution is powerful. Your pipeline definition (the sequence of Bronze, Silver, and Gold tables) remains the same, and the execution mode is simply a configuration that tells the DLT engine how often to run the logic.

This is analogous to concepts in physics where you can describe a system with a set of equations (e.g., a Hamiltonian) that are independent of the specific time-evolution path taken. The system's definition is separate from its execution over time.

To see how this works under the hood, let's watch a clip from Michael Armbrust, one of Spark's original creators, as he explains how Structured Streaming supports this flexibility.

Practical Configuration and Exam Tips

In the Databricks UI, selecting the execution mode is as simple as choosing an option from a dropdown menu when you configure your pipeline.

For your certification exam, it's crucial to correctly interpret scenario-based questions. Words matter, and there are common pitfalls to avoid.

As highlighted in the article, be very careful with the phrasing:

• A pipeline that runs and then stops automatically is Triggered.
• A pipeline that runs until manually stopped is Continuous. This is a key giveaway, even if the question mentions "defined intervals" (which in continuous mode refers to the micro-batch trigger interval, not a scheduled run).

A Note on Development vs. Production Mode

When you configure the execution mode, you'll also see another option called Pipeline Mode, with choices for Development and Production. While this isn't our main topic today, it's closely related.

• Development Mode: Optimized for iterative development. It reuses clusters to avoid startup delays and disables automatic retries, allowing you to iterate on your code faster.
• Production Mode: Designed for reliability. It uses a new, dedicated cluster for each run and enables automatic retries to handle transient failures, ensuring your pipeline is robust.

You typically use Development mode while building your pipeline and switch to Production mode for your stable, deployed workflows.

Conclusion

In this lesson, you've learned to differentiate between the two DLT execution modes. This knowledge is fundamental to designing pipelines that meet your business requirements for data freshness while respecting your budget for compute resources.

Key Takeaways

• Triggered Mode: Best for batch processing. Runs once per trigger, processes available data, and then shuts down the cluster. It prioritizes cost savings.
• Continuous Mode: Best for streaming data. Runs constantly to process new data as it arrives. It prioritizes low latency.
• The primary trade-off when choosing a mode is Cost vs. Latency.
• DLT's declarative nature allows you to switch between modes without changing your transformation logic.
• For the exam, remember that a pipeline running "until manually stopped" is in Continuous mode.

Next Steps

You now understand a critical DLT pipeline setting. In the next lesson, we will broaden our scope to cover other important configurations. You will learn to configure DLT pipeline settings, including target schema and storage location, giving you more complete control over your pipeline's behavior and organization.