Managing Task Dependencies

From Parallel to Sequential: Defining Execution Order

In the previous lesson, when you added two tasks to a job, they appeared side-by-side with no connecting lines. This visual represents parallel execution—Databricks will try to run both tasks simultaneously.

To enforce an order, you need to create a dependency. This turns your workflow into a Directed Acyclic Graph (DAG).

• Directed: The relationship has a direction (Task A runs before Task B).
• Acyclic: The workflow cannot contain loops (Task A can't depend on Task B if Task B already depends on Task A).

This graph is the blueprint for your pipeline's execution.

Configuring a Basic Dependency

Let's see how to create a simple sequential dependency where one task follows another. This is done using the Depends on field in the task configuration UI.

The following video demonstrates this by adding a "Silver Layer" task that depends on the initial "Ingestion Layer" task.

As you saw, you can select one or more prerequisite tasks in the Depends on dropdown. A task will only start once all the tasks it depends on have completed successfully.

A common pattern is a "fan-in" dependency, where a final task (e.g., 'Aggregate Data') depends on several parallel tasks that must all finish first. This is a powerful way to orchestrate complex workflows efficiently.

Handling Success and Failure with "Run if" Conditions

By default, a task only runs if its dependencies have all succeeded. But what if you need to handle failures? For instance, if an ingestion task fails, you might want to run a separate task to send a notification or clean up partially loaded data.

This is where the Run if condition comes into play. It gives you fine-grained control over the logic that triggers a task.

Let's review the official documentation to see all available options.

Here’s a quick summary of the most common Run if conditions and their use cases:

• All succeeded (Default): The standard case for a sequential pipeline. Task B runs only if Task A is successful.
• At least one failed: Perfect for error handling. A "Send Alert" task can be configured to run if any of its upstream dependencies fail.
• All done: Use this for tasks that must execute regardless of the outcome, such as logging the end of a job run or cleaning up temporary resources.

Advanced Control Flow: The "If/else condition" Task

Sometimes, you need to branch your workflow based on a dynamic value, not just success or failure. For example: "If today is the last day of the month, run the 'Generate Monthly Report' task; otherwise, run the 'Generate Daily Summary' task."

For this, Databricks provides a special task type: If/else condition. This task evaluates a boolean expression and directs the workflow down one of two paths: a true path or a false path.

The video below demonstrates this concept perfectly.

This powerful feature allows you to build workflows that can adapt their behavior based on the data they are processing or other external parameters.

Your Turn: Hands-On Practice

It's time to put these concepts into practice. We'll modify the job you created in the last lesson.

Prerequisites: You should have the My Practice Workflow job and the two notebooks, task_1_notebook and task_2_notebook, from the previous lesson.

Part 1: Create a Sequential Dependency

1. Navigate to your My Practice Workflow job in the Workflows UI.
2. Click on the task that runs task_2_notebook.
3. In the task configuration panel on the right, find the Depends on field.
4. Select your first task (the one that runs task_1_notebook) from the dropdown menu.
5. Click Save task. The UI should now show an arrow from Task 1 to Task 2.
6. Click Run now and navigate to the Runs tab. Observe the execution. You should see Task 1 run to completion before Task 2 begins.

Part 2: Implement Failure Handling

Now, let's add an error-handling path.

1. Create a new notebook: Name it task_3_cleanup. Add a single cell with the code print("A previous task failed. Running cleanup.").
2. Induce an error: Open task_2_notebook and change its content to cause a failure. A simple way is to add a line with raise Exception("This is an intentional failure").
3. Add the cleanup task to your job:
   • Go back to your My Practice Workflow job.
   • Click the + icon to add a new task.
   • Configure this task to run your task_3_cleanup notebook.
   • Set its Depends on field to be your Task 2.
   • Crucially, change the Run if dependencies condition to At least one failed.
   • Save the task. Your graph should now show Task 3 branching off Task 2.
4. Run the job again. This time, you should see Task 1 succeed, Task 2 fail (marked in red), and then Task 3 run successfully. This confirms your failure handling logic is working!

Conclusion

Excellent work! You've moved beyond simple parallel tasks and can now orchestrate complex workflows with precise control over execution order and logic. This is a fundamental skill for building reliable and resilient production data pipelines.

Key Takeaways

• Task dependencies create a Directed Acyclic Graph (DAG) that defines the execution order.
• The Depends on field is used in the UI to create a sequential relationship between tasks.
• The Run if condition allows you to handle different outcomes, such as task failures, by triggering specific downstream tasks (e.g., for notifications or cleanup).
• The If/else condition task type enables dynamic branching in your workflow based on the evaluation of a boolean expression.

Next Steps

You've defined what your job does and in what order. The next logical question is when it should run. In our next lesson, we will explore how to configure a schedule to run a Databricks job at a specific time or interval, moving us one step closer to full automation.