Job Dependencies and Execution Order

1. Beyond Linear: Parallel and Fan-In Dependencies

Most data pipelines involve more than a single stream of data. For instance, you might ingest data from several different sources simultaneously. Databricks Workflows allows you to model this by running tasks in parallel and then having a subsequent task wait for all of them to complete.

• Parallel Tasks: Tasks that have no dependencies on each other (or depend on the same parent task) can run concurrently, saving valuable time.
• Fan-In Dependency: A single task that depends on the completion of multiple upstream parallel tasks. This is common for join or aggregation steps.

The following image of a workflow DAG (Directed Acyclic Graph) illustrates these concepts perfectly.

Hands-On: Modifying Your Job for Parallel Execution

Let's apply this to the job you created in the previous lesson. You will modify it to include a parallel task and a fan-in dependency.

1. Create a Dummy Notebook: In your Databricks workspace, create a new, simple notebook named Log_Job_Start. Add a single cell with the following Python code. This notebook will act as a simple parallel task.

   import datetime
   
   start_time = datetime.datetime.now()
   print(f"Job triggered at: {start_time}")
   

2. Edit Your Existing Job:

   • Navigate to your job in the Workflows UI and go to the Tasks tab.
   • Add a new task named log-start-time that uses the Log_Job_Start notebook you just created.
   • Crucially, do not set any dependencies for this new task. This makes it a "root" task that runs in parallel with your other root task, retrieve-baby-names.
   • Select the filter-baby-names task. In its configuration panel, find the Depends on setting. It should currently be set to retrieve-baby-names.
   • Click into the Depends on field and add log-start-time as a second dependency.

Your updated workflow should now look like this, with two tasks running in parallel before the final task.

This diagram shows the target structure for your modified job: two parallel tasks feeding into a single "fan-in" task.

Run the job and observe in the Runs view how the first two tasks execute concurrently.

2. Conditional Execution with If/else Tasks

A powerful feature of any orchestrator is the ability to make decisions. You might want to run a full data load on a Sunday but only an incremental update on other days. The If/else condition task type in Databricks Workflows enables this branching logic.

To create conditional logic, you typically need three components:

1. A task that produces an output value (e.g., the day of the week).
2. An If/else condition task that evaluates this value.
3. Downstream tasks that depend on the true or false outcome of the condition.

To pass values between tasks, you use the dbutils.jobs.taskValues utility.

• dbutils.jobs.taskValues.set(key="my_key", value="my_value") is used in an upstream task to output a value.
• {{tasks.TASK_KEY.values.my_key}} is a dynamic value reference used in a downstream task's configuration to retrieve the value.

The following video provides a detailed, step-by-step demonstration of building a conditional workflow from scratch.

This walkthrough gives you a complete, hands-on model for implementing branching logic in your own data pipelines. You don't need to build this example yourself right now, but make sure you understand the mechanics involved.

3. Advanced Dependency Conditions: The run_if Setting

By default, a task runs only if all its dependencies succeed (ALL_SUCCESS). But what if you need to run a cleanup task regardless of whether the main job succeeded or failed? Or trigger an alert only if a task fails?

The run_if setting gives you this granular control. When you define a dependency, you can change this condition.

Here is a summary of the most common run_if conditions and their use cases:

This setting provides the backbone for robust error handling and pipeline management.

4. For Reference: Defining Dependencies Programmatically

Given your background in deploying data platforms, you'll appreciate that workflows can be defined as code. This is essential for CI/CD and managing infrastructure as code.

When defining a job in a JSON (for the REST API) or YAML (for Databricks Asset Bundles) file, dependencies are declared explicitly.

REST API (JSON) Example:
Notice the depends_on array, which lists the task_key of the parent tasks.

{
  "name": "etl_daily_pipeline",
  "tasks": [
    {
      "task_key": "ingest_raw_data",
      "notebook_task": { ... }
    },
    {
      "task_key": "clean_and_validate",
      "depends_on": [{"task_key": "ingest_raw_data"}],
      "notebook_task": { ... }
    }
  ]
}

This snippet is adapted from the "How to Orchestrate Databricks Notebooks..." article (LINK).

Databricks Asset Bundle (YAML) Example:
The structure is similar, using a depends_on list.

resources:
  jobs:
    my-job:
      name: my-job
      tasks:
        - task_key: condition_task
          condition_task:
            op: LESS_THAN
            left: '{{job.repair_count}}'
            right: '5'
        - task_key: notebook_task
          depends_on:
            - task_key: condition_task
              outcome: 'true'
          notebook_task:
            notebook_path: ../src/notebook.ipynb

This example from the Databricks documentation (LINK) also shows a dependency on a specific outcome (true) from a condition task, which is how branching is defined in code.

You don't need to memorize this syntax now, but it's important to recognize the pattern. The visual graph you build in the UI is simply a representation of these underlying code-based definitions.

Conclusion

You have now moved beyond simple linear pipelines and learned how to orchestrate complex workflows. This is a critical skill for any data engineer working on the Databricks platform.

Key Takeaways:

• You can create parallel tasks by removing dependencies and fan-in dependencies by making a task depend on multiple upstream tasks.
• The If/else condition task is the primary tool for creating branching logic within your jobs.
• dbutils.jobs.taskValues is the mechanism for passing information between tasks to enable dynamic, conditional workflows.
• The run_if setting provides fine-grained control for building robust error-handling and cleanup procedures (e.g., running a task on failure or regardless of outcome).

Preview of the Next Lesson:

In our last lesson, you were introduced to the difference between job clusters and all-purpose clusters. In the next lesson, "Configure job clusters vs. all-purpose clusters for workflow tasks," we will dive much deeper. You'll learn how to optimize cluster configurations for different types of tasks within the same job, apply cluster policies to enforce standards, and make strategic decisions to balance performance and cost in your production workflows.