Passing Data Between Tasks

1. The Building Blocks of Dynamic Workflows

Before we dive into the code, it's important to understand the different ways you can parameterize a job in Databricks.

To summarize the key distinction:

• Job/Task Parameters are primarily for providing input to your workflow from the outside (either statically defined or at runtime).
• Task Values are for passing output generated by one task during a run to be used as input by another task in the same run. This is the mechanism for inter-task communication.

2. The Core Mechanism: Task Values

The primary tool for enabling communication between tasks is the dbutils.jobs.taskValues utility. A task can use dbutils.jobs.taskValues.set() to declare a named output, and a downstream task can then reference this value.

Let's look at a practical demonstration of how this works.

As the documentation notes, the value you set must be convertible to JSON and have a size limit of 48 KiB. This is suitable for passing control information, metadata, small lists, or configuration details, but not for passing large datasets.

3. Referencing Task Values

Once a task value is set, how do other tasks access it? There are two primary methods, with one being strongly recommended over the other.

Method 1: Dynamic Value References (Recommended)

The modern, recommended approach is to use dynamic value references in the Databricks Jobs UI. This method lets you reference a task value without hardcoding any task names into your notebook code, making your tasks more reusable and your workflow less brittle.

Now, let's see this powerful feature in action. The video uses the task value we saw being set earlier to control an If/else condition task.

Method 2: dbutils.jobs.taskValues.get() (Legacy)

An older method involves using dbutils.jobs.taskValues.get(taskKey="...", key="...") directly within a downstream notebook. The video briefly demonstrates this approach. While it works, it creates a tight coupling between your tasks. If you rename the upstream task in the Jobs UI, you must also update the code in the downstream task, which is a common source of errors. For this course and any professional development, you should prefer dynamic value references.

Hands-On: Building a Dynamic Pipeline

It's time to put these concepts into practice. You will build a two-task job that simulates a common ETL pattern: one task discovers what to process, and a second task iterates through the work items.

Notebook 1: discover_work

Create a new Python notebook with the following code. This notebook will simulate discovering a list of tables to process and an output path for the data.

# 1. Define a widget to accept an input parameter for the environment
dbutils.widgets.text("environment", "development", "Environment (dev/prod)")




# 2. Get the value from the widget
env = dbutils.widgets.get("environment")




# 3. Generate a dynamic output path and a list of tables based on the environment
if env == "production":
    output_path = "s3://my-prod-bucket/processed_data/"
    tables_to_process = ["sales", "customers", "inventory", "stores"]
else:
    output_path = "s3://my-dev-bucket/processed_data/"
    tables_to_process = ["sales_sample", "customers_sample"]




# 4. Set these generated values as task values
dbutils.jobs.taskValues.set(key="data_path", value=output_path)
dbutils.jobs.taskValues.set(key="table_list", value=tables_to_process)




# 5. Print the values for easy debugging in the logs
print(f"Set data_path to: {output_path}")
print(f"Set table_list to: {tables_to_process}")

Notebook 2: process_table

Create a second Python notebook. This one will receive the path and a single table name to process.

# 1. Define widgets to receive parameters
dbutils.widgets.text("table_name", "", "Table to process")
dbutils.widgets.text("path", "", "Output data path")




# 2. Get values from the widgets
table = dbutils.widgets.get("table_name")
data_path = dbutils.widgets.get("path")




# 3. Simulate processing
print(f"--- Starting processing for table: {table} ---")
print(f"Reading from source and writing transformed data to: {data_path}{table}")
print(f"--- Finished processing for table: {table} ---")

Job Configuration Steps:

1. Create a New Job: Go to Workflows and create a new job. Name it Dynamic ETL Pipeline.

2. Create Task 1 (discover_work_task):

   • Name the task discover_work_task.
   • Select the discover_work notebook.
   • Select a cluster.
   • In the Parameters section, leave the environment parameter with its default value of development.
   • Click Create task.

3. Create Task 2 (process_tables_task):

   • Click the + icon below the first task to add a new one.
   • For the task type, select For each. This task type allows you to loop over a list.
   • Name the task process_tables_task.
   • In the Iterate over field, enter the dynamic value reference to get the list from the first task: {{tasks.discover_work_task.values.table_list}}.
   • Now, configure the task that will run for each item in the list.
     • Task type: Notebook.
     • Select the process_table notebook.
     • In the Parameters section for this nested notebook task, add two parameters:
       • Key: path, Value: {{tasks.discover_work_task.values.data_path}} (This passes the path from the first task).
       • Key: table_name, Value: {{input}} (The special {{input}} value refers to the current item from the For each loop).
   • Ensure this task Depends on discover_work_task.
   • Click Create task.

4. Run and Verify:

   • Run the job now.
   • When the run completes, click on the process_tables_task. You will see it expanded into two sub-runs, one for sales_sample and one for customers_sample.
   • Click into each of those sub-runs and inspect the output. You should see the print statements confirming that each table was processed with the correct output path (s3://my-dev-bucket/processed_data/).
   • Bonus: Run the job again using Run now with different parameters. Change the environment parameter to production and observe how the For each task now runs four times, using the production path.

Conclusion

In this lesson, you've unlocked one of the most powerful features of Databricks Workflows: the ability to pass data between tasks. This skill elevates you from building static, linear pipelines to creating dynamic, intelligent, and modular workflows.

Key Takeaways:

• Job Parameters provide top-level configuration, while Task Values enable dynamic data flow within a job run.
• The dbutils.jobs.taskValues.set(key, value) command is used in a notebook to produce an output.
• The {{tasks.TASK_NAME.values.KEY}} dynamic value reference is the standard, most robust way to consume that output in the UI.
• This mechanism is the key to building advanced patterns like conditional logic (If/else task) and dynamic parallelism (For each task).

Preview of the Next Lesson:

You've now covered all the core components of building, orchestrating, and monitoring workflows. To prepare these workflows for a true production environment, you need to handle sensitive information like API keys, database passwords, and service account credentials securely. In the next lesson, "Manage secrets using Databricks secret scopes and the Secrets API," you will learn how to store and access secrets without hardcoding them in your notebooks.