Scheduling and Monitoring Databricks Jobs

1. Automating Execution: Scheduling Databricks Jobs

Manually running jobs is fine for development and testing, but production pipelines need to run automatically. Databricks Workflows supports several trigger mechanisms to accommodate different use cases.

For now, we'll focus on the most common type: the Scheduled (batch-based) trigger. This allows you to run your job on a recurring basis, such as hourly, daily, or using a custom cron expression. Your experience with schedulers like Airflow will make the concepts here familiar.

Now, let's see how to configure a schedule in the UI.

As you saw, setting a basic schedule is straightforward. You can find the Schedule panel on the right-hand side of the job configuration page. For more complex scheduling needs, you can click Show Cron Syntax to define a schedule with greater precision (e.g., "run at 8:15 AM on the first Monday of every month").

2. Monitoring Job Run History

Once your job is scheduled, you need to be able to track its execution. Databricks provides powerful monitoring capabilities at two levels: for an individual job and for all jobs across your workspace.

Monitoring a Single Job

When you want to check the status of a specific pipeline, you'll go to that job's detail page and click on the Runs tab. This view gives you a detailed history of every time the job was executed.

Reading about the UI is one thing, but seeing it in action solidifies the concepts. The following video demonstrates how to navigate from a job run to the detailed task view to diagnose an issue.

When a run fails, it's important to understand why Databricks marked it as "Failed." A job's final status is determined by its leaf tasks—tasks that have no downstream dependencies.

Workspace-Level Monitoring

Inspecting a single job is useful, but in a production environment with dozens or hundreds of pipelines, you need a high-level overview. The Job runs tab in the main Workflows section provides this "single pane of glass" view.

This workspace-level view is invaluable for identifying systemic issues. For example, if you see a sudden spike in failed runs in the Finished runs count graph, it could indicate a widespread problem, such as a database outage or an issue with cloud storage. The Top 5 error types table can also quickly surface recurring problems across different jobs.

Hands-On: Schedule and Monitor Your Resilient Pipeline

It's time to apply what you've learned. We will use the Resilient Pipeline Demo job you created in the previous lesson.

1. Schedule the Job:

   • Navigate to your Resilient Pipeline Demo job in the Databricks Workflows UI.
   • In the Job details panel on the right, find the Schedule section and click Edit schedule.
   • Set the job to run on a schedule. Choose Every 5 Minutes from the 0 minute of the hour.
   • Ensure the correct Time zone is selected.
   • Click Save. You should now see the schedule details in the UI.

2. Monitor the Runs:

   • Wait for about 10-15 minutes to allow the job to run two or three times. Since the risky_task notebook is designed to fail intermittently, you should get a mix of successful and failed runs.
   • Go to the Runs tab for your Resilient Pipeline Demo job.
   • Analyze the history:
     • Examine the matrix view. Can you see which runs succeeded and which failed? Can you see the Run_Risky_Task being retried (indicated by a longer bar or multiple attempts within a single run)?
     • Click on the Start time of a Failed run. In the run details page, trace the execution path. You should see that the workflow executed the Run_Failure_Path task.
     • Go back and click on the Start time of a Succeeded run. You should see that the workflow executed the Run_Success_Path task.

3. Explore Workspace-Level Monitoring:

   • In the left-hand sidebar, click Workflows, then select the Job runs tab at the top.
   • Find your Resilient Pipeline Demo job in the list. Observe its recent run statuses.
   • Look at the Finished runs count graph at the top. You should see bars representing the successful and failed runs of your job over the last period.

4. Clean Up:

   • Once you are finished, go back to your job's configuration and pause the schedule to prevent it from running further.

This exercise gives you direct experience in both setting up a schedule and using the monitoring tools to interpret the results of a job with complex, conditional outcomes.

Conclusion

In this lesson, you've learned the essential operational skills of scheduling a Databricks job and monitoring its execution history. Automating and observing your pipelines are fundamental activities for any data engineer, and the Databricks UI provides a rich set of tools to support these tasks.

Key Takeaways:

• Jobs can be scheduled to run on a batch (time-based) schedule using a simple UI or a more powerful cron syntax.
• The Runs tab for a specific job is your primary tool for deep-dive analysis, allowing you to view historical performance and drill down into task-level logs and errors.
• The workspace-level Job runs tab provides a high-level "mission control" view, perfect for monitoring the overall health of all pipelines and identifying trends or widespread issues.
• Understanding run statuses and how to trace the execution path in the UI is a critical troubleshooting skill.

Preview of the Next Lesson:

So far, our tasks have been mostly self-contained. To build truly powerful and modular pipelines, tasks need to be able to communicate with each other. In our next lesson, "Use job parameters and task values to pass data between tasks," we will explore how to make your workflows more dynamic by passing values from one task to another, enabling you to build more complex and reusable logic.