Scheduling Databricks Jobs

From Manual Runs to Automated Schedules

So far, you have been triggering your jobs manually using the "Run now" button. While useful for development and testing, production pipelines require automation. Databricks Workflows provides several types of triggers for this purpose. The most common is a time-based schedule.

Let's begin by understanding the different trigger types available.

As you saw, while event-based (File arrival) and streaming (Continuous) triggers are powerful, the Scheduled trigger is your tool for classic, time-based batch processing—for example, running a report every morning or updating a table every hour.

Configuring a Job Schedule

Adding a schedule to a job is straightforward. You'll find the configuration options in the Job details panel of the Jobs UI.

Let's watch a quick, official tutorial that walks through the process of adding a simple daily schedule.

Simple vs. Advanced Schedules

Databricks offers two modes for setting up a time-based schedule:

1. Simple: This mode is for common, repeating intervals. You select a period (e.g., Minute, Hour, Day, Week) and an interval (e.g., every 12 Hours). It's intuitive and covers many use cases.

2. Advanced (Cron Syntax): For more complex schedules, you can use Cron syntax. Cron is a time-based job scheduling standard used across many systems. It allows for highly specific schedules, such as "at 10:15 AM on the first Monday of every month."

To learn more about the specifics of the UI and the power of Cron syntax, let's turn to the official documentation.

Key Considerations for Production Scheduling

As you read in the documentation, setting a schedule involves more than just picking a time. Here are the key takeaways for creating robust, predictable schedules:

• Time Zones are Critical: A job scheduled for 9:00 AM will run at very different times depending on whether you select UTC or a local time zone like America/Los_Angeles. For pipelines that need to align with global business operations, using UTC is often the best practice to avoid ambiguity and issues with Daylight Saving Time.
• The Scheduler is Not for Low Latency: Databricks guarantees that a job will not run before its scheduled time, but it does not guarantee it will run exactly at that time. Due to cloud conditions or cluster startup times, a run might be delayed by several minutes. Do not use the job scheduler for tasks requiring sub-minute precision.
• Cron is a Powerful Tool: While you don't need to be a Cron expert for the exam, understanding that it's the underlying mechanism for advanced scheduling is important. The syntax 0 0 12 ? * MON-FRI * might look cryptic, but it precisely means "run at 12:00 PM every weekday." Many online tools can help you generate Cron expressions.

Your Turn: Hands-On Practice

Let's apply what you've learned by scheduling the My Practice Workflow job you built in the previous lessons.

Prerequisites: You should have the My Practice Workflow job ready from our last session.

Part 1: Configure and Test a Simple Schedule

1. Navigate to your My Practice Workflow job in the Workflows UI.
2. In the Job details panel on the right, click Add trigger.
3. Set the Trigger type to Scheduled.
4. Configure a Simple schedule to run Every 5 Minutes. (We're using a short interval for testing purposes). Select your local time zone.
5. Click Save. The UI should now show that the job is scheduled and display the time of the next run.
6. Wait for the job to trigger automatically. Go to the Runs tab to observe the new run initiated by the "Schedule" trigger.
7. Once you've confirmed it runs, go back to the job's main page, click Edit trigger, and then click Pause. This is a crucial step to prevent the job from running repeatedly and consuming resources.

Part 2: Configure an Advanced (Cron) Schedule

Now, let's try a more specific schedule using Cron.

1. Go back into the schedule configuration for your job by clicking Edit trigger.
2. Switch the Schedule type to Advanced.
3. You will see a Cron expression. Let's create a schedule to run the job at the 15-minute mark of every hour (e.g., 1:15, 2:15, 3:15).
4. Enter the following Cron expression into the box: 0 15 * * * ?
5. Click Save.
6. Observe that the UI correctly interprets this as "Every hour at 15 minutes past the hour."
7. Remember to Pause the schedule again after you've confirmed it's configured correctly.

Conclusion

Great work! You have successfully automated a Databricks job using a time-based schedule. You now have control over the entire pipeline lifecycle, from defining the tasks and their dependencies to specifying exactly when the entire workflow should run.

Key Takeaways

• Jobs are automated using triggers, with the Scheduled trigger being the most common for time-based execution.
• Schedules can be Simple (e.g., every 5 hours) or Advanced (using Cron syntax for precise timing).
• Configuring the correct Time Zone is essential for predictable execution, with UTC being a safe choice to avoid daylight saving issues.
• The Databricks job scheduler is designed for reliability, not low-latency execution. Expect potential minor delays in job start times.
• Active schedules can be Paused and resumed, which is useful for maintenance or temporarily disabling a pipeline.

Next Steps

You've mastered triggering jobs based on a clock. But what if you want a pipeline to run not at a specific time, but as soon as new data arrives? In our next lesson, we will explore another powerful trigger type and learn how to configure a file arrival trigger to run a Databricks job when new data lands.