Managing Task Failures and Retries

1. Automatic Recovery with Retry Policies

Not all failures are created equal. Some are permanent, like a syntax error in your code. Others are transient—temporary issues like a brief network outage, a database connection drop, or an API that's momentarily unavailable. For these transient errors, simply trying the task again a few moments later is often enough to succeed.

Databricks allows you to automate this process by setting a retry policy for any task in your job.

To understand the specifics of how to set this up, let's turn to the official Databricks documentation.

As the documentation states, you can specify how many times a task should be retried and the delay between each attempt. This simple configuration can dramatically improve the reliability of your workflows against intermittent issues.

Let's see a visual demonstration of where these settings are located in the UI.

Configuring a retry policy is a best practice for production data pipelines, especially for tasks that communicate with external systems.

2. Managing Task Execution Time

Besides outright failure, another common issue is a task that runs for too long. This can happen for many reasons: an unexpected volume of data, an inefficient query, or a deadlock in a database. Unchecked, a long-running task can waste expensive compute resources and delay downstream processes.

Databricks provides two thresholds to manage task duration:

1. Warning Threshold: Sets an expected completion time. If the task exceeds this duration, it is flagged with a warning, but continues to run. This is excellent for performance monitoring and alerting you when a job is running slower than usual, which might indicate a problem or a need for optimization.
2. Timeout Threshold: Sets a maximum allowed completion time. If the task exceeds this duration, it is forcibly stopped and marked as "Timed Out". This is a critical safeguard to prevent runaway jobs and control costs.

This image shows how a warning threshold appears in the job runs UI, providing a clear visual cue for performance monitoring.

Now, let's consult the documentation to see how to configure these thresholds.

A key takeaway is that if you configure both a Timeout and a Retry Policy, the timeout applies to each individual retry attempt, not the total time across all attempts.

3. Hands-On: Building a Resilient Job

It's time to apply these concepts. We will configure failure handling for the My Practice Workflow job you've been working on.

Part A: Prevent Runaway Jobs with a Timeout

First, let's see the timeout feature in action.

1. Navigate to Your Job: In your Databricks workspace, go to Workflows and select your My Practice Workflow job.
2. Select the Task: Click on the task in the graph to open its configuration panel on the right.
3. Set a Timeout: Under the Advanced settings, find Metric thresholds. Click + Add.
   • Set Metric to Run duration.
   • Leave Warning blank for now.
   • Set Timeout to 1 minute.
   • Click Confirm.
4. Introduce a Delay: Go to the notebook associated with this task. Add a new cell at the end with a command that waits for longer than the timeout.
   • If using Python:

     import time
     print("Starting a long sleep...")
     time.sleep(70) # 70 seconds is > 1 minute
     print("Sleep finished.")
     

   • If using SQL:

     -- This command will run for 70 seconds
     SELECT java_method('java.lang.Thread', 'sleep', 70000L);
     

5. Run and Observe: Go back to the workflow and click Run now. Go to the Runs tab and monitor the new run. After about a minute, it should fail with the status Timed Out. Click on the run to see the details. This confirms your safeguard works.
6. Clean Up: Remove the time.sleep or java_method cell from your notebook.

Part B: Automatically Retry on Failure

Now, let's configure the job to retry after a recoverable error.

1. Introduce a Transient Error: In your notebook, add a cell with the code that failed in the previous lesson's exercise. This simulates a temporary issue we want to retry.
   • If using SQL: SELECT * FROM non_existent_table;
   • If using Python: print(undefined_variable)
2. Configure Retries: Go back to your workflow task settings.
   • Keep the Timeout of 1 minute for now.
   • Under Advanced settings, find Retries. Click + Add.
   • Set Retries to 2.
   • Set Interval to 10 seconds. This tells Databricks to retry the task up to 2 times, waiting 10 seconds between each failed attempt.
   • Click Confirm and then Save changes on the job.
3. Run and Observe: Run the job again. This time, when you view the job run details, you will see the task attempt to run, fail, wait 10 seconds, and then try again. You'll see icons indicating the multiple attempts. The job will ultimately fail (since the error is permanent), but you'll have witnessed the retry policy in action.
4. Final Cleanup: Remove the error-producing cell from your notebook. Also, you can remove the Retry and Timeout configurations from your job task for now.

4. Manual Intervention: Repair and Rerun

Automatic retries are for transient errors. What about permanent ones, like the NameError you just simulated? The system will try a few times and then fail for good. At that point, a data engineer needs to intervene:

1. Diagnose the root cause (as you learned in the previous lesson).
2. Fix the code in the notebook.
3. Rerun the failed part of the job.

For complex workflows with many tasks, you don't want to rerun the entire job from the beginning. Databricks provides a Repair and Rerun feature that intelligently reruns only the failed tasks and their downstream dependencies.

This feature is a huge time-saver in development and production, allowing you to recover from failures without re-processing data that was already successfully transformed.

Conclusion

You've now moved beyond simply monitoring jobs to actively making them more resilient. By intelligently configuring failure handling, you can build data pipelines that are more robust, cost-effective, and require less manual intervention.

Key Takeaways

• Retry Policies are your first line of defense against transient, temporary errors. They are simple to configure and can greatly increase pipeline reliability.
• Duration Thresholds help you manage task execution time. Use the Warning threshold for performance monitoring and the Timeout threshold as a critical safeguard against runaway jobs.
• When automatic retries fail due to a permanent error, the Repair and Rerun feature allows for efficient manual recovery after you've fixed the root cause.

Next Steps

In our next lesson, we will continue exploring how to make jobs more flexible and dynamic. You will learn how to pass parameters to a notebook within a job using widgets. This allows you to create generic, reusable tasks that can be configured at runtime, a powerful technique for building sophisticated workflows.