Automating Databricks Jobs with the REST API

1. Preparing for API Interaction

Before making any API calls, we need two essential pieces of information:

1. Your Databricks Workspace URL: This is the base URL for all API endpoints, for example, https://<your-workspace-instance>.cloud.databricks.com.
2. A Personal Access Token (PAT): This acts as your password for API authentication. It's a secure string you generate in your user settings.

The following short video explains exactly where to find your workspace URL and how to generate a PAT. It also introduces Postman, a popular tool for testing APIs, which is an excellent way to get started interactively.

With your workspace URL and PAT, you are ready to start making API calls.

2. Triggering a Job with the REST API

The primary endpoint for running a job on-demand is POST /api/2.1/jobs/run-now. To use it, you need to know the job_id of the job you want to run. You can find this ID in the Databricks UI on the job's detail page.

Let's see how to structure this API call using Postman. This hands-on demonstration will show you how to trigger a job and even override its default parameters.

Key Components of the run-now Request

As you saw, a typical request to trigger a job includes:

• Method: POST
• URL: https://<your-workspace-url>/api/2.1/jobs/run-now
• Headers:
  • Authorization: Bearer <your-personal-access-token>
  • Content-Type: application/json
• Body (JSON):

  {
    "job_id": 123456789,
    "notebook_params": {
      "input_path": "/some/new/data.csv",
      "process_date": "2024-08-15"
    }
  }
  

The response to a successful run-now call contains the run_id. This ID is essential for the next step: monitoring.

3. Monitoring a Job Run

Once a job is triggered, it goes through several states: PENDING (while waiting for a cluster), RUNNING, and finally a terminal state like TERMINATED (if successful) or FAILED. To build reliable automation, you need to be able to track this progress.

The endpoint for this is GET /api/2.1/jobs/runs/get. It takes the run_id you received from the run-now call as a parameter.

A common pattern is to poll this endpoint in a loop until the job reaches a terminal state. The following video demonstrates exactly this process using a Python script.

A Python Snippet for Triggering and Monitoring

Combining these concepts, here is a practical Python script that demonstrates the end-to-end flow. This leverages your experience with Python for a more robust implementation than manual curl or Postman calls.

import requests
import json
import time
import os




# Best practice: use environment variables for secrets and configs
DATABRICKS_HOST = os.getenv("DATABRICKS_HOST") # e.g., "https://<your-workspace-instance>.cloud.databricks.com"
DATABRICKS_TOKEN = os.getenv("DATABRICKS_TOKEN")
JOB_ID = 12345 # Replace with your job's ID




# --- 1. Trigger the Job ---
run_now_url = f"{DATABRICKS_HOST}/api/2.1/jobs/run-now"
headers = {"Authorization": f"Bearer {DATABRICKS_TOKEN}"}
payload = {
    "job_id": JOB_ID,
    "notebook_params": {
        "my_parameter": "value_from_api"
    }
}

response = requests.post(run_now_url, headers=headers, json=payload)

if response.status_code == 200:
    run_id = response.json()["run_id"]
    print(f"Successfully triggered job. Run ID: {run_id}")
else:
    print(f"Error triggering job: {response.status_code} - {response.text}")
    exit()




# --- 2. Monitor the Job Run ---
get_run_url = f"{DATABRICKS_HOST}/api/2.1/jobs/runs/get"
params = {"run_id": run_id}

while True:
    response = requests.get(get_run_url, headers=headers, params=params)
    if response.status_code == 200:
        run_state = response.json()["state"]
        life_cycle_state = run_state.get("life_cycle_state")
        result_state = run_state.get("result_state", "N/A")
        
        print(f"Run ID {run_id} is in state: {life_cycle_state}")

        if life_cycle_state in ["TERMINATED", "SKIPPED", "INTERNAL_ERROR"]:
            print(f"Job run finished with result state: {result_state}")
            if result_state != "SUCCESS":



                # Handle failure
                pass
            break
        
        time.sleep(30) # Wait for 30 seconds before polling again
    else:
        print(f"Error getting run status: {response.status_code} - {response.text}")
        break

This script encapsulates the entire process: triggering the job, extracting the run_id, and then polling the runs/get endpoint until the job completes.

Conclusion

In this lesson, you've bridged the gap between UI-based automation and full programmatic control of your Databricks workflows. By mastering the REST API, you unlock the ability to integrate Databricks into any external system, from a custom Python script to a sophisticated CI/CD platform.

Key Takeaways:

• Authentication is Key: All API requests must be authenticated using a Personal Access Token (PAT) in the Authorization header.
• Trigger with run-now: The POST /api/2.1/jobs/run-now endpoint is used to start a job. You must provide a job_id, and you can optionally pass parameters in the JSON body. This call returns a run_id.
• Monitor with runs/get: The GET /api/2.1/jobs/runs/get endpoint, using the run_id, allows you to poll the status of a specific job run.
• The State Model: A job run progresses through a life_cycle_state (e.g., PENDING, RUNNING, TERMINATED). Upon completion, it has a result_state (e.g., SUCCESS, FAILED).
• Programmatic Control is Power: Using tools like Postman for exploration and Python scripts for implementation gives you the power to automate, test, and integrate Databricks jobs at a professional level.

Preview of the Next Lesson:

We have now seen how to trigger and monitor jobs that already exist. But what about managing the job definitions themselves? In the next lesson, "Update job configurations and promote changes between environments using the REST API," we will explore how to use the API to create, update, and delete job specifications, enabling true end-to-end automation for your deployment pipelines.