Managing Job Configurations with the REST API

1. The "Why": Automating Job Management for CI/CD

In a professional data engineering environment, you typically work with multiple workspaces, such as development (dev), staging, and production (prod). Code and configurations are developed in the dev environment, tested in staging, and finally deployed to prod.

Manually recreating or updating job settings in each environment using the UI is time-consuming and highly prone to error. The goal of CI/CD is to automate this promotion process.

By treating your job definitions as JSON documents, you can store them in version control (like Git) and use the REST API to programmatically deploy and update them. This ensures consistency and reliability across all your environments.

2. The Core Workflow: Get, Modify, Apply

The fundamental pattern for programmatically managing job configurations is a three-step process:

1. GET: Retrieve the job's current JSON configuration from Databricks.
2. MODIFY: Alter the JSON object locally to make the desired changes (e.g., change the schedule, update a notebook path, increase cluster size).
3. APPLY: Send the modified JSON back to Databricks to update the job's configuration.

A great way to get started is to configure a job in the UI just the way you want it, and then export its JSON definition. This gives you a valid template to use in your automation scripts.

The following video demonstrates this "UI-to-JSON" technique.

Once you have the job's JSON, you can use API endpoints to manage it.

3. Updating Job Definitions: update vs. reset

The Databricks Jobs API provides two primary endpoints for modifying an existing job. It is critical to understand their differences to avoid accidental changes.

• POST /api/2.1/jobs/update: Performs a partial update. It only modifies the fields you specify in the new_settings object. Any fields you omit from the request will remain unchanged. This is ideal for simple tweaks like changing a schedule or a parameter.
• POST /api/2.1/jobs/reset: Performs a complete overwrite. It replaces the entire job configuration with the new_settings you provide. If you omit a setting (e.g., email notifications), it will be removed. This is necessary for more complex changes like altering task dependencies or removing tasks, but it requires you to provide the full desired configuration.

The following article gives a detailed breakdown of these two endpoints with clear examples.

4. Practical Application: Promoting a Job Between Environments

Let's put this all together in a Python script that embodies a real-world CI/CD task: promoting a job from a development workspace to a production workspace.

Scenario: We have a job in our dev workspace that runs a specific notebook. We want to deploy this same job to the prod workspace, but with a few changes:

• The job name should be prefixed with [PROD].
• It should run on a dedicated prod_cluster.
• It should have a stricter timeout.

Here is a Python script that automates this promotion. Given your background in Python and data pipelines, this pattern should feel familiar.

import requests
import json
import os




# --- Configuration for Dev and Prod Workspaces ---
# Best practice: Store these as environment variables or in a secrets manager.
DEV_HOST = os.getenv("DATABRICKS_DEV_HOST")
DEV_TOKEN = os.getenv("DATABRICKS_DEV_TOKEN")
PROD_HOST = os.getenv("DATABRICKS_PROD_HOST")
PROD_TOKEN = os.getenv("DATABRICKS_PROD_TOKEN")

DEV_JOB_ID = 12345  # The ID of the job in the dev workspace
PROD_JOB_NAME = "[PROD] Ingest and Process Sales Data"




# --- 1. GET the job definition from the DEV workspace ---
get_url = f"{DEV_HOST}/api/2.1/jobs/get"
get_headers = {"Authorization": f"Bearer {DEV_TOKEN}"}
get_params = {"job_id": DEV_JOB_ID}

print(f"Fetching job definition for job ID {DEV_JOB_ID} from DEV workspace...")
response = requests.get(get_url, headers=get_headers, params=get_params)
response.raise_for_status() # Will raise an exception for HTTP error codes




# The job definition is in the 'settings' key
job_settings = response.json()["settings"]
print("Successfully fetched job definition.")




# --- 2. MODIFY the settings for the PROD environment ---
print("Modifying settings for PROD deployment...")
prod_settings = job_settings.copy()




# Change the name
prod_settings["name"] = PROD_JOB_NAME




# Change the cluster - here we switch from a job cluster to an existing all-purpose cluster
# First, remove the job cluster definition if it exists
prod_settings.pop("new_cluster", None) 
for task in prod_settings.get("tasks", []):
    task.pop("new_cluster", None)
    task["existing_cluster_id"] = "prod_cluster_id_123" # Replace with your actual prod cluster ID




# Set a stricter timeout
prod_settings["timeout_seconds"] = 3600 # 1 hour




# --- 3. APPLY the new settings to the PROD workspace ---
# We will use jobs/create to deploy it as a new job in prod.
# In a real CI/CD pipeline, you might check if the job exists and use jobs/reset instead.
create_url = f"{PROD_HOST}/api/2.1/jobs/create"
create_headers = {"Authorization": f"Bearer {PROD_TOKEN}"}




# The payload for 'create' is just the settings object
create_payload = prod_settings 

print(f"Deploying job '{PROD_JOB_NAME}' to PROD workspace...")
response = requests.post(create_url, headers=create_headers, json=create_payload)

if response.status_code == 200:
    prod_job_id = response.json()["job_id"]
    print(f"Successfully created job in PROD with ID: {prod_job_id}")
else:
    print(f"Error creating job in PROD: {response.status_code} - {response.text}")

This script demonstrates a complete promotion cycle: fetching a configuration from one environment, transforming it to fit another, and deploying it.

5. Advanced Use Case: Bulk Updates with Scripting

Your expertise with shell scripting and tools like dbt and Airflow means you appreciate the power of command-line automation. The Databricks CLI, which is a convenient wrapper around the REST API, can be combined with other tools like jq for powerful bulk operations.

Imagine you need to pause the schedule for all jobs related to a specific project before a maintenance window. Doing this manually is tedious. A script can do it in seconds.

The following resource shows a practical shell script for just such a task.

This example perfectly illustrates how the API and its CLI wrapper enable you to manage your Databricks environment at scale, just as you would with any other infrastructure.

Conclusion

In this lesson, we elevated your automation skills from simply running jobs to managing their entire lifecycle as code. This is a critical competency for the Databricks Certified Data Engineer Professional exam and for real-world production data engineering.

Key Takeaways:

• Jobs as Code: Job definitions are JSON documents that can be version-controlled, automated, and promoted across environments.
• The "Get, Modify, Apply" Pattern: This is the standard workflow for programmatically updating job configurations.
• update vs. reset: Use jobs/update for partial, targeted changes and jobs/reset for complete configuration overwrites.
• CI/CD Enablement: The Jobs API is the key to building automated deployment pipelines for your Databricks workflows, ensuring consistency and reducing manual error.
• Scaling with Scripts: Combining the API or CLI with scripting tools like Python or shell scripts with jq allows for powerful bulk operations.

Preview of the Next Lesson:

We have now covered several key aspects of running production jobs, from automation to orchestration. Next, we will shift our focus to the underlying infrastructure security. In the upcoming lesson, "Evaluate network security options (e.g., VPC peering, private endpoints) for a given workspace deployment scenario," you will learn how to secure your Databricks environment at the network level, another vital component of a production-ready deployment.