Deploying with Databricks Asset Bundles

1. What are Databricks Asset Bundles? A Quick Recap

As we've discussed, a Databricks Asset Bundle (DAB) is a standardized way to package all your project's components—notebooks, Python code, Delta Live Tables pipelines, and Databricks Jobs—along with their settings. The entire bundle is defined in a databricks.yml file, which acts as the single source of truth for your project.

The real power of bundles for CI/CD lies in their ability to define and manage deployments to multiple environments.

2. Structuring a Bundle for Multiple Environments

The key to managing deployments across different environments is the targets block in your databricks.yml file. A target is a named configuration that represents a specific deployment destination, typically corresponding to a Databricks workspace or a logical environment within a workspace (e.g., dev, staging, prod).

Here’s a conceptual example of a databricks.yml file with three targets:

bundle:
  name: my-data-project




# Include resource definitions from other files
include:
  - resources/*.yml




# Define the deployment environments
targets:



  # Development target
  dev:



    # 'development' mode creates user-isolated resources (e.g., job named 'dev_user@email_com_my_job')
    mode: development
    default: true
    workspace:
      host: https://<dev-workspace>.databricks.net




  # Staging target for integration testing
  staging:
    workspace:
      host: https://<staging-workspace>.databricks.net



    # Deploy jobs and assets with their defined names, no user prefix
    mode: production




  # Production target
  prod:
    workspace:
      host: https://<prod-workspace>.databricks.net
    mode: production



    # Requires an explicit --target prod flag to deploy, preventing accidental deployments

Your experience with tools like Airflow, which often require managing connections and variables for different environments, provides a good parallel. In DABs, targets serve a similar purpose, but they codify the entire environment's configuration, including compute, permissions, and asset-specific parameters, directly within your version-controlled project.

3. Managing Environment-Specific Configurations

Simply defining targets isn't enough. Your staging environment will need different configurations than prod—for example, different source data paths, cluster sizes, or target schemas. Hardcoding these values in your job definitions would defeat the purpose of having a single codebase.

The solution is parameterization using variables. You define variables for all environment-specific settings and then provide different values for each target.

A Structured Approach to Variables

A robust pattern is to separate your variable definitions into a dedicated folder, with files for common and environment-specific values.

By adopting this structure, your databricks.yml file becomes cleaner, and managing configurations for a new environment is as simple as adding a new variable file.

Reusing Configuration with YAML Anchors

For complex configurations that are reused across multiple jobs within the same file, such as a cluster definition, you can use YAML anchors to avoid repetition. This adheres to the Don't Repeat Yourself (DRY) principle.

4. Deploying Pipelines and Associated Assets

The term "assets" in a bundle includes everything your pipeline needs to run. This means not just the job or DLT pipeline definition, but also the notebooks it calls and any custom Python libraries (.whl files) it depends on. When you run databricks bundle deploy, all referenced local files are packaged and uploaded to the target workspace.

For shared code used across multiple projects or bundles, a common pattern is to build it as a Python wheel file and upload it to a location on DBFS or in the workspace. The bundle can then reference this pre-built library.

5. Automating Deployment with GitHub Actions

Now, let's tie everything together. We will extend the CI workflow from the previous lesson to perform Continuous Deployment (CD). The strategy is to trigger deployments based on merges to specific branches, a common practice in Git-based workflows.

The core of this automation is the databricks bundle deploy command, executed within a GitHub Actions workflow.

# Example step in a GitHub Actions workflow
- name: Deploy to Production
  run: databricks bundle deploy --target prod
  env:



    # Service principal credentials for authentication
    DATABRICKS_HOST: ${{ secrets.DATABRICKS_HOST }}
    DATABRICKS_CLIENT_ID: ${{ secrets.DATABRICKS_CLIENT_ID }}
    DATABRICKS_CLIENT_SECRET: ${{ secrets.DATABRICKS_CLIENT_SECRET }}

The following video provides a comprehensive walkthrough of a professional CI/CD pipeline that implements this branching and deployment strategy.

This video clearly distinguishes between CI (running tests on a PR) and CD (deploying after a merge), bringing together the concepts from this lesson and the previous one.

6. Putting it all together: A Practical Scenario

Let's consolidate these concepts with a practical thought exercise.

Scenario:
You are tasked with deploying a Delta Live Tables (DLT) pipeline across three environments. The pipeline's definition is in resources/dlt_pipeline.yml.

Requirements:

1. dev Environment:
   • Target name: dev
   • Runs in development mode to create user-specific instances.
   • Uses a dev catalog.
   • Cluster size: small (1 worker).
2. staging Environment:
   • Target name: staging
   • Runs in production mode.
   • Uses a staging catalog.
   • Cluster size: medium (4 workers).
   • Deployment is automated when code is merged into the release branch.
3. prod Environment:
   • Target name: prod
   • Runs in production mode.
   • Uses a main catalog.
   • Cluster size: large_autoscale (8-16 workers).
   • Deployment is triggered manually in the GitHub Actions UI after a merge to the main branch.

Your Task:
Based on what you've learned, how would you structure your project? Think about:

• What would the targets block in databricks.yml look like?
• How would you organize your variable files (variables/)?
• What would the variable definitions for catalog_name and cluster_config look like?
• How would you reference these variables in your resources/dlt_pipeline.yml file?
• What would the triggers look like in your GitHub Actions workflow file(s)?

Take a few minutes to sketch out a solution. This exercise directly mirrors the kind of thinking required for the Databricks certification exam and real-world projects.

Conclusion

Congratulations! You have now learned how to implement a full CI/CD lifecycle for your Databricks projects. By leveraging Databricks Asset Bundles, you can confidently and automatically deploy your data pipelines and all their associated assets across multiple environments.

Key Takeaways:

• targets are King: The targets block in databricks.yml is the cornerstone of multi-environment deployment.
• Parameterize Everything: Use variables to abstract away environment-specific configurations like catalogs, schemas, and cluster settings. A structured variable file system is a best practice.
• Deploy with a Command: The databricks bundle deploy --target <env> command is the engine that drives your deployments.
• Automate with CI/CD: Integrate the deployment command into your GitHub Actions workflow, triggered by merges to specific branches (dev, main), to achieve Continuous Deployment.
• Bundles Include All Assets: A deployment includes not just the job definition but also all dependent notebooks and libraries, ensuring a complete and consistent package.

Preview of the Next Module:

This lesson concludes our deep dive into CI/CD for Databricks. You have now covered the entire development lifecycle, from writing code and testing it, to deploying it automatically.

In the next module, "Professional Certification Scenarios," we will pivot to applying this knowledge. You will be presented with realistic, exam-style scenarios that require you to analyze a problem and select the most appropriate Databricks tools and patterns to solve it—drawing on everything you've learned about data ingestion, the Medallion Architecture, orchestration, governance, and CI/CD.