Databricks Repos for Source Control

Why Go Beyond Basic Revision History?

As we briefly saw, every Databricks notebook automatically saves a history of its changes. This is great for quickly reverting a mistake. However, it has significant limitations:

• It's isolated: The history is tied to that specific notebook within that specific workspace. It doesn't track changes across an entire project (multiple notebooks, scripts, etc.).
• It lacks structure: It doesn't support structured workflows like feature branching, code reviews via pull requests, or atomic commits that group related changes across multiple files.
• It's not portable: If the workspace is deleted, the history is lost.

For professional development, we need a more powerful solution. This is where Databricks Repos comes in, providing full Git integration.

Let's clarify the distinction with a quick read.

The Databricks Repos Workflow

Integrating Git with Databricks follows a standard development workflow, but with the convenience of performing most actions within the Databricks UI.

The overall process can be visualized as follows:

Our focus today is on the User workflow: cloning a repository, creating branches, developing code, and synchronizing it back to the remote Git provider.

Setting Up Your Git Integration

For this section, I recommend having your Databricks workspace and a GitHub account open so you can follow along. The most common way to connect Databricks to a private Git repository is by using a Personal Access Token (PAT).

This first video from Databricks Academy provides a perfect walkthrough of the entire setup process.

Your Turn: Set Up the Connection

1. Generate a PAT in your GitHub account under Settings > Developer settings > Personal access tokens. Remember to copy it to a safe, temporary place.
2. Add the PAT to Databricks under User Settings > Git Integration. Select 'GitHub' as the provider, enter your GitHub username, and paste the token.
3. Create a new, empty repository in your GitHub account. Give it a name like databricks-de-course. Make sure to initialize it with a README file.
4. Clone the repository into Databricks by going to Workspace > Repos, clicking Add Repo, and pasting your repository's URL.

You should now see your new repository listed under the Repos section in your workspace.

Managing Code: The Edit-Commit-Push Cycle

With the repository cloned, you can now start developing. The best practice is to never commit directly to the main branch. Instead, you create a separate branch for each new feature or bug fix. This isolates your work and makes it easy to manage changes through pull requests.

1. Branching and Making Changes

Let's continue with the Databricks video to see how to create a branch, add a notebook, and commit your changes.

The interface for committing changes gives you a clear view of what has been added, modified, or deleted.

Your Turn: Make Your First Commit

1. In your cloned Databricks Repo, click the main button and select Create Branch. Name it something like my-first-feature.
2. With your new branch selected, create a new notebook inside the repository.
3. Add a simple code cell to the notebook, for example: print("Hello from Databricks Repos!").
4. Click the branch button again (my-first-feature). You will now see your new notebook listed as an unstaged change.
5. Write a brief commit summary (e.g., "Add initial notebook") and click Commit & Push.
6. Go to your repository on GitHub. You should now see your new branch with the notebook you just created.

2. Pulling Changes from Remote

Collaboration often means other team members will be pushing changes to the same repository. To get their latest work, you need to pull those changes into your Databricks workspace.

This synchronization is crucial. Always remember to pull regularly to avoid merge conflicts, especially when working on a shared branch.

Conclusion

In this lesson, you've bridged the gap between interactive development and professional software engineering practices. By integrating Git into your Databricks workspace, you can now manage your data engineering projects with the same rigor as any other software project.

Key Takeaways

• Databricks Repos provides full Git integration, which is superior to the built-in notebook revision history for collaborative and production-level work.
• The connection to a Git provider like GitHub is typically configured using a Personal Access Token (PAT) in your User Settings.
• The standard workflow is to clone a remote repository, create a feature branch to isolate your work, make your changes, and then commit and push them back to the remote.
• To get the latest updates from your collaborators, you use the pull operation.

Next Steps

You now know how to manage your development environment and your source code. The next logical step is to understand the resources that will execute that code.

In our next lesson, we will cover the learning outcome: Create, configure, and manage clusters for running workloads. We will dive into how to set up and manage the compute power that brings your notebooks and data pipelines to life.