Structuring Your Data Engineering Repository

Hello! Let's continue building our local development environment.

In our last lesson, we put a critical foundation in place by setting up a reproducible Python environment using uv. We now have a reliable way to manage our project's dependencies with pyproject.toml and uv.lock. With our tooling for Python code established, we can now zoom out and consider the structure of the entire project. A clean environment is only half the battle; we also need a clean and logical "filing system" for all the different pieces of our pipeline.

This lesson addresses exactly that. We will design and implement a logical directory structure for our project repository. You will learn the principles behind organizing a complex data project that involves multiple technologies—Python scripts, dbt models, Airflow DAGs, and Docker configurations. A well-organized repository is not just a matter of tidiness; it is essential for maintainability, scalability, and clarity, making your project easier to understand, test, and evolve.

Why Project Structure Matters

Before we start creating folders, it's important to understand the 'why'. A disorganized project, where files are thrown together without a clear system, quickly becomes unmanageable.

Installing Clean Code Best Practices for Data Engineering Repositories

This article from The Dutch Engineer's Newsletter, with a guest post by Daniel Beach, highlights the common pitfalls of poorly structured repositories and the benefits of getting it right from the start.

Please read the sections "Common pitfalls and mistakes" and "Benefits of a clean and simple repository". As you read, notice how a lack of structure not only creates technical debt but also signals to other developers (or your future self) that shortcuts are acceptable.

The core principle we will follow to avoid these pitfalls is separation of concerns. This means that files and code related to different functionalities should be kept in distinct, clearly named directories. This makes the project predictable and easy to navigate.

The Anatomy of a Modern Data Project

A modern data engineering project is often a "polyglot" environment, containing not just Python code but also SQL, YAML configurations, shell scripts, and more. Our structure must accommodate all the components of our chosen tech stack: Python for ingestion, dbt for transformation, Airflow for orchestration, and Docker for containerization.

Let's look at the standard conventions for a modern Python project, which will form the core of our repository.

What does the structure of a modern Python project look like?

This video from InfoWorld provides a concise overview of how a modern Python project should be laid out, following guidelines from the Python Packaging Authority.

Watch the following segments: The introduction, explaining the value of standards: Introduction The explanation of the src directory: The src layout The role of pyproject.toml, which you are already familiar with: pyproject.toml The final summary: Sum-up Pay close attention to the recommendation of placing all application code inside a src directory. This practice prevents common import-related issues and clearly separates your own code from configuration files and other artifacts at the project root.

Building on this, a data engineering project requires additional directories for its specialized components.

Installing Clean Code Best Practices for Data Engineering Repositories

Let's return to the Dutch Engineer article. It outlines the foundational pieces every data engineering repository should have.

Read the list under the section "Introduction to baseline DE repository setup". We've already addressed the first point (dependency management) with pyproject.toml. This lesson is about creating the skeleton for the other points, like tests, src, and Docker files.

Designing Our Project's Directory Structure

Now, let's synthesize these best practices into a concrete structure for our pipeline project. We will create a top-level structure that logically separates each component of our stack.

Here is the blueprint for our repository:

data-pipeline-project/
├── .venv/                      # Virtual environment (managed by uv, ignored by Git)
├── airflow/
│   ├── dags/                   # Airflow DAG definition files
│   ├── logs/                   # Local log storage for Airflow
│   └── plugins/                # Custom Airflow plugins (if needed)
├── dbt_project/                # Our dbt project will be initialized here
├── src/                        # Our custom Python source code
│   └── ingestion/              # Scripts for API, CSV, and streaming ingestion
├── tests/                      # Tests for our Python source code
├── .env                        # Untracked file for environment variables (secrets)
├── .gitignore                  # Specifies files for Git to ignore
├── docker-compose.yml          # Defines and configures our services (Postgres, Airflow, etc.)
├── pyproject.toml              # Project metadata and Python dependencies
├── README.md                   # Project documentation and setup instructions
└── uv.lock                     # Pinned versions of all Python dependencies

Let's break down the key directories and their purpose:

airflow/: This directory is dedicated to Airflow. The dags/ subfolder is the most important part; Airflow will automatically discover and load any Python files we place here as DAGs. We will later configure Docker to mount this local directory into our Airflow container.
dbt_project/: This will be a self-contained dbt project, created using the dbt init command. We will set this up in a future module, but we designate a home for it now.
src/: Following modern Python conventions, this is where our own Python code lives. We create a sub-package ingestion to house the scripts that will fetch data from APIs and files. This separates our application logic from the orchestration logic in the airflow/dags folder.
tests/: A dedicated folder for unit and integration tests. Its structure will mirror the src/ directory, which is a standard convention that makes it easy to locate tests for specific modules.
Root files (docker-compose.yml, .env, README.md): The root of the project is reserved for top-level configuration and documentation. docker-compose.yml will be the heart of our local stack, defining all the services we need.

You can see this separation of concerns in practice in many real-world projects.

How to build an automated data pipeline using Airflow, dbt, Postgres, and Superset (Windows 11 WSL)

This video by Calvin Yoon, while covering a full project build, demonstrates this organizational principle organically.

Notice how he creates specific folders for specific tools as he needs them: He creates the airflow/dags structure at this timestamp. Later, when working with dbt, he creates a nested structure within the models folder for sources at this point. This reflects the natural workflow of assigning dedicated locations for different types of files.

Organizing dbt Models

Within our dbt_project directory, the models sub-directory is where all our transformation logic will reside. The organization inside this models folder is critical for maintaining a clean data transformation workflow. It's standard practice to structure it according to data modeling layers, which directly corresponds to the Medallion Architecture we discussed in Module 1.

As the image shows, we will avoid organizing our models by business department or data source. Instead, we will use a layered approach:

models/staging: For initial cleaning, casting, and renaming of raw data from our bronze layer.
models/intermediate: For combining and transforming staging models into more complex, reusable building blocks.
models/marts: For creating the final, wide tables (facts and dimensions) that will power our analytics dashboard and NLP service—our silver and gold layers.

This layered structure creates a clear, directed graph of dependencies, making the entire transformation process traceable, debuggable, and scalable.

This diagram shows how project structures can evolve. Our project will use the "Layered dbt project structure" within a single repository, which is a robust and scalable starting point.

Conclusion

We have now designed the complete architectural skeleton for our data pipeline repository. This structure provides a logical home for every file we will create, from ingestion scripts and transformation models to orchestration DAGs and container definitions.

Here are the key takeaways from this lesson:

Structure is a Feature: A well-organized repository is crucial for project success, preventing technical debt and promoting maintainability.
Separate Your Concerns: The core principle is to group files by their function and the tool they belong to (e.g., airflow/, dbt_project/, src/).
Follow Conventions: We are adopting standard industry practices, such as the src layout for Python projects and a layered approach for dbt models (staging, intermediate, marts).
Our Blueprint: We have a concrete directory structure that will guide our development through the rest of this course.

In your own time, you can go ahead and create these empty directories and files (.gitignore, README.md, etc.) in your local project folder to build out this skeleton.

Next time

With the blueprint of our project repository defined, the next logical step is to bring our architecture to life. In the upcoming lesson, we will write the docker-compose.yml file. This powerful configuration will define and launch all the services our pipeline needs—Postgres, Airflow, and Kafka—and wire them together, turning our static directory structure into a dynamic, running development environment.