Reproducible Folder Structures

Hello! Welcome to the final module of our course, which is dedicated to bringing everything together in an end-to-end project.

In our previous lesson, we dove deep into model interpretability using SHAP. You learned how to explain not just the model's overall behavior but also the "why" behind every single prediction. This skill is a hallmark of an advanced data practitioner.

Now that our analysis and modeling are complete, it's time to address a crucial aspect of professionalism and practicality in data science: how the project is organized. A brilliant analysis can lose much of its value if it's trapped in a messy, unreproducible state.

This lesson directly addresses the learning outcome: Organize the project using a clean, reproducible folder structure. We will move your work from standalone notebooks into a professional, industry-standard project layout that makes it easy to understand, share, and build upon.

1. Why a Single Notebook Isn't Enough

As you've experienced, an end-to-end project involves many steps: data loading, cleaning, exploration (EDA), preprocessing, modeling, and evaluation. While Jupyter notebooks are fantastic for the exploratory phases, a real-world project quickly becomes unwieldy if everything is stuffed into one or two large notebook files.

A well-organized project structure is essential for several reasons:

Reproducibility: Anyone (including your future self!) can easily understand the project flow and re-run your analysis.
Collaboration: Team members can work on different parts of the project without creating conflicts.
Maintainability: It's easier to debug, update, and extend the project over time.
Professionalism: It signals that you follow best practices and treat your data science work as a robust engineering discipline.

To formalize these ideas, let's review some core principles of project organization.

Chapter 3 Organizing your project

This chapter from a course on reproducibility at Johns Hopkins University provides a concise and clear explanation of why a good project structure is so important and what principles define one.

Please read the sections 'Learning Objectives', 'Organizational strategies', and 'Tips for organizing your project'. Focus on the four reasons for organization and the specific, actionable tips provided (like separating raw and processed data).

The key takeaway is that a good structure isn't random; it follows a logical, consistent pattern that separates different components of your work, such as data, code, notebooks, and results.

2. A Standard Template for Data Science Projects

While there's no single "perfect" structure, a widely adopted standard has emerged in the data science community. Understanding this template is a major step toward professionalizing your work.

The following article provides a great breakdown of this common layout.

Best Practices For Data Science Project Workflows and File Organizations

Let's examine a widely-accepted layout for data science projects. This article from Neptune.ai provides an excellent overview of a standard template.

Read the sections 'Why is structure important?' and 'Proposed layout' (including its subsections on Data, Models, Notebooks, and Src). Pay close attention to the breakdown of the data directory into raw, interim, and processed, as this is a critical concept.

This proposed structure can be visualized to better understand the relationships between folders and the flow of data.

Data Science Project Structure and Data Flow — This diagram visualizes a standard project structure. On the left, you see the folder hierarchy, including the important `data`, `models`, `notebooks`, and `src` directories. On the right, it shows how data flows from its raw form to a processed state, connecting the conceptual workflow to the physical file organization.

Here's a summary of the core directories and their purpose:

data/: Holds all project data.
- data/raw/: The original, immutable data. You never modify files in here.
- data/interim/: Intermediate data that has been transformed.
- data/processed/: The final, cleaned data ready for modeling.
notebooks/: Jupyter notebooks, primarily for exploration, analysis, and visualization. These are not for production code.
src/: Source code. Holds reusable Python scripts (.py files) for tasks like data processing, feature engineering, and model training.
models/: Trained and serialized models (e.g., saved .pkl or .joblib files).
reports/: Generated analysis, such as figures, plots, or presentations.
README.md: The top-level file that explains what the project is, who created it, and how to run it.

3. Automating the Setup with Cookiecutter

Creating this folder structure manually for every new project would be tedious and error-prone. Thankfully, the community has built tools to automate this. The most popular is Cookiecutter Data Science.

Cookiecutter is a command-line tool that creates projects from templates. The cookiecutter-data-science template is a pre-built project structure that includes all the directories we just discussed and more.

The following video gives a great practical demonstration.

Data Science Tutorial - Project Structure

This video from Dave Ebbelaar demonstrates cookiecutter, a popular tool for generating a project skeleton from a template. It's a great way to see how to get started quickly.

Watch the video from 3:01 to 7:17. This section covers installing and running cookiecutter, walking through the generated folders, and customizing the template for your own use. Focus on the commands and the explanation of each directory's purpose.

4. Your Turn: Organize Your Salary Prediction Project

Now it's time for the most important step: applying this to your own work. You will use Cookiecutter to create a new, clean project structure and then migrate your existing work into it.

Task:

Install Cookiecutter: Open your terminal or command prompt and run:
```
pip install cookiecutter
```
Generate the Project: In your terminal, navigate to the directory where you want to create your new project folder and run:
```
cookiecutter https://github.com/drivendata/cookiecutter-data-science
```
Cookiecutter will ask you a series of questions (project_name, repo_name, author_name, etc.). Fill them out for your salary prediction project.
Migrate Your Files: Now you should have a new folder with the project structure. Open it and move your existing files into the appropriate new locations:
- Move your original dataset (.csv file) into the data/raw/ directory.
- Move your Jupyter Notebook(s) containing your EDA, modeling, and evaluation into the notebooks/ directory. If you have multiple notebooks, consider numbering them to show the order (e.g., 01-initial-exploration.ipynb, 02-modeling-and-evaluation.ipynb).
- If you saved your final model as a file (e.g., final_model.pkl), move it to the models/ directory.
- Open the README.md file in a text editor and write a brief, one-paragraph description of your project.

After completing these steps, take a moment to browse the new project folder. It should feel much more organized and professional.

5. Using Notebooks the "Right Way"

A key benefit of this structure is separating exploratory code (in notebooks) from reusable, production-ready code (in src scripts). The ideal workflow is to prototype a function in a notebook, and once it works, move it into a .py file in the src directory. You can then import that function into any notebook that needs it.

This avoids code duplication and makes your core logic easier to test and maintain.

How to structure your ML project code

This short clip explains how to use Jupyter notebooks for exploration while keeping reusable code in Python scripts within your src folder. This is a powerful workflow.

Watch from 2:53 to 3:42. The video demonstrates how to import functions from your source files into a notebook and use the %autoreload magic command, which automatically re-imports your code when you make changes to the source file.

This approach gives you the best of both worlds: the interactive, exploratory power of notebooks and the robust, modular nature of scripts.

Conclusion

Fantastic work! You have taken a completed analysis and refactored it into a professional, reproducible project structure. This is a non-trivial step that is absolutely essential for doing data science in a team or production environment.

Key Takeaways:

A standard project structure (with data, src, notebooks, models, etc.) is vital for reproducibility, collaboration, and maintainability.
The data directory should be organized to reflect the data's lineage: raw (original), interim (transformed), and processed (model-ready).
Tools like cookiecutter-data-science automate the creation of this structure, saving time and ensuring consistency.
The best practice is to use notebooks for exploration and move stable, reusable code into Python scripts in the src folder.

Preview of the next lesson:
When you created the project with Cookiecutter, it automatically generated a file called .gitignore. This file is used by Git, the industry-standard version control system. Now that we have a clean, organized project structure, the next logical step is to learn how to track its history and manage changes. In our next lesson, we will cover how to use basic git commands for version control.