Structuring a Reproducible Research Repository

In our previous lessons, you've developed key skills for the initial phase of a computational project: wrangling data with dplyr and tidyr, and then visualizing it with ggplot2. You now have notebooks, scripts, data files, and plots. But a collection of files is not a project. Without a coherent organizational system, it's easy for your work to become difficult to understand, rerun, or share—even with your future self.

This lesson addresses that challenge directly. We will focus on how to organize your notebooks, source code, data, and outputs into a reproducible research repository. Think of this as the computational equivalent of a well-kept lab notebook and organized freezer boxes. A standardized project structure is a hallmark of professional computational work, making your research more robust, credible, and easier for others (including potential PhD advisors and collaborators) to build upon. This is a foundational skill for creating the high-quality portfolio you're aiming for.

Why a Standard Structure Matters

When starting a project, it's tempting to just put all your files in one folder. As the project grows, this leads to chaos. You might have files like data_final.csv, data_final_v2.csv, and analysis_latest.Rmd, with no clear record of how they were generated or which one is correct. A standardized structure solves this by creating a logical place for everything.

Consistency is more important than finding one "perfect" structure. Adopting a common template across your projects minimizes the mental effort needed to switch between them and makes collaboration much smoother. The video below explains this problem well and introduces a popular template that provides an excellent starting point.

Data Science Tutorial - Project Structure

Please watch the first part of this video from Dave Ebbelaar. He introduces the problem of inconsistent project structures and then walks through how to get started with a widely-used template called "Cookiecutter Data Science."

Focus on the discussion from the beginning up to the point where he shows the generated project folder. Watch from the introduction of the problem and the Cookiecutter template.

A Blueprint for Reproducible Projects

The "Cookiecutter Data Science" template introduced in the video provides a language-agnostic and comprehensive blueprint. While it has a slight Python flavor, its principles are universal.

Let's examine the structure it proposes, which is also championed by The Turing Way, a leading guide for reproducible research.

GitHub - the-turing-way/reproducible-project-template: Template repository for setting a reproducible research project. · GitHub

This GitHub repository is a template implementing The Turing Way's recommendations. We will focus on the directory structure it advocates.

Please examine the ASCII diagram in the section Repo Structure. This layout provides a logical separation for the different components of a data analysis project.

The core idea is separation of concerns. Let's break down the key directories from that template:

data/: This directory holds all project data. Crucially, it's subdivided:
- raw/: Contains the original, immutable data. This data should never be modified by scripts. It's your source of truth.
- processed/: Contains cleaned, transformed, or intermediate datasets generated by your code. Since these files can be regenerated, they are often excluded from version control.
notebooks/: This is where your exploratory analysis lives, in the form of Jupyter or R Markdown notebooks. This is the "lab bench" where you experiment with the data.
src/: For source code. As you identify reusable logic in your notebooks (e.g., a specific data cleaning function or a custom plotting theme), you should extract it into separate scripts in this directory.
reports/: Holds the final outputs of your analysis, such as rendered PDF or HTML reports.
- figures/: A subdirectory for generated plots and figures that will be included in your reports or publications.

This structure creates a clear, automated workflow: scripts in src/ read from data/raw, write to data/processed, and notebooks in notebooks/ use the processed data to generate figures in reports/figures.

The R-centric Approach: Research Compendia

The R community has a well-established convention for project organization called the Research Compendium. It adapts the structure of an R package for research projects, which provides powerful tools for managing dependencies and documentation. This is an excellent model to follow for the R-based work we've been doing.

GitHub - ropensci/rrrpkg: Use of an R package to facilitate reproducible research · GitHub

This guide from rOpenSci explains the "what, why, and how" of research compendia. It shows how a simple project can evolve into a more complex but robust structure.

Read the introduction to understand what a compendium is and why it is beneficial. Then, carefully study the section "How to make a research compendium." Pay attention to the three example structures: simple, intermediate, and complex. Notice how they map to the concepts we've already discussed.

As you can see, the research compendium structure mirrors the Cookiecutter template but with R-specific conventions:

analysis/ serves the same role as notebooks/. It holds your main R Markdown files.
R/ is the direct equivalent of src/, containing your reusable R functions.
The DESCRIPTION file is a unique advantage. It is a simple text file that lists your project's dependencies (e.g., tidyverse, ggplot2). Anyone with your project can then install all required packages with a single command, which is a massive step towards reproducibility.

From Exploration to Reusable Code

A common theme in both templates is the distinction between exploratory notebooks and reusable source code. Your projects should reflect this workflow. A notebook is excellent for trying things out, but once a piece of code is finalized—especially if it's used in multiple places—it should be moved into a function in the src/ or R/ directory.

This practice has several advantages: it makes your notebooks cleaner and focused on narrative, it allows you to test your functions independently, and it prevents you from having multiple, slightly different versions of the same code snippet scattered across your project. The following video explains this crucial point.

7 Tips To Structure Your Python Data Science Projects

In this clip from a video on structuring data science projects, the presenter makes a strong case for moving code out of notebooks and into shared packages or modules.

Watch Tip number five, which focuses on refactoring reusable code.

Bringing It All Together: A Visual Model

Let's visualize a synthesized, language-agnostic project structure that combines these best practices. This is a robust starting point for any computational biology project.

A simplified diagram showing the key directories and their relationships. Code in src/ and analysis/ reads data from data/ and produces outputs in reports/. The tests/ directory contains code to verify that src/ works as expected.

Notice the inclusion of a tests/ directory. Writing automated tests for your functions (especially those in src/ or R/) is a critical practice for ensuring your code is correct and reliable. It guards against accidentally breaking things when you make changes. We will dedicate a future lesson to writing unit tests, but it's important to plan a place for them in your project structure from the beginning.

Conclusion

In this lesson, we've moved from creating individual analysis artifacts to organizing them into a coherent, professional, and reproducible research repository. You've learned that a standardized project structure is not just about neatness; it is fundamental to the reliability and longevity of your work.

Here are the key takeaways:

Adopt a Consistent Structure: Use a standard template like Cookiecutter Data Science or an R Research Compendium for all your projects.
Separate Concerns: Keep your data, code, and outputs in distinct directories.
Protect Raw Data: Your data/raw directory is sacred. Treat it as read-only. All data cleaning and processing should be done programmatically, saving results to a data/processed directory.
Evolve Your Code: Begin exploration in notebooks (analysis/ or notebooks/), but refactor reusable, stable code into functions within a source directory (R/ or src/).
Document Everything: Use a README.md file as the entry point to your project, explaining what it does and how to run it. Use a DESCRIPTION file or requirements.txt to make dependencies explicit.

You now have a blueprint for organizing your work in a way that will be impressive in a portfolio and invaluable in a collaborative research environment.

In our next lesson, we'll address how to manage the evolution of this well-structured project. You'll learn how to use a Git feature-branch workflow to track changes, experiment with new ideas safely, and integrate them back into your main project.