Structuring Multimodal Data for Organization

Principles of a Well-Structured Data Science Project

A good project structure isn't just about tidy folders; it's about enforcing practices that lead to better science. The goal is to make your work logical, transparent, and reproducible.

The following article from Innovatiana outlines several best practices for organizing a dataset. While the original is in French, the key ideas are universal and crucial for us to adopt.

These principles help ensure that anyone (including your future self) can understand the project, trace data from its raw form to the final results, and reproduce your findings.

A Standard Template: Cookiecutter Data Science

Rather than reinventing the wheel, we can start with a widely accepted template. The "Cookiecutter Data Science" project structure is a popular, logical, and comprehensive template for data science work.

The following video by Dave Ebbelaar provides an excellent introduction to why a consistent structure is important and demonstrates how to use the Cookiecutter template.

While you don't need to install and run the cookiecutter command yourself, the structure it proposes is the model we will adapt for your project.

Designing a Structure for the Midday Meal Project

Let's apply these principles and create a concrete structure for your multimodal project. This structure clearly separates raw data, processed data, source code, and exploratory notebooks.

Here is the directory structure we will use. You should create these folders within your main project directory.

midday-meal-monitoring/
├── README.md
├── requirements.txt
├── .gitignore
│
├── data/
│   ├── 01_raw/
│   │   ├── images/         # Scraped meal photos (.jpg, .png)
│   │   ├── text_complaints/ # Scraped text data (.csv, .json)
│   │   └── documents/        # Scraped reports (.pdf)
│   │
│   ├── 02_interim/         # Resized images, cleaned text
│   │
│   └── 03_processed/       # Annotated data for modeling
│
├── notebooks/
│   └── 01_eda.ipynb        # Notebook for exploratory data analysis
│
└── src/
    ├── __init__.py
    ├── data_processing/    # Scripts for processing data
    └── models/             # Scripts for training models

Let's break down the key components:

• data/: This is the heart of your data management. The numbered prefixes (01_, 02_, 03_) enforce the workflow.
  • 01_raw/: This folder is immutable. It holds the original, untouched data you scraped. You should never modify files in here. This ensures you can always go back to the source. We've subdivided it by data type.
  • 02_interim/: This folder holds intermediate data. For example, after you resize all images to a standard dimension, you would save them here.
  • 03_processed/: This folder contains the final, clean datasets that are ready to be fed into your machine learning models. For instance, your annotated image dataset will live here.
• notebooks/: This is where you do your exploratory work. Jupyter notebooks are excellent for iteration, visualization, and initial analysis, but they are not ideal for reproducible pipelines. Your goal is to explore in notebooks and then move mature code into Python scripts within the src/ directory.
• src/: This directory contains all your Python source code. By organizing code into modules and packages, you make it reusable and maintainable. Your data cleaning scripts, model training pipelines, and utility functions will live here.
• README.md: The front page of your project. It should explain what the project is, how to set it up, and how to run the code.
• .gitignore: As you know from your Git experience, this file tells Git what to ignore. It is crucial to add data/ to your .gitignore to avoid committing large data files to your repository. You should also ignore environment files, model artifacts, and Python cache directories (__pycache__).

Automating Organization with a Python Script

Manually sorting hundreds or thousands of scraped files is tedious and error-prone. We can automate this process with a simple Python script. The script will look at files in a "source" folder (e.g., your unsorted downloads) and move them to the correct location within data/01_raw/ based on their file extension.

Python's built-in os and shutil libraries are perfect for these file system operations.

Now, let's create our own organization script. Create a new file named organize_raw_data.py inside your src/ directory.

import os
import shutil




# --- Configuration ---
# The folder where your scraped files are currently located.
# Change this to your actual downloads folder path.
SOURCE_DIR = "/path/to/your/downloads" 




# The root directory for your project's raw data.
# This assumes you run the script from the project's root folder.
RAW_DATA_DIR = "data/01_raw"




# Define mappings from file extensions to destination folders.
# Keys are tuples of extensions (lowercase), values are the folder names.
EXTENSION_MAP = {
    ('.jpg', '.jpeg', '.png', '.gif'): 'images',
    ('.csv', '.json', '.txt'): 'text_complaints',
    ('.pdf', '.doc', '.docx'): 'documents',
}




# --- Script Logic ---

def organize_files(source_path, base_dest_path, mapping):
    """
    Organizes files from a source directory into a structured destination directory
    based on their file extensions.
    """



    # 1. Create all necessary destination subdirectories
    for dest_folder in set(mapping.values()):
        os.makedirs(os.path.join(base_dest_path, dest_folder), exist_ok=True)
    
    print(f"Scanning source directory: {source_path}")




    # 2. Iterate over files in the source directory
    for filename in os.listdir(source_path):
        source_file_path = os.path.join(source_path, filename)
        



        # Skip if it's a directory
        if os.path.isdir(source_file_path):
            continue




        # Find the correct destination folder
        file_ext = os.path.splitext(filename)[1].lower()
        destination_folder = None
        
        for extensions, folder_name in mapping.items():
            if file_ext in extensions:
                destination_folder = folder_name
                break
        



        # 3. Move the file if a mapping was found
        if destination_folder:
            dest_dir = os.path.join(base_dest_path, destination_folder)
            dest_file_path = os.path.join(dest_dir, filename)
            
            print(f"Moving '{filename}' to '{dest_dir}'...")
            shutil.move(source_file_path, dest_file_path)
        else:
            print(f"Skipping '{filename}' (unrecognized extension).")

if __name__ == "__main__":



    # Ensure the base raw data directory exists
    os.makedirs(RAW_DATA_DIR, exist_ok=True)
    
    organize_files(SOURCE_DIR, RAW_DATA_DIR, EXTENSION_MAP)
    print("\nFile organization complete!")

To use this script:

1. Save it as src/organize_raw_data.py.
2. Crucially, update the SOURCE_DIR variable to the absolute path of the folder containing your scraped files.
3. Open your terminal, navigate to the root of your midday-meal-monitoring project folder.
4. Run the script using the command: python src/organize_raw_data.py.

The script will automatically create the subdirectories inside data/01_raw/ and move your images, text files, and documents into their designated homes, leaving your downloads folder cleaner and your project perfectly organized.

Conclusion

In this lesson, we transitioned from being data collectors to being project architects. By establishing a clean, logical, and standard project structure, you've laid a critical foundation that will pay dividends throughout the project lifecycle. This organization prevents chaos, ensures reproducibility, and makes your workflow more efficient.

Key Takeaways:

• A standardized project structure (like the one adapted from Cookiecutter) separates concerns: data, src, notebooks.
• The data directory should be organized into a pipeline: 01_raw (immutable), 02_interim, and 03_processed.
• Always add your data directories to .gitignore to keep your repository lean.
• File system operations can be easily automated with Python's os and shutil libraries, saving time and reducing errors.

Now that your raw data is neatly organized, we are ready to begin processing it. In our next lesson, we will focus on the image data. We will start with fundamental preprocessing tasks using libraries like OpenCV and Pillow, such as loading, resizing, and converting images to prepare them for the annotation and modeling stages.