Organizing Python Classes into Importable Modules

What is a Python Module?

At its core, a Python module is simply a file containing Python code. Any file with a .py extension can be a module. The functions, classes, and variables you define in that file can be made available for use in other files. In fact, the data_processing.py and test_data_processing.py files we discussed in the last lesson are both modules.

The primary way to use code from one module in another is with the import statement. The Real Python article "Python Modules and Packages" provides an excellent introduction to this concept.

Importing Code From Modules

As you've just read, the import statement is your tool for accessing code from other modules. There are several ways to use it, each with its own advantages. Understanding them allows you to write cleaner and more readable code.

For a practical demonstration of these import styles, Corey Schafer's tutorial is very clear and concise.

To summarize the key import styles from the video:

• import my_module: Imports the entire module. You access its contents using dot notation, like my_module.my_function(). This is explicit and avoids naming conflicts.
• from my_module import my_function: Imports a specific function or class directly into the current file's namespace. You can then call it directly as my_function(). This can be more convenient but increases the risk of name collisions if you import items with the same name from different modules.
• import my_module as mm: Imports the entire module under a shorter alias. This is a common convention for widely used libraries, like import numpy as np.
• from my_module import *: Imports all public names from a module. As the video explains, this is generally discouraged in production code because it can be unclear where a function or variable came from and can lead to unexpected name conflicts.

Refactoring Our Code: Introducing a Class

Let's apply these concepts to the benchmark processing code from our previous lesson. To better represent our data, we'll first define a simple class. A class is a blueprint for creating objects, grouping data (attributes) and functions (methods) that operate on that data. It's a powerful way to structure your program. In professional Python, it's common to place each significant class in its own file.

We'll create a BenchmarkRecord class to hold the data for a single run. Instead of passing around dictionaries, we will now work with a list of BenchmarkRecord objects, which is more explicit and less error-prone.

Here is what our refactored code for processing data might look like in a file named data_processing.py:

# data_processing.py

from typing import List, Dict

class BenchmarkRecord:
    """A simple class to hold a single benchmark record."""
    def __init__(self, model: str, batch_size: int, latency_ms: float):
        if not isinstance(model, str) or not model:
            raise ValueError("Model name must be a non-empty string.")
        if not isinstance(batch_size, int) or batch_size <= 0:
            raise ValueError("Batch size must be a positive integer.")
        if not isinstance(latency_ms, float) or latency_ms < 0:
            raise ValueError("Latency must be a non-negative float.")
            
        self.model = model
        self.batch_size = batch_size
        self.latency_ms = latency_ms

def summarize_benchmark_data(records: List[BenchmarkRecord]) -> List[Dict]:
    """
    Calculates the average latency for each model configuration from a list of records.

    Args:
        records: A list of BenchmarkRecord objects.

    Returns:
        A list of dictionaries, each summarizing results for one configuration.
    """
    processed_data = {}
    
    for record in records:
        config_key = (record.model, record.batch_size)
        
        if config_key not in processed_data:
            processed_data[config_key] = []
        processed_data[config_key].append(record.latency_ms)

    summary_list = []
    for config, latencies in processed_data.items():
        avg_latency = sum(latencies) / len(latencies)
        summary_list.append({
            "model": config[0],
            "batch_size": config[1],
            "num_runs": len(latencies),
            "avg_latency_ms": round(avg_latency, 2)
        })
        
    return summary_list

This module now contains both our BenchmarkRecord class and our summarize_benchmark_data function.

Making a Module Executable

What if you want a file to be both an importable module and a runnable script? For example, you might want data_processing.py to run a self-test or a demonstration if executed directly from the command line.

Python has a special variable, __name__, for this purpose. When you run a file directly, Python sets __name__ for that file to the string "__main__". When you import the file as a module, __name__ is set to the module's filename. We can use this to create a block of code that only runs when the file is executed as a script.

Using this pattern, a file that reads data, processes it, and prints a summary would be structured like this:

# main_script.py
from data_processing import BenchmarkRecord, summarize_benchmark_data



# ... other imports like csv ...

def load_data(filename):



    # Logic to load data from a file and return a list of BenchmarkRecord objects
    pass

def main():
    """The main logic of the script."""
    records = load_data("benchmarks.csv")
    summary = summarize_benchmark_data(records)
    print(summary)




# This block will only execute when you run `python main_script.py`
if __name__ == "__main__":
    main()

Here, the main() function contains the core logic of the script. The if __name__ == "__main__": guard prevents this logic from running automatically if another script, like a test file, were to import main_script.

Grouping Modules into Packages

As you create more modules, you'll want to group them into a directory structure. In Python, a directory of modules is called a package. Packages allow you to create a hierarchical namespace, like benchmark.data_processing or utils.parsers. This is essential for organizing large projects.

To turn a directory into a package, you simply need to place a file named __init__.py inside it. This file can be empty, or it can contain initialization code for the package.

Let's organize our project into a package. A typical project structure would look like this:

benchmark_project/
├── benchmark/
│   ├── __init__.py
│   └── data_processing.py
├── tests/
│   └── test_data_processing.py
└── main.py

• benchmark_project/: The root directory of our project.
• benchmark/: A package containing our application's core logic.
• benchmark/__init__.py: Marks the benchmark directory as a Python package. It can be empty.
• benchmark/data_processing.py: Our module containing the BenchmarkRecord class and processing function.
• tests/: A directory for all our tests.
• main.py: The main script, or entry point, for our application.

With this structure, our main.py would now import the function like this:

# main.py
from benchmark.data_processing import summarize_benchmark_data

And our test file tests/test_data_processing.py would do the same:

# tests/test_data_processing.py
from benchmark.data_processing import BenchmarkRecord, summarize_benchmark_data
import pytest

def test_summarize_happy_path():
    """Tests the basic functionality with valid BenchmarkRecord objects."""
    records = [
        BenchmarkRecord(model="Llama-3-8B", batch_size=8, latency_ms=100.0),
        BenchmarkRecord(model="Mistral-7B", batch_size=16, latency_ms=50.0),
        BenchmarkRecord(model="Llama-3-8B", batch_size=8, latency_ms=102.0),
    ]
    summary = summarize_benchmark_data(records)



    # ... rest of the test ...

Note that for these imports to work correctly, you should run your scripts and pytest from the root directory (benchmark_project/). Python adds the current working directory to its search path, so it can find the benchmark package.

The following resources provide more detail on creating and using packages.

Conclusion

In this lesson, you've learned how to organize your Python project into a clean, maintainable structure. Moving beyond single files and functions is a critical step towards writing professional-grade code.

Here are the key takeaways:

• Any Python file (.py) can be used as a module.
• The import statement allows you to use code from one module in another. Being specific with your imports (from module import name) is generally better than importing everything.
• A class is a blueprint for creating objects, allowing you to bundle data and functionality together.
• The if __name__ == "__main__": idiom is a standard pattern that allows a file to be both an importable module and a runnable script.
• A directory containing an __init__.py file is a package, which you can use to group related modules hierarchically.

You now have a solid framework for structuring a Python application. However, as you start using external packages like pytest and building your own, you'll need a way to manage these dependencies and ensure your project runs consistently across different machines. Our next lesson will address this by teaching you how to create isolated Python environments and manage packages using standard tools.