Building a Reproducible Python Project Structure

Hello. In the previous lesson, you separated an automated trading system into data, signal, portfolio, risk, execution, and monitoring responsibilities. Those boundaries need a practical home: a project where research can be repeated, code can be tested, and a later broker integration cannot accidentally depend on files or packages installed months ago.

This lesson sets up that home. You will create a small but professional Python repository for an Indian-equity swing-research project: an isolated environment, importable source modules, exploratory notebooks, deterministic tests, dependency records, and Git history. The aim is not a large framework. It is a structure that remains understandable when the project grows from one return calculation into a trading workflow.

Study time: about 40 minutes.

---

Reproducibility is an operational control

A notebook that worked once on one laptop is not yet reliable research. For trading, you need to be able to answer basic questions later:

• Which Python version and package versions produced this result?
• Which code version generated this signal?
• Can a clean machine run the tests and reproduce the calculation?
• Which files are disposable local artifacts, and which are part of the system’s record?
• Where should exploratory work stop and reusable logic begin?

Reproducibility has several layers. This lesson establishes code and environment reproducibility. Later modules will add data snapshots, market-calendar logic, adjustment policies, and run metadata. A clean project structure makes those later controls much easier to add.

A useful rule is:

Keep reusable decisions in source code, exploratory investigation in notebooks, verification in tests, and the history of all of them in Git.

The short video below gives a visual orientation to this layout before you build it.

---

The repository: a place for each kind of work

The Cookiecutter-style organization shown below is a useful reference: project-level files live at the root, notebooks are distinct from source code, and data-related artifacts do not get mixed into implementation modules.

For this course, create a project directory called swing-research. Its initial layout should look like this:

swing-research/
├── .gitignore
├── README.md
├── pyproject.toml
├── requirements.lock
├── data/
│   └── README.md
├── notebooks/
│   └── 01_explore_prices.ipynb
├── src/
│   └── swing_research/
│       ├── __init__.py
│       └── features.py
└── tests/
    └── test_features.py

The names have deliberate meanings:

Location	Purpose	Should Git track it?
README.md	Setup instructions, project purpose, data policy, and test command	Yes
pyproject.toml	Project metadata, direct dependencies, build settings, and tool configuration	Yes
requirements.lock	Exact installed dependency versions for a known working environment	Yes
src/swing_research/	Reusable importable Python code	Yes
notebooks/	Exploration, charts, and research narration	Yes, when curated
tests/	Automated checks of deterministic source-code behavior	Yes
data/	Local datasets or pointers to retrieval procedures	Usually no for large or licensed data
.venv/	The local virtual environment	No
.env	Local secrets such as API credentials	No

A few choices matter especially for an algo-trading project:

• Use the distribution name swing-research, but the Python import name swing_research. Hyphens work in package-distribution names; underscores work in Python imports.
• Do not store API keys, request tokens, broker credentials, or account exports in a notebook, source file, or Git commit.
• Do not treat a local CSV file as an undocumented permanent data source. For now, keep a data/README.md explaining what belongs there. A later module will build the proper raw, cleaned, and feature-ready data layers.

The src directory is useful because it prevents your project root from accidentally masquerading as installed application code. Tests and notebooks must import the package you installed, which is closer to how a scheduled research or trading job will actually run.

---

Create an isolated Python environment

A virtual environment gives this repository its own Python packages. Installing pandas for this project will not modify another project, and an unrelated global package upgrade is less likely to change a backtest silently.

From a terminal, create the directory, enter it, initialize Git, and create the environment:

mkdir swing-research
cd swing-research
git init
python -m venv .venv

Activate the environment using the command appropriate to your shell:

# macOS or Linux
source .venv/bin/activate

# Windows PowerShell
.venv\Scripts\Activate.ps1

:: Windows Command Prompt
.venv\Scripts\activate.bat

Then upgrade the package installer within the environment:

python -m pip install --upgrade pip

A cross-platform way to confirm that the environment is active is:

python -c "import sys; print(sys.executable)"

The displayed path should contain .venv. If it does not, stop and activate the environment before installing anything.

Declare direct dependencies

Create pyproject.toml in the project root:

[build-system]
requires = ["setuptools>=68"]
build-backend = "setuptools.build_meta"

[project]
name = "swing-research"
version = "0.1.0"
description = "Research tools for a rules-based Indian equity swing strategy"
requires-python = ">=3.11"
dependencies = [
    "numpy>=1.26,<3",
    "pandas>=2.2,<3",
]

[project.optional-dependencies]
dev = [
    "jupyterlab>=4,<5",
    "pytest>=8,<9",
]

[tool.pytest.ini_options]
testpaths = ["tests"]

This file records the direct packages your project requires:

• pandas and numpy will support market-data transformations and feature calculations.
• pytest runs tests.
• jupyterlab supports research notebooks.
• setuptools provides the minimal machinery needed to install your source package.

Install the project itself in editable mode, including development tools:

python -m pip install -e ".[dev]"

Editable installation means imports use your current files under src/; changing features.py does not require reinstalling the package each time.

Now record the full, exact set of packages that actually installed:

python -m pip freeze --exclude-editable > requirements.lock

This creates a lock-style record that includes indirect dependencies too. Keep both files:

• pyproject.toml explains the intended direct dependencies and compatible version ranges.
• requirements.lock records the exact known-working environment.

To recreate the environment later on a clean machine, first create and activate a new virtual environment, then run:

python -m pip install -r requirements.lock
python -m pip install -e .

The environment directory itself is never the reproducibility record. It is an expendable local artifact that can always be recreated.

---

Turn a calculation into an importable module

Create src/swing_research/__init__.py. It can initially be empty. Its purpose is to make the directory an explicit Python package.

Then create src/swing_research/features.py:

import pandas as pd


def simple_returns(close: pd.Series) -> pd.Series:
    """Calculate one-period arithmetic returns from positive close prices."""
    close = close.astype("float64")

    if (close <= 0).any():
        raise ValueError("close prices must be positive")

    return close.pct_change(fill_method=None)

This is intentionally small. The key design choice is not the formula; it is the location and interface.

The function:

• takes a pandas.Series of close prices;
• produces a pandas.Series of one-period arithmetic returns;
• does not download data;
• does not read a notebook-global variable;
• does not submit an order;
• rejects an input that cannot represent a valid positive close price.

That makes it reusable by a notebook, a backtest, a scheduled data job, or a future signal module.

Your notebook may call it like this:

from swing_research.features import simple_returns

returns = simple_returns(close_prices)

The notebook should explain the question being investigated, display charts or tables, and call reusable functions. Once a calculation is worth trusting or repeating, move it into src/swing_research/ and leave the notebook as evidence of the research process.

Before committing a notebook, use its “restart kernel and run all” command. If the notebook fails after a clean restart, it probably relied on hidden state created by cells executed in an unusual order.

---

Test the source code, not the notebook state

A test is a compact statement of a behavior that must remain true. It is especially valuable in market-data code because a seemingly harmless refactor can shift a series, change a missing-value convention, or alter a calculation without producing an obvious crash.

Create tests/test_features.py:

import pandas as pd
import pytest

from swing_research.features import simple_returns


def test_simple_returns_known_prices():
    close = pd.Series([100.0, 110.0, 100.0], name="close")

    result = simple_returns(close)

    assert result.iloc[0] != result.iloc[0]
    assert result.iloc[1] == pytest.approx(0.10)
    assert result.iloc[2] == pytest.approx(-1 / 11)


def test_simple_returns_rejects_non_positive_prices():
    close = pd.Series([100.0, 0.0])

    with pytest.raises(ValueError, match="positive"):
        simple_returns(close)

The first assertion uses the fact that a missing floating-point value does not equal itself. The test then verifies the two known returns:

$$\frac{110}{100}-1=0.10$$

and

$$\frac{100}{110}-1=-\frac{1}{11}$$

Run the tests from the repository root:

python -m pytest

Keep test data small, explicit, and deterministic. Do not make a unit test depend on a broker API, today’s market data, an internet connection, or a live account. Those are integration concerns, not reliable unit tests.

The tests/ directory mirrors the source conceptually: features.py is tested by test_features.py. As the project grows, this convention makes it easy to find the verification for a calculation.

---

Make Git record the project, not your machine

Create .gitignore:

# Python environments and caches
.venv/
__pycache__/
*.py[cod]
.pytest_cache/

# Notebook temporary files
.ipynb_checkpoints/

# Secrets
.env

# Local trading artifacts and downloaded data
data/*
!data/README.md
models/

Then create data/README.md with a short statement such as:

# Local data

Downloaded market data is not committed to this repository.
Record the source, retrieval time, symbols, and processing procedure with each research run.

This policy is deliberately conservative. Historical equity data can be large, revised, licensed, or account-specific. Later, you will formalize exactly how data snapshots and metadata are stored. For now, the repository records the code and instructions needed to obtain and process data, not arbitrary local downloads.

Write a practical README.md. At minimum, include:

# Swing Research

Research code for a rules-based Indian equity swing strategy.

## Setup
1. Create and activate `.venv`.
2. Install the package with `python -m pip install -e ".[dev]"`.
3. Run tests with `python -m pytest`.

## Repository rules
- Reusable code lives in `src/swing_research`.
- Notebooks import source modules rather than duplicating logic.
- Credentials and downloaded data are never committed.

Inspect what Git sees:

git status

Then make the first commit:

git add .gitignore README.md pyproject.toml requirements.lock data src tests notebooks
git commit -m "Initialize reproducible swing research project"

A .gitignore only protects files that have not already been committed. If a secret is ever committed, adding it to .gitignore afterward does not remove it from Git history. Avoid placing secrets in the repository in the first place.

Use Git history as an engineering record. Commit when a coherent change works: for example, “add return calculation,” “validate close prices,” or “add price-data loader.” Small, meaningful commits make it possible to identify when a research result or implementation behavior changed.

---

A working routine

From this point, a disciplined research cycle is simple:

1. Activate .venv.
2. Pull or update the repository before beginning work.
3. Explore a question in a numbered notebook.
4. Move stable logic into src/swing_research.
5. Add or update tests for that logic.
6. Run python -m pytest.
7. Update dependency records only when dependencies actually change.
8. Review git status before each commit.

This routine creates a useful boundary between experimentation and system behavior. A notebook can be messy while you are investigating a hypothesis; the code that turns the hypothesis into a repeatable signal should not be.

---

Key takeaways

You now have the foundation for reproducible trading research:

• A virtual environment isolates project dependencies from the rest of your machine.
• pyproject.toml records project intent, while requirements.lock captures an exact working dependency snapshot.
• The src layout keeps reusable, installable code separate from root-level project files.
• Notebooks are for exploration and explanation; reusable calculations belong in source modules.
• Tests verify deterministic behavior with small known inputs.
• Git records code, tests, configuration, and curated notebooks, while .gitignore excludes environments, secrets, caches, and local data artifacts.

Next, you will begin working with actual multi-stock price data in pandas: loading it, indexing it correctly, filtering it, and joining it without losing track of symbols or dates.