Hello! In our last lesson, we focused on using conftest.py to create shared, reusable fixtures, which is fundamental for managing test dependencies in a clean and scalable way. We saw how this helps us apply the DRY (Don't Repeat Yourself) principle to our test setup code.
Now, we'll apply a similar organizational philosophy to the tests themselves. This lesson will teach you how to structure a test suite by separating fast, isolated unit tests from slower, more comprehensive integration tests. You'll then learn to use pytest markers to categorize these tests, enabling you to selectively run specific test groups. For a data engineer, this is a critical skill for building efficient CI/CD pipelines where you might run quick logical checks on every commit, but reserve heavy tests involving real databases or services for nightly or pre-deployment runs.
Unit vs. Integration Tests: A Structural Approach
A mature test suite is not a flat collection of tests; it's typically structured like a pyramid. The base is made of many fast-running unit tests, while the top consists of fewer, slower, but more comprehensive integration or end-to-end tests.
-
Unit Tests: These verify a small, isolated piece of code (a "unit," like a single function or method) without involving external dependencies like databases, APIs, or the filesystem. They are fast and reliable. In your work, this could be a test for a Spark UDF or a data transformation function that operates purely on input data.
-
Integration Tests: These verify that different parts of your application work together correctly, often including real external services. They are essential for ensuring the system works as a whole but are slower and can be more brittle. For example, a test that runs an Airflow DAG and checks if data is correctly written to a data warehouse is an integration test.
A common and effective way to organize these is by creating separate directories within your tests folder.
pytest will automatically discover and run tests from all subdirectories. While this directory-based separation is a great starting point, pytest offers a more powerful and flexible mechanism for categorization: markers.
Introducing Pytest Markers
Markers are decorators that you can apply to test functions or classes to label them with metadata. Think of them as tags. Once your tests are tagged, you can use these tags to selectively run or skip certain groups of tests.
While pytest has useful built-in markers like @pytest.mark.skip and @pytest.mark.xfail, the real power for organizing a suite comes from creating your own. The BugBytes video below gives a solid introduction to the concept.
pytest Markers - Custom and Built-In Markers for Test Filtering
This video introduces the concept of pytest markers, shows how to define custom ones, and demonstrates how to filter tests using the -m command-line option.
First, watch the brief introduction to understand the purpose of markers. Next, skip ahead to see how to define custom markers in your pyproject.toml file. This registration step is crucial. Finally, watch the demonstration on using custom markers to filter test runs. Pay close attention to the pytest -m "not slow" and pytest -m slow commands.
Creating and Filtering with Custom Markers
As the video showed, the process involves two main steps: registering the marker and then applying it.
1. Register Your Markers
To avoid typos and to provide a clear, central definition of your test categories, you must register custom markers in your pytest.ini or, more commonly in modern projects, your pyproject.toml file.
How to Configure pytest for Python Testing
This article shows a clear example of a pyproject.toml file configured for pytest.
Focus on the [tool.pytest.ini_options] section. Specifically, look at the markers list where custom markers like slow, integration, and unit are registered with a brief description. This is considered best practice.
2. Apply and Filter
Once registered, you can apply your markers to any test function or class. A common pattern is to mark any test that relies on external services (like a database or API) as an integration test.
# tests/integration/test_database.py
import pytest
@pytest.mark.integration
def test_user_creation_in_db(database_connection):
# This test talks to a real database
# ...
assert True
@pytest.mark.slow
@pytest.mark.integration
def test_large_data_ingestion(database_connection):
# This test is slow AND an integration test
# ...
assert True
Now, you can use the -m flag to execute specific subsets of your test suite. This is where the organizational structure pays off, especially for creating efficient workflows.
This guide provides an excellent overview of using marker expressions for fine-grained test selection.
Please read the section on Marker Expressions. Pay close attention to the examples combining markers with boolean logic like and, or, and not. The practical CI/CD examples further down illustrate how these commands are used in real-world automation pipelines.
Here are the most common commands you'll use:
-
Run only unit tests (fast feedback):
pytest -m "not integration"
This is perfect for running locally or in the initial stage of a CI pipeline. It executes everything except the tests you've marked asintegration. -
Run only integration tests:
pytest -m "integration"
This is ideal for a dedicated stage in your CI pipeline that runs after the unit tests pass, perhaps against a staging environment. -
Run quick database tests:
pytest -m "database and not slow"
This shows the power of combining markers to run a very specific subset of tests.
Advanced Technique: Auto-Applying Markers
Manually adding @pytest.mark.integration to every test in the tests/integration/ directory can be tedious and error-prone. A more professional approach is to automate this. You can use the pytest_collection_modifyitems hook in your root conftest.py to automatically apply markers based on a test's file path.
This ensures consistency and reduces boilerplate code. Any test added to the tests/integration/ directory will automatically be marked as an integration test.
This section explains a powerful, advanced technique for applying markers dynamically.
Read the section on auto-applying markers. The code snippet for pytest_collection_modifyitems shows exactly how to check a test's path (item.fspath) and add a marker if it's in a specific directory. The accompanying pytest_configure hook shows how to register these auto-applied markers programmatically.
This automatic approach, combined with the directory structure, creates a robust and low-maintenance system for organizing a large test suite.
Conclusion
Today, you've learned how to impose a professional structure on your test suite. By distinguishing between unit and integration tests and using markers for categorization, you gain precise control over which tests to run and when. This is not just an organizational exercise; it directly translates to faster developer feedback and more efficient, cost-effective CI/CD pipelines.
Key Takeaways:
- Separate tests by type: Use directories like
tests/unit/andtests/integration/to physically group tests. - Use custom markers as tags: Decorators like
@pytest.mark.integrationprovide a flexible way to categorize tests. - Register your markers: Always define custom markers in
pytest.iniorpyproject.tomlto provide documentation and avoid warnings. - Filter with
-m: Usepytest -mwith boolean expressions (not,and,or) to run specific subsets of your test suite. - Automate for consistency: Use the
pytest_collection_modifyitemshook to auto-apply markers based on file location, making your test suite easier to maintain.
In our next and final lesson of this module, we will address a crucial question: now that we can write and organize tests, how do we know which parts of our code are actually being tested? We will explore how to generate and interpret test coverage reports to identify untested code paths.