Skip to main content
grasp.study
About
Explore
Pricing
Log in
Try it for free
Back to course
👋
Welcome to a Grasp lesson
Create a free account
to keep studying and create your own study path.
CI/CD for PySpark data pipelines
·
Module 2
Data Quality Testing and Code Standards
1
PySpark Data Quality Checks
Implement data quality assertion functions (e.g., null checks, schema validation, row count checks) for PySpark DataFrames.
2
PySpark ETL Integration Testing
Write integration tests that validate end-to-end PySpark ETL job outputs against expected results.
3
Mocking External Dependencies in PySpark Tests
Mock external dependencies, such as database connections or file reads, in PySpark tests.
Configuring Ruff for Python/PySpark Linting and Formatting
Configure ruff to lint and format a Python/PySpark project.
5
Automating Code Quality with Pre-Commit Hooks
Set up pre-commit hooks to automatically enforce linting and formatting before each commit.
Previous module
Testing Fundamentals for PySpark ETL
Next module
Building a CI Pipeline with GitHub Actions