Skip to main content
grasp.study
About
Explore
Pricing
Log in
Try it for free
Back to course
👋
Welcome to a Grasp lesson
Create a free account
to keep studying and create your own study path.
CI/CD for PySpark data pipelines
·
Module 4
Containerizing PySpark ETL Jobs
Building a PySpark ETL Docker Image
Write a Dockerfile for a PySpark ETL application using an appropriate base image and dependencies.
2
Optimizing Docker Images with Multi-Stage Builds
Use multi-stage Docker builds to create smaller, optimized production images.
3
Testing PySpark Jobs Locally with Docker
Validate the containerized PySpark job by executing it locally using 'docker run'.
Build and Push Docker Images with GitHub Actions
Add a GitHub Actions step to build and push a Docker image to a container registry upon successful CI.
5
Traceable Docker Images: Tagging with Git Commit SHAs
Tag Docker images with the Git commit SHA to ensure version traceability.
6
Running PySpark ETL on Kubernetes with Jobs
Write a Kubernetes Job manifest to run a containerized PySpark ETL job on a cluster.
Previous module
Building a CI Pipeline with GitHub Actions
Next module
GitOps-based CD with Rancher Fleet