Skip to main content
grasp.study
About
Explore
Pricing
Log in
Try it for free
Back to course
👋
Welcome to a Grasp lesson
Create a free account
to keep studying and create your own study path.
Data Engineering Pipeline Building
·
Module 2
Local Development Stack with Docker
1
Reproducible Python Projects
Set up a reproducible Python project using virtual environments and pinned dependencies.
2
Structuring Your Data Engineering Repository
Organize a data engineering project repository with a logical directory structure.
3
Orchestrating Data Pipelines: Airflow, Kafka, and Postgres with Docker Compose
Write a Docker Compose file to run Postgres, Airflow, and Kafka services locally.
4
Configuring Services with .env Files
Configure services with environment variables using a .env file.
5
Connecting to PostgreSQL and Inspecting Schemas
Use a Postgres client to connect to the containerized database and inspect schemas.
6
Pipeline Secrets and Connections Management
Manage secrets and connections for the pipeline services.
Previous module
Data Pipeline Fundamentals
Next module
Implementing Batch Ingestion Patterns