In our last session, you learned how to classify data as structured, semi-structured, or unstructured. Now that you can identify what the data is, we'll move on to the next logical step: understanding the journey it takes through a data platform.
This lesson focuses on tracing data from its origin (the source) to its final destination (the sink). You'll learn to identify the four key stages of this journey: ingestion, transformation, storage, and serving. This sequence, known as a data pipeline, is the fundamental pattern underlying almost every data engineering project. Mastering this concept will give you a mental map for organizing and understanding any data system you encounter.
The Data Pipeline: From Raw Ingredients to a Finished Meal
Before we dive into the technical terms, let's start with a simple analogy. Think of a data pipeline as a restaurant kitchen.
- Source & Ingestion: Raw ingredients (vegetables, meat, spices) arrive from suppliers. This is like data coming from various sources (apps, sensors, files). The process of receiving the ingredients at the kitchen's loading dock is ingestion.
- Transformation: The chefs wash, chop, season, and cook the ingredients. They are transforming the raw materials into something refined and palatable. In data engineering, transformation involves cleaning, formatting, and enriching raw data to make it useful.
- Storage: The prepared meal is plated and ready in the pass. It's stored in a structured way, waiting to be picked up. In a data pipeline, the transformed data is loaded into a storage system, like a database or a data warehouse.
- Serving & Sink: A waiter picks up the plate and serves it to the customer, who is the final consumer (or sink). The serving stage is how the final data product is delivered to users, dashboards, or applications that need it.
This end-to-end process of turning raw ingredients into a finished meal is exactly what data engineers do with data.
A Visual Journey Through the Pipeline
To get a dynamic overview of this entire process, let's watch a short video from AltexSoft. It visualizes the flow of data through a typical engineering system, covering the key stages we'll be dissecting in this lesson.
Please watch the entire video, "How Data Engineering Works". It's a fantastic high-level introduction to the whole lifecycle.
As you watch, try to map the steps shown in the animation to the kitchen analogy: Where does the data come from? (Data sources) How is it processed and cleaned? (Data processing) Where is it stored? (Data storage) How is it finally used? (Data serving)
Deconstructing the Data Pipeline Stages
The video gave you a great visual model. Now, let's formalize the stages using standard industry terminology. A data pipeline consists of a few major components.
What Data Pipeline Architecture should I use?
This Google Cloud blog post provides an excellent architectural breakdown. We'll start with the high-level components.
Focus on the section that begins right after the initial introductory paragraphs. Read the description of the three major components of a pipeline: Source data, Processing, and Target storage.
As the article explains, we have a source, some processing in the middle, and a target. This is our basic framework. Let's map this to the four stages from our learning outcome.
1. Source and Ingestion
A data pipeline begins with data from one or more sources. These can be anything from a transactional database powering a web application to files uploaded by users or real-time data from IoT sensors.
Ingestion (or Extraction) is the process of retrieving this data from its source and bringing it into our data platform.
What Data Pipeline Architecture should I use?
Let's look closer at the different types of sources.
Read the section describing the different formats of source data, such as Batch, Transactions, and Stream data. Notice how these relate to the data types we discussed in the previous lesson.
2. Transformation
Raw data is rarely usable in its original state. It might have errors, missing values, or be in the wrong format. The transformation stage is where data engineers apply their craft, cleaning, validating, and restructuring the data to prepare it for analysis.
This could involve:
- Filtering out irrelevant records.
- Removing duplicate entries.
- Converting data types (e.g., changing a text string "2023-10-27" into a proper date).
- Enriching the data by joining it with other datasets (e.g., adding city and state information to a record that only has a zip code).
3. Storage (The Load)
After transformation, the clean, structured data needs a home. The storage stage involves loading the data into a target system, often called a sink. This sink is optimized for the way the data will be used later.
- For structured data used for business analytics, the sink is often a data warehouse.
- For a mix of structured, semi-structured, and unstructured data, the sink might be a data lake.
We now have the three core components of a famous acronym in data engineering: ETL.
The Classic Pattern: ETL (Extract, Transform, Load)
The sequence we've just described—Extract (Ingest), Transform, Load (Store)—is the classic data pipeline pattern, known as ETL.
This Google Cloud article provides a clear definition of ETL and its modern counterpart, ELT.
First, read the section How cloud-based ETL works to solidify your understanding of the three stages. Then, read the section on ETL vs. ELT. Pay close attention to the key difference: when the transformation happens.
As you just read, the main alternative to ETL is ELT (Extract, Load, Transform). In this pattern, you extract the raw data, load it directly into a powerful modern data warehouse (like Google BigQuery), and then transform it inside the warehouse. The rise of powerful, scalable cloud data warehouses has made the ELT pattern increasingly popular, especially for large volumes of data.
4. Serving
The final stage is serving, which is the ultimate purpose of the pipeline. The data has been ingested, transformed, and stored—now it's ready to be used. The serving layer makes the data available to consumers, which could be:
- Business intelligence (BI) tools for creating dashboards and reports.
- Data scientists building machine learning models.
- Other applications that need access to the processed data.
The storage system itself (like a data warehouse) is often also the serving layer.
A Real-World GCP Pipeline in Action
Theory is great, but seeing a real pipeline makes it click. The following video from Google Cloud Tech demonstrates building a complete ETL pipeline using core Google Cloud services. This will bridge the concepts we've discussed to the actual tools you'll use as a data engineer.
How to build a data pipeline with Google Cloud
Watch this demonstration to see how Google Cloud services map to the ETL stages.
First, watch the introduction to understand the architecture (ETL pipeline intro). The components are Cloud Storage, Dataflow, and BigQuery. Ingestion/Extraction: Watch how raw data is landed in Cloud Storage (data in Cloud Storage). Think of Cloud Storage as the kitchen's loading dock. Transformation: Next, see how Dataflow is used to process the data (processing with Dataflow). Dataflow is the chef, transforming the raw ingredients. Load & Serving: Finally, watch how the transformed data is loaded into BigQuery (loading into BigQuery). BigQuery acts as both the final plate (storage) and the serving table for analytics. You can also watch the final recap which summarizes the entire flow.
By watching that demo, you've now traced data from a source file, through ingestion into Cloud Storage, transformation with Dataflow, and finally storage and serving with BigQuery. You've seen a complete source-to-sink journey on Google Cloud.
Conclusion
In this lesson, we mapped out the fundamental journey of data. You've gone from a high-level analogy to the specific Google Cloud services that execute each step.
Here are the key takeaways:
- A data pipeline is the process that moves data from a source to a sink.
- The four main stages are ingestion, transformation, storage, and serving.
- ETL (Extract-Transform-Load) is a classic pipeline pattern where data is transformed before being loaded into the target system.
- ELT (Extract-Load-Transform) is a modern alternative where raw data is loaded first and transformed inside a powerful data warehouse.
- On Google Cloud, a common batch pipeline pattern uses Cloud Storage for ingestion, Dataflow for transformation, and BigQuery for storage and serving.
We've covered a lot of ground at a high level. In the upcoming lessons, we will start to zoom in on each of these areas. We'll begin with one of the most common destinations for structured data: the relational database. In the next lesson, you will learn how these databases are organized by identifying primary keys, foreign keys, and relationships within a relational schema.