Hello! Welcome to the fourth module of your certification journey. In the previous modules, we covered the foundations of data storage and security, real-time ingestion with Kinesis, and managed streaming with MSK. You've learned how to get vast amounts of data into the AWS cloud. Now, the question is: how do you manage, organize, and prepare that data for analysis and transformation?
This is where AWS Glue comes in. This module will focus on making you proficient with this core data engineering service. Today, we begin with its most fundamental component. Our learning outcome is to configure the AWS Glue Data Catalog as a central metadata repository for a data lake.
Given your experience with S3 and Athena, you've already interacted with the Glue Data Catalog, perhaps without realizing it. When you define a table in Athena over your S3 data, you are creating an entry in the Data Catalog. This lesson will formalize that knowledge, exploring what the catalog is, why it's the backbone of a data lake on AWS, and how to configure its core components.
1. What is a Data Catalog?
Before we dive into AWS Glue, let's establish the "why." Imagine a massive library with millions of books but no index cards, no Dewey Decimal System, and no search computer. Finding anything would be impossible. A data lake without a catalog is exactly that: a vast storage repository (like S3) full of data files that are undiscoverable and unusable.
The Data Catalog acts as that structured index for your data. It doesn't hold the data itself, but it stores metadata—data about your data.
To understand the key role of a data catalog, let's start with a clear definition of its purpose and components. This article provides an excellent overview.
Read the sections 'What is Data Cataloguing?' and 'Architecture Overview'. Focus on understanding the type of information a catalog stores (schema, location, etc.) and its importance for making data discoverable and enabling ETL pipelines.
As the article highlights, the workflow is straightforward but powerful: data resides in a service like S3, a process discovers its schema, and that schema is stored as metadata in the Data Catalog. Once cataloged, services like Amazon Athena, Amazon EMR, and AWS Glue ETL jobs can use this metadata to read and process the data without needing to know the low-level details of file formats or locations.
2. The Core Components: Databases and Tables
The AWS Glue Data Catalog has a simple, two-level hierarchy that should feel familiar from traditional database systems:
-
Databases: A database in Glue is simply a logical grouping or a namespace for your tables. It helps you organize tables that belong to a specific project, data source, or stage in your pipeline (e.g.,
bronze_layer_db,gold_layer_db). It doesn't store data itself. -
Tables: This is the most critical concept. A Glue Table is a metadata definition that represents a dataset. It is crucial to remember that the table does not store the actual data. It's a pointer, containing information like:
- Schema (column names and data types)
- Data location (e.g., the S3 path)
- Data format (e.g., CSV, JSON, Parquet)
- Partition information
AWS Glue Tutorial for Beginners [NEW 2024 - FULL COURSE]
Let's watch a short video that clearly explains what the Glue Data Catalog is and reinforces the distinction between the metadata (in the catalog) and the actual data (in S3).
Watch the sections where the presenter discusses the Data Catalog, Glue databases, and Glue tables. Pay close attention to his repeated emphasis that the data itself does not move and resides in its original store.
This separation of metadata from data is what makes the AWS data lake architecture so flexible and scalable. Your data can live in cost-effective S3 storage, while the lightweight metadata in the Glue Data Catalog makes it structured and queryable.
3. Populating the Data Catalog
Now that we understand the components, how do we get metadata into the catalog? There are several ways, but they fall into two main categories: manual and automatic.
-
Automatic Discovery (Crawlers): This is the most common and powerful method. You point a Glue Crawler at your data source (e.g., an S3 prefix), and it automatically infers the schema and creates the table definition for you. We will dedicate the entire next lesson to crawlers.
-
Manual Definition: You can define databases and tables directly using the AWS Management Console, CLI, or SDKs. This is useful when you have a non-standard data format that a crawler can't understand, or when you want explicit control over the table definition.
-
Through other services:
- AWS Glue ETL jobs can create new tables in the catalog as outputs.
- Amazon Athena
CREATE TABLEDDL statements register the table in the Glue Data Catalog. This is what happens when you create a table via the Athena query editor.
To truly understand how a Glue Table is constructed, let's walk through the manual process.
AWS Glue Tutorial for Beginners [NEW 2024 - FULL COURSE]
The same video provides a clear, step-by-step demonstration of creating a database and then manually adding a table to it in the AWS console.
Follow along as the presenter first creates a database named 'raw-data'. Then, watch carefully as he manually adds the 'customers_raw' table. Notice all the details he has to specify: the table name, the database it belongs to, the S3 location of the data, the data format (CSV), and each column name and data type.
This manual process makes it very clear what a Glue Table truly is: a collection of metadata attributes that, when combined, describe a dataset stored elsewhere.
Test your understanding!
A colleague has just uploaded a new dataset as a collection of Parquet files into the s3://my-company-datalake/new-project/ bucket. They ask you to "load the data into the Glue Data Catalog."
What is technically incorrect about this request, and what would be the correct description of the task?
Show answer
The technically incorrect part is the phrase "load the data into the Glue Data Catalog." The Data Catalog stores only metadata, not the actual data itself. The data remains in the S3 bucket.
The correct description of the task would be: "Create a table in the AWS Glue Data Catalog that defines the schema and location of the new dataset in the s3://my-company-datalake/new-project/ bucket." This correctly reflects that you are creating a metadata entry that points to the data in S3.
4. The Data Catalog in a Data Lake Lifecycle
The Data Catalog isn't just for raw data. It serves as the central inventory for your data at every stage of its lifecycle. A common pattern for organizing a data lake is the Medallion Architecture, which consists of three main layers:
- Bronze (Raw): Data as it was ingested, in its original format.
- Silver (Cleaned/Enriched): Data that has been cleaned, filtered, and joined with other datasets.
- Gold (Aggregated): Data that is aggregated and ready for business intelligence and reporting.
For each of these layers, which are typically different prefixes in S3, you will have corresponding tables in the Glue Data Catalog.
Building Data Lakes on AWS: Build a simple Data Lake on AWS with AWS Glue, Amazon Athena, and S3
Let's see a practical example of how the Data Catalog is used across the different layers of a data lake. This video demonstrates building a data lake using the medallion architecture.
Watch the following segments to see how the Data Catalog evolves as data moves through the pipeline: Architecture (04:49 - 05:27): Understand how the catalog fits into the overall architecture. Cataloging Source Systems (07:48 - 13:33): See how crawlers populate the catalog with metadata about the original databases (this is a preview for our next lesson). Bronze Layer Tables (19:10 - 20:54): Observe the new 'bronze' tables appearing in the catalog after the first ETL step. Silver Layer Tables (24:10 - 24:55): Note the creation of 'silver' tables representing cleaned data. Gold Layer Tables (35:54 - 38:46): Finally, see the 'gold' tables that represent the final, curated datasets ready for analysis. You don't need to follow every detail of the ETL jobs themselves; focus on how the list of tables in the Data Catalog grows and becomes more refined at each stage.
This demonstration shows that the Glue Data Catalog provides a unified view of all data assets, from their rawest form to the final, business-ready aggregations, making it the indispensable backbone of a modern data platform on AWS.
Conclusion
In this lesson, we've established the foundational role of the AWS Glue Data Catalog. It is not just a feature of AWS Glue but a cornerstone of the entire AWS analytics ecosystem.
Key Takeaways:
- The Glue Data Catalog is a centralized, Hive-compatible metastore that acts as an index for your data lake.
- It stores metadata (like schema and location), not the actual data, which typically resides in Amazon S3.
- The catalog is organized into Databases (logical containers) and Tables (metadata definitions).
- You can populate the catalog manually through the console/API or automatically using Glue Crawlers.
- It serves as the single source of truth for data across all stages of an ETL pipeline, as seen in the Medallion Architecture.
Next Up:
Manually defining tables is insightful, but it doesn't scale. In our next lesson, we will dive deep into the primary method for populating the Data Catalog: how to "Create and run AWS Glue Crawlers to discover and catalog data schemas" automatically.