Skip to main content
Back to path

Welcome to a Grasp lesson

Create a free account to keep studying and create your own study path.

Lesson 68

Automated Data Validation with AWS Glue Data Quality

Recommended resources

Hello! Welcome back to our module on Data Governance and Auditing.

In the last lesson, we focused on the "who" of data governance by learning how to apply data classification and tagging to control access with AWS Lake Formation. We established that a solid governance strategy starts with knowing your data and who can use it. Now, we'll shift our focus from access control to quality control. This is a topic that should resonate strongly with your professional experience as a Data Quality Engineer, as we'll be exploring how AWS automates these principles within its data ecosystem.

Today’s lesson directly addresses the learning outcome: Configure AWS Glue Data Quality rules for automated data validation within pipelines. We will cover how to define, measure, and enforce data quality standards for datasets in your data lake and within your ETL jobs, a critical capability for building trustworthy data pipelines and a key topic for the certification exam.

1. The "What" and "Why" of AWS Glue Data Quality

Before diving into the configuration, it's essential to understand the core concepts of AWS Glue Data Quality and the problems it's designed to solve. As you know, data pipelines are only as reliable as the data that flows through them. Garbage in, garbage out. Glue Data Quality provides a managed, serverless way to measure and monitor the quality of your data, preventing bad data from propagating downstream.

Let's start with a short video that introduces the service and its fundamental components.

AWS Tutorials - AWS Glue Data Quality - Automated Data Quality Monitoring

This video from 'AWS Tutorials' provides an excellent conceptual overview of AWS Glue Data Quality, explaining its purpose and the main building blocks you'll be working with.

Please watch from the beginning to 06:44. Focus on understanding: The core problem Glue Data Quality solves in a data lake environment. The two main ways it operates: on data-at-rest (in the Data Catalog) and data-in-motion (within an ETL job). The four essential building blocks: the Glue Data Catalog table, the Rule Set, the Evaluation Task, and the Output (CloudWatch/S3).

To summarize, Glue Data Quality works in two primary modes:

  1. Data Quality on Data-at-Rest: You can run data quality evaluations directly on tables defined in your AWS Glue Data Catalog. This is useful for monitoring the quality of datasets in your S3 data lake on a schedule, independent of an ETL job.
  2. Data Quality on Data-in-Motion: You can embed a data quality check directly into an AWS Glue ETL job as a transformation step. This allows you to validate data as it's being processed and take immediate action, such as stopping the job or routing bad records to a separate location.

2. Defining Rules with the Data Quality Definition Language (DQDL)

The heart of Glue Data Quality is the Data Quality Definition Language (DQDL). This is a simple, domain-specific language you use to author your quality rules. While the syntax is straightforward, it's powerful enough to express a wide range of checks.

A set of rules is grouped into a Ruleset. Here are a few examples:

  • IsComplete "email": Checks that the "email" column has no null values.
  • IsUnique "customer_id": Checks that all values in the "customer_id" column are unique.
  • RowCount > 1000: Checks that the dataset contains more than 1000 rows.
  • ColumnValues "state" in ["CA", "NY", "TX"]: Checks that the "state" column only contains the specified values.

To get a feel for the structure and capabilities of DQDL, let's look at the official documentation.

Data Quality Definition Language (DQDL) reference

The 'Data Quality Definition Language (DQDL) reference' is your comprehensive guide to all available rules and syntax. We'll focus on the core concepts to get you started.

Please review the following sections: Skim the list of rule types under the 'DQDL rule type reference' to see the variety of checks available (Completeness, Uniqueness, Mean, etc.). Read the 'DQDL syntax' section to understand the basic structure of a ruleset and an individual rule. Review the 'Expressions' table to see how you can create conditions (e.g., > 100, between 80 and 100, matches "[a-zA-Z]*"). Don't worry about memorizing every rule; the goal is to understand how rules are constructed.

3. A Practical Walkthrough

Now that we have the foundational concepts, let's see AWS Glue Data Quality in action. The following video from AWS provides a complete demonstration, from automatically generating rules to integrating them into a pipeline.

Measure and Monitor Data Quality of your Datasets in AWS Glue Data Catalog | Amazon Web Services

This official AWS demo, 'Measure and Monitor Data Quality of your Datasets in AWS Glue Data Catalog,' walks through a realistic use case. It's a great way to see how all the pieces fit together.

Watch the video from start to finish (about 9.5 minutes). As you watch, pay attention to these key stages in the process: Automated Rule Recommendation: Notice how Glue analyzes the dataset to suggest a baseline set of rules, saving significant manual effort (01:23). Rule Customization: Observe how the presenter edits the auto-generated rules and adds more complex ones, like ReferentialIntegrity and CustomSQL, to meet specific business requirements (02:48). Evaluation and Analysis: See how to run the evaluation, interpret the data quality score, and drill down into failed rules to understand the root cause (05:02). Notifications & Actions: The video mentions publishing metrics to CloudWatch and EventBridge, which enables automated alerting—a crucial aspect for production pipelines (05:02). ETL Integration: The final part shows how to use the 'Evaluate Data Quality' transform within a Glue ETL job to check data in-motion (08:22).

4. Deep Dive: Data Quality in an ETL Job

The most common use case for a data engineer is to enforce data quality within a pipeline. This is done by adding the Evaluate Data Quality transform to your Glue Studio job. This transform takes your data as input, applies a ruleset, and can then branch the output based on the results.

AWS Glue Studio Data Quality ETL Job Configuration and Results
This image shows the AWS Glue Studio visual editor. A data source is processed by the 'Evaluate data quality' transform. The panel on the right shows configuration options, such as failing the job on a quality error, and the bottom panel previews the results of the rule evaluations.

Let's solidify this concept by reviewing the official AWS tutorial on this specific topic. This will reinforce what you saw in the video with a step-by-step guide.

Evaluating data quality for ETL jobs in AWS Glue Studio

The documentation page 'Evaluating data quality for ETL jobs in AWS Glue Studio' provides a detailed, text-based walkthrough of the process.

Please read through these steps in the tutorial: Step 1: How to add the 'Evaluate Data Quality' node to your visual job. Step 2: How to create a rule using the DQDL builder. Step 3: How to configure the outputs. Pay close attention to the option to 'Add new columns to indicate data quality errors,' which allows for row-level error handling. Step 4: How to configure actions, specifically the options to fail the job if quality checks do not pass.

This ability to programmatically halt a pipeline or divert bad records is fundamental to building robust, automated data engineering solutions.

Test your understanding!

You are building a Glue ETL job to process new customer registrations from a raw S3 bucket to a cleaned S3 bucket. You must enforce the following rules before the data is written:

  1. The email column must contain a valid email format (e.g., user@domain.com).
  2. The age column must contain values between 18 and 120.
  3. The entire job should fail immediately if more than 5% of the records fail these checks.

How would you configure the Evaluate Data Quality transform in your Glue Studio job to meet these requirements? Describe the DQDL rules and the action you would set.

Show answer

Here's how you could configure the transform:

  1. DQDL Ruleset: You would define a ruleset with rules for both email and age, using a threshold for evaluation.

    Rules = [
    ColumnValues "email" matches `^[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\\.[A-Za-z]{2,6}$` with threshold > 0.95,
    ColumnValues "age" between 18 and 120 with threshold > 0.95
]
    • The matches expression uses a regular expression to validate the email format.
    • The between expression checks the numerical range for age.
    • with threshold > 0.95 for each rule means that the rule passes only if over 95% of rows comply. If either rule has less than 95% compliance, it will fail.

  2. Data Quality Action:

    • In the transform's settings, under Data quality actions, you would select the option "Fail job without loading to target data". This ensures that if the ruleset fails (i.e., if either of the rules fails its threshold), the job stops immediately, and no data is written to the cleaned bucket.

Conclusion

In this lesson, you've learned how to bring automated quality control into your data pipelines using AWS Glue Data Quality. This is a crucial skill for any data engineer, ensuring that your data lake and analytics systems are built on a foundation of trustworthy, validated data.

Key Takeaways:

  • AWS Glue Data Quality automates the process of measuring, monitoring, and managing the quality of your data.
  • It operates in two modes: at-rest on Glue Data Catalog tables and in-motion as a transform within Glue ETL jobs.
  • Rules are defined using the Data Quality Definition Language (DQDL), which supports a wide range of checks from completeness and uniqueness to custom SQL.
  • Glue can recommend a baseline ruleset automatically, which you can then customize.
  • Within an ETL job, you can configure actions to fail the pipeline or route bad records when quality standards are not met, preventing the spread of bad data.

Next Up:

We will continue our journey through data governance by looking at another key feature of AWS Glue. Having controlled access to the data and ensured its quality, the next logical question is: "Where did this data come from, and how has it changed?" In the next lesson, we will explore this by learning how AWS Glue automatically captures and visualizes data lineage.