Skip to main content
Back to path

Welcome to a Grasp lesson

Create a free account to keep studying and create your own study path.

Lesson 28

DynamicFrame Transformations for Semi-Structured Data

Recommended resources

Hello! Welcome back to our module on Scalable ETL with AWS Glue.

In the last lesson, we designed a complete Glue ETL job, establishing the core pattern of Extract, Transform, and Load. We introduced the two main data structures, AWS Glue's DynamicFrame and Apache Spark's DataFrame, and discussed the strategy of using DynamicFrames for robust ingestion and DataFrames for complex analytical logic.

Today, we'll build directly on that foundation. Our learning outcome is to implement DynamicFrame transformations for semi-structured data processing. We will take a deep dive into the powerful, specialized transformations that DynamicFrames offer to tackle the messy realities of modern data, such as nested JSON and inconsistent data types. Mastering these is crucial for building resilient data pipelines and a key topic for the certification exam.

1. The Challenge of Semi-Structured Data

As a Data Quality Engineer, you're likely familiar with data that doesn't fit neatly into rows and columns. Sources like NoSQL databases (e.g., DynamoDB), application logs, or IoT event streams often produce semi-structured data, typically in JSON format. This data presents two common challenges for ETL processes:

  1. Structural Complexity: Data can be nested, containing objects within objects or arrays of values.
  2. Type Inconsistency: The same field might contain a string in one record and a number in another, leading to schema conflicts.

Spark DataFrames require a single, strict schema, which makes handling this variability difficult. This is precisely where DynamicFrames excel. They were designed to ingest this kind of data without failing, representing inconsistencies in a way that allows you to programmatically resolve them.

2. Handling Structural Complexity: Flattening Nested Data

A common requirement is to flatten nested data into a tabular format suitable for analysis in tools like Athena or loading into a data warehouse like Redshift. DynamicFrames provide powerful transforms for this.

Semi-structured to Relational Schema Transformation
This diagram illustrates the goal: transforming a hierarchical, nested schema (left) into a flat, relational schema (right). Nested elements become columns, and arrays are often broken out into related tables linked by a key.

The Relationalize Transform

The Relationalize transform is the most powerful tool for this task. It unnests a DynamicFrame's nested structures and pivots arrays into a collection of new DynamicFrames that resemble tables in a relational database.

Let's watch a practical demonstration of this transform.

AWS Glue PySpark: Flatten Nested Schema (JSON)

This video from DataEng Uncomplicated provides an excellent walkthrough of using the Relationalize transform to flatten a nested JSON dataset.

Watch the following segments to understand the end-to-end process: The Problem (00:00 - 00:55): Understand the input data: a JSON file with a nested array of addresses for each customer. Creating the DynamicFrame (01:40 - 03:16): Observe how the initial DynamicFrame is created from the cataloged data, preserving the nested structure. Applying Relationalize (03:16 - 05:13): This is the core of the technique. Pay close attention to how relationalize is called and how it produces a DynamicFrameCollection—a group of DynamicFrames. Notice that it creates one for the root data and another for the unnested address data, linked by a generated ID. Joining the Results (05:13 - 07:16): Finally, see how to select the individual DynamicFrames from the collection and use the join transform to combine them into a single, flat, wide table.

Other Flattening Transforms: Unnest and Unbox

While Relationalize is comprehensive, AWS Glue offers other, sometimes simpler, transforms for flattening:

  • Unnest: This transform is a more direct way to flatten struct types. It promotes nested fields to top-level fields, using dot notation for the new field names (e.g., a field address with a sub-field city becomes address.city). It's simpler than Relationalize but less suited for handling arrays of structs.
  • Unbox: This transform handles a unique case where a field in your dataset is a string that contains structured data (e.g., a JSON object stored as a string). Unbox parses this string and converts it into a proper struct type within your DynamicFrame, which you can then process further.

To see examples of these, you can refer to the AWS documentation.

DynamicFrame class - AWS Glue

The official AWS Glue documentation is the definitive source for understanding all DynamicFrame transforms. Let's look up the unnest and unbox methods.

Quickly review the example code in these two sections: Find the section for unbox. Notice how it takes a field named AddressString and converts it from a string into a struct. Find the section for unnest. See how it takes a DynamicFrame with a nested Address struct and promotes its sub-fields (Zip.Code, City, etc.) to top-level columns.

3. Handling Type Inconsistency: The ResolveChoice Transform

When a Glue crawler or a DynamicFrame reader encounters a column with multiple data types (e.g., some records have order_id as 123 (an integer) and others have it as "A-123" (a string)), it doesn't fail. Instead, it creates a special ChoiceType in the schema. This preserves the data but requires you to decide how to handle the ambiguity. The resolveChoice transform is the tool for this.

You have several strategies you can use with resolveChoice:

  • cast:<type>: Attempts to cast all values to a single, specified type (e.g., cast:string).
  • make_cols: Creates a new column for each data type found (e.g., order_id_long and order_id_string), with nulls where the type doesn't apply.
  • project:<type>: Keeps only the values of the specified type and sets the rest to null.
  • make_struct: Creates a struct containing a field for each possible type.

Let's see the most common strategy, casting, in action.

AWS Glue PySpark: Change Column Data Types

This short video clearly demonstrates how to use resolveChoice to correct data types that Glue may have inferred incorrectly or identified as a ChoiceType.

Please watch the following segments: Initial Schema (00:36 - 01:36): The video starts by reading a JSON file where all fields have been incorrectly inferred as string type. Using resolveChoice (01:36 - 03:18): Observe the syntax for the resolveChoice method. The specs parameter is used to provide a list of tuples, where each tuple specifies a column and the desired cast action. Verifying the New Schema (03:18 - 04:06): See how printSchema() confirms that the data types for order_id, total_amount, and order_date have been successfully converted.

Test your understanding!

You are processing a dataset where a product_id column contains both numeric IDs (like 12345) and alphanumeric legacy IDs (like "SKU-54321"). You need to keep all IDs for reporting purposes. Which resolveChoice action would be most appropriate: cast:long or cast:string? Why?

Show answer

The most appropriate action would be cast:string.

Attempting to use cast:long would successfully convert the numeric IDs, but it would fail to convert the alphanumeric IDs, resulting in null values and data loss for those records. By casting everything to a string, you can safely represent both types of IDs in a single column without losing any information.

4. A Practical Workflow for Semi-Structured Data

Combining these transformations gives you a robust pattern for processing semi-structured data in AWS Glue:

  1. Extract: Read the raw, semi-structured data from S3 into a DynamicFrame using create_dynamic_frame.
  2. Resolve Types: Examine the schema for ChoiceType fields. Use resolveChoice to cast columns to consistent, appropriate data types.
  3. Flatten Structure: Use Relationalize or Unnest to flatten nested JSON objects and arrays into a tabular structure.
  4. Join (if needed): If Relationalize produced multiple DynamicFrames, use the join transform to combine them back into a single, wide frame.
  5. Load (Sink): Write the final, clean, and structured DynamicFrame to a new S3 location, preferably in a columnar format like Apache Parquet, using write_dynamic_frame.

This process transforms messy, complex source data into a clean, query-optimized format that is ready for analytics.

Conclusion

In this lesson, we focused on the specialized tools AWS Glue provides for handling the complexities of semi-structured data. You've learned how DynamicFrames are not just a simpler version of Spark DataFrames, but a powerful abstraction with unique capabilities.

Key Takeaways:

  • DynamicFrames are designed to handle structural complexity (nested data) and type inconsistency (ChoiceType).
  • The Relationalize transform is a powerful method to convert nested data into a relational model with multiple DynamicFrames.
  • The Unnest and Unbox transforms provide simpler, targeted ways to flatten structs and parse structured strings.
  • The ResolveChoice transform is essential for fixing schema inconsistencies, allowing you to cast data to a consistent type and prevent data loss.

Next Up:
So far, our jobs have been designed to run on a full dataset. However, in the real world, data arrives continuously. The next lesson, "Configure Glue job bookmarks to process incremental data loads," will teach you how to make your Glue jobs stateful, allowing them to process only new data since the last run. This is a critical feature for building efficient and cost-effective data pipelines.