Orchestrating ETL with Step Functions, Glue, and Lambda

A Common ETL Pipeline Architecture

Let's start by visualizing a typical serverless ETL pipeline that we might want to orchestrate. The goal is often to take raw data, validate it, transform it for analytics, and handle any errors along the way.

This architecture represents a common pattern you'll encounter. Our task today is to design the Step Functions state machine that brings this diagram to life, calling the right service at the right time.

Step 1: Integrating AWS Glue Jobs for Transformation

AWS Glue is your workhorse for large-scale data transformation. A common step in an ETL pipeline is to run a Glue ETL job. In Step Functions, you can do this using a Task state.

Step Functions provides different integration patterns. For services that run longer than a few seconds, like a Glue job, the most important pattern to know is the synchronous or .sync pattern.

When you use the synchronous integration (arn:aws:states:::glue:startJobRun.sync), Step Functions handles the waiting for you. The Task state will only complete (and transition to the next state) after the Glue job has finished, succeeded, or failed. This simplifies your workflow definition significantly, as you don't have to build your own polling logic for the job itself.

Step 2: Using AWS Lambda for Glue Logic and Validation

While Glue handles the heavy data processing, AWS Lambda functions are perfect for the "glue" logic that holds the pipeline together. Given your background in computer science, you can think of these as small, event-driven helper functions that perform specific, lightweight tasks.

In a Step Functions workflow, Lambda is often used to:

• Validate input data: Check if a file schema is correct before starting an expensive Glue job.
• Move files in S3: Archive a raw file or move a failed file to an error bucket.
• Start processes that lack a .sync integration: As we'll see next, some services require you to kick them off and then check their status later.
• Send custom notifications: Send a detailed message to an SNS topic or Slack channel.

The AWS Prescriptive Guidance pattern we saw earlier uses several Lambda functions for these exact purposes, such as validation.py, move_file.py, and start_crawler.py.

Step 3: A Deeper Challenge - Orchestrating Glue Crawlers

Here's a scenario that frequently appears in real-world pipelines and exam questions. After your Glue job transforms data and writes it to S3, you need to run a Glue Crawler to update the Data Catalog so services like Athena can query it.

However, unlike Glue jobs, Glue Crawlers do not have a .sync integration pattern in Step Functions. You can start a crawler, but Step Functions won't automatically wait for it to finish.

So, how do we solve this? We build a polling loop using the state machine components we learned in the last lesson. This is a fantastic practical application of your knowledge.

The pattern looks like this:

1. Task State: Invoke a Lambda function (start_crawler.py) that starts the Glue Crawler.
2. Wait State: Pause the workflow for a short period (e.g., 30 seconds).
3. Task State: Invoke another Lambda function (check_crawler_status.py) that checks the status of the crawler run.
4. Choice State: Inspect the output from the status-checking Lambda.
   • If the status is SUCCEEDED or FAILED, move to the next step in the pipeline.
   • If the status is RUNNING, loop back to the Wait state.

This polling pattern is a powerful technique and a great example of combining multiple state types to handle asynchronous tasks.

Step 4: Designing the Full ETL Workflow

Now we can assemble all the pieces into a complete, multi-step ETL workflow. Let's design a pipeline that performs the following:

1. Run Glue Job 1: Transforms raw data to a "cleansed" format (e.g., CSV to Parquet).
2. Run Crawler 1: Catalogs the cleansed data. This step must wait for the crawler to finish.
3. Run Glue Job 2: Aggregates the cleansed data to create a final "curated" dataset for reporting.
4. Send Notification: A final Lambda function sends a success notification.

Here is what the state machine design would look like conceptually:

1. Start -> TransformRawData (Task)

   • Service Integration: AWS Glue: startJobRun.sync
   • Waits for the first Glue job to complete.

2. TransformRawData -> StartCleansedCrawler (Task)

   • Service Integration: AWS Lambda: invoke
   • Calls our Lambda function to start the crawler.

3. StartCleansedCrawler -> WaitForCrawler (Wait)

   • Waits for a configured duration (e.g., 30 seconds).

4. WaitForCrawler -> GetCrawlerStatus (Task)

   • Service Integration: AWS Lambda: invoke
   • Calls our Lambda to check the crawler's status.

5. GetCrawlerStatus -> IsCrawlerReady? (Choice)

   • Rule 1: If $.Payload.state == 'RUNNING', transition back to WaitForCrawler.
   • Default: Transition to AggregateCleansedData.

6. IsCrawlerReady? -> AggregateCleansedData (Task)

   • Service Integration: AWS Glue: startJobRun.sync
   • Runs the second Glue job.

7. AggregateCleansedData -> SendSuccessNotification (Task)

   • Service Integration: AWS Lambda: invoke
   • Calls a Lambda to send an SNS notification.

8. SendSuccessNotification -> End (Succeed)

This workflow combines synchronous tasks (Glue jobs) with an asynchronous, polled task (the Glue crawler), all orchestrated cleanly within a single state machine.

Conclusion

Today we've moved from knowing the components of Step Functions to designing a functional data pipeline. You've seen how to integrate core data engineering services like AWS Glue and AWS Lambda to build a robust, sequential, and conditional workflow.

Key Takeaways:

• Step Functions orchestrates ETL workflows by defining a sequence of Task states that invoke services like Glue and Lambda.
• For long-running tasks like Glue jobs, use the .sync integration pattern to have Step Functions wait for completion automatically.
• For services without a .sync pattern, like Glue Crawlers, you must design a polling loop using a combination of Task, Wait, and Choice states.
• Lambda functions serve as the flexible "glue" for validation, notification, and initiating or checking on asynchronous tasks.

Next Up:
Our current design represents the "happy path"—what happens when everything succeeds. But production pipelines must be resilient. In our next lesson, we will implement error handling and retry logic within a Step Functions state machine, learning how to catch failures and automate recovery.