Serverless Stream Processing with Kinesis and Lambda

1. The Kinesis-Lambda Integration: Event Source Mapping

The component that connects a Kinesis Data Stream to an AWS Lambda function is called an Event Source Mapping (ESM). You don't interact with the stream directly from your Lambda in the same way you would with an SDK; instead, the ESM acts as a managed poller and event broker.

The ESM is responsible for:

• Polling each shard in your Kinesis stream for new records.
• Batching these records together. You can control the batch size and a batching window.
• Invoking your Lambda function synchronously with a batch of records.
• Checkpointing, which means it automatically keeps track of the last successfully processed record in each shard. This relieves you of the complex task of state management.
• Handling retries when your function fails.

This serverless integration simplifies the consumer side of your streaming architecture significantly.

2. Building a Kinesis-Processing Lambda Function

Let's look at the practical aspects of creating a Lambda function that processes Kinesis records.

When the Event Source Mapping invokes your function, it passes an event object that contains a list of records. Your function code needs to iterate through this list. A crucial detail is that the actual data payload within each Kinesis record is Base64 encoded. Your code must decode this data before you can work with it.

The video you just watched continues with a full walkthrough of creating the necessary components: the Kinesis stream, a DynamoDB table to store the results, and the Python Lambda function itself.

This simple setup works perfectly as long as every record can be processed successfully. But what happens when things go wrong?

3. The Challenge: Blocking Retries and "Poison Pills"

The default error handling behavior of the Kinesis-Lambda integration is robust but can cause significant problems if not configured correctly.

When your Lambda function fails to process a batch (by timing out or throwing an unhandled exception), the Event Source Mapping treats the entire batch as a failure. It will then retry invoking the function with the exact same batch of records. This retry process will continue until the function succeeds or until the records in the batch expire from the stream (the default data retention is 24 hours).

This creates a blocking scenario. A single malformed record—a "poison pill"—can cause the entire batch to fail repeatedly, preventing the processor from moving on to new records in that shard. This can halt your real-time data processing for hours.

The following video explains this critical behavior and introduces the modern solutions.

4. Configuring Resilient Error Handling

To prevent a single bad record from blocking your entire pipeline, you must configure the error handling settings on the Event Source Mapping. These settings give you fine-grained control over the retry behavior and provide a way to safely isolate failed records.

Key Configuration Options:

• On-Failure Destination (DLQ): This is the most important setting. You can configure an Amazon SQS queue or an Amazon SNS topic as a destination for batches that have failed processing. After exhausting all retries, the ESM sends a message containing details about the failed batch to this destination, allowing the stream to continue processing new records.
• Maximum Retry Attempts: Instead of retrying indefinitely, you can specify a finite number of retries (e.g., 3). After the last attempt fails, the batch is sent to the on-failure destination if one is configured.
• Bisect Batch on Function Error: When a batch fails, this setting instructs the ESM to split the batch in half and retry each half separately. This process repeats, quickly isolating the specific "poison pill" record that is causing the failure. This is highly recommended.
• Maximum Record Age in Seconds: You can configure the ESM to discard records that have been in the stream for too long, sending them to the DLQ if configured.

The following AWS documentation provides the definitive guide to these settings.

It's important to remember that for stream-based sources like Kinesis and DynamoDB Streams, these error handling settings are on the Event Source Mapping, not on the Lambda function's general configuration (which applies to asynchronous sources like S3 or SNS).

5. What's in the DLQ Message?

When a batch is sent to an SQS or SNS on-failure destination, the message does not contain the original data payloads. Instead, it contains metadata about the failed batch, including:

• streamArn
• shardId
• startSequenceNumber and endSequenceNumber
• The error message (functionError)

You must use this metadata to build a separate process (e.g., another Lambda function) that can fetch the original records from the Kinesis stream using the GetRecords API before they expire. This is a critical point for the exam. If you configure S3 as a destination, Lambda will send the full payload.

The AWS documentation you read earlier also provides examples of these payloads. Take a moment to review them.

Conclusion

In this lesson, you've learned how to build a robust, serverless pipeline to process real-time data from Kinesis Data Streams using AWS Lambda.

Key Takeaways:

• The Event Source Mapping (ESM) is the managed component that connects Kinesis to Lambda, handling polling, batching, and checkpointing.
• By default, a function failure causes a blocking retry of the entire batch, which can be halted by a single "poison pill" record.
• To build a resilient pipeline, you must configure the ESM's error handling with an On-Failure Destination (DLQ), a limited number of Maximum Retry Attempts, and ideally enable Bisect Batch on Function Error.
• When using SQS or SNS as a DLQ, the message contains only metadata. You must write a separate process to retrieve the original records from the stream before they expire.

You now have a comprehensive understanding of both ingesting data into Kinesis and processing it in a scalable and resilient way.

In our next lesson, we will begin a new module on Apache Kafka. We will compare Amazon MSK with Kinesis for streaming data ingestion scenarios, helping you understand when to choose one powerful streaming service over the other.