Optimizing Kinesis Shard Capacity

1. The Shard: The Building Block of a Kinesis Data Stream

To understand capacity, you must first understand the shard. A shard is the fundamental unit of throughput in a Kinesis Data Stream. Think of it as a single, ordered lane on a data highway. The total capacity of your stream (the number of lanes on the highway) is determined by the number of shards you configure.

Each shard has a fixed capacity:

• Write Capacity (Ingress): Up to 1 MB per second OR 1,000 records per second.
• Read Capacity (Egress): Up to 2 MB per second. This read capacity is shared among all standard consumers reading from that shard.

When a producer application sends a data record to Kinesis, it must include a partition key. Kinesis uses an MD5 hash of this partition key to determine which specific shard the record will be written to. This mechanism is crucial because it dictates how data is distributed across your available shards. We'll revisit this in a future lesson on performance, but for now, remember that an even data distribution is key to utilizing all your provisioned capacity.

2. Choosing Your Capacity Mode: Provisioned vs. On-Demand

AWS offers two primary capacity modes for Kinesis Data Streams. Choosing the right one depends entirely on the nature of your workload.

• Provisioned Mode: You manually specify and manage the number of shards for your stream. You are billed per shard-hour. This mode is best for predictable and stable traffic patterns where you can accurately forecast your capacity needs.
• On-Demand Mode: AWS automatically manages the shards for you, scaling capacity up and down based on your traffic. You are billed based on the volume of data written and read. This mode is ideal for unpredictable, spiky, or new workloads where operational ease outweighs the need for fine-grained cost control.

As the video illustrates, choosing Provisioned mode for a highly variable workload can lead to significant waste, as you pay for peak capacity even during idle periods. Conversely, On-demand mode perfectly handles spiky traffic but might be more expensive than a well-tuned Provisioned stream for a very stable workload.

For the certification exam, you'll need to analyze scenarios and decide which mode is more appropriate from a cost and performance perspective.

3. Calculating Shard Requirements in Provisioned Mode

When you choose Provisioned mode, you must determine the initial number of shards. This is a common calculation you might face in the exam. To do this, you need to estimate your workload based on three variables:

1. Average Record Size (in KB): The average size of a single data record you'll be sending.
2. Records per Second: The maximum number of records you expect to write to the stream per second.
3. Number of Consumers: The number of separate applications that will be reading from the stream.

With these inputs, you can calculate the required number of shards.

The Formula Summarized

As you saw in the video, the calculation involves determining the shards needed for both writing (ingress) and reading (egress) and taking the maximum of the two. Let's formalize the formulas presented in the video and found in the AWS documentation.

1. Calculate Incoming Write Bandwidth (KB/s):

   $$\text{IncomingWriteBandwidth}=\text{AverageRecordSizeInKB}\times \text{RecordsPerSecond}$$

2. Calculate Outgoing Read Bandwidth (KB/s):

   $$\text{OutgoingReadBandwidth}=\text{IncomingWriteBandwidth}\times \text{NumberOfConsumers}$$

3. Calculate Number of Shards:
   A single shard provides 1024 KB/s of write capacity and 2048 KB/s of read capacity.

   $$\text{ShardsForWrite}=\frac{\text{IncomingWriteBandwidth}}{1024}$$ $$\text{ShardsForRead}=\frac{\text{OutgoingReadBandwidth}}{2048}$$

   The total number of shards is the maximum of these two values, rounded up to the nearest whole number.

   $$\text{NumberOfShards}=\lceil \max (\text{ShardsForWrite},\text{ShardsForRead})\rceil$$

You can also read more about this in the official AWS documentation.

4. Adjusting Capacity: Scaling a Provisioned Stream

In Provisioned mode, your work isn't done after the initial setup. If your traffic patterns change, you may need to scale your stream. This is known as resharding. Kinesis provides APIs to adjust the shard count of an active stream without downtime.

• Splitting (SplitShard): You can split an existing shard into two new shards. This increases the stream's total capacity and is useful when traffic grows. It's also used to isolate a "hot shard"—a shard that is receiving a disproportionate amount of data due to a popular partition key.
• Merging (MergeShards): You can merge two adjacent shards into a single shard. This decreases the stream's capacity and is useful for reducing costs when traffic decreases.

For more straightforward scaling, the UpdateShardCount API allows you to scale to a target number of shards, and Kinesis will perform the necessary splits or merges in the background.

Conclusion

You now have a comprehensive understanding of how to configure capacity for Kinesis Data Streams, a core skill for an AWS Data Engineer.

Key Takeaways:

• Shards are the base unit of capacity in KDS, each with a fixed write limit (1 MB/s or 1000 records/s) and read limit (2 MB/s).
• Provisioned Mode offers fine-grained control and is best for predictable workloads. You are responsible for calculating and managing the shard count.
• On-Demand Mode provides automatic scaling and is ideal for unpredictable or new workloads, simplifying operations.
• The number of shards in Provisioned mode is determined by calculating the capacity needed for both write throughput and read fan-out and choosing the greater of the two.

In our next lesson, we will shift our focus from configuring the stream to interacting with it. Now that you know how to build the data highway, it's time to learn how to drive on it. We will cover how to implement producer and consumer applications for Kinesis Data Streams using the AWS SDK and KCL.