Hello and welcome to the third module of our course!
In the previous module, we took a deep dive into Amazon Kinesis. We covered how to ingest data using Kinesis Data Streams and how to build resilient, serverless processing applications with AWS Lambda, including critical error handling using Dead-Letter Queues. You now have a solid foundation in AWS's native streaming service.
This lesson kicks off our new module on Managed Streaming with Apache Kafka (MSK). We'll begin by addressing a fundamental question for any data engineer on AWS: when should you use Kinesis, and when should you opt for MSK? Our learning outcome is to compare Amazon MSK with Kinesis for streaming data ingestion scenarios. This comparison is not just academic; it's at the heart of many scenario-based questions on the certification exam.
1. The Streaming Landscape on AWS
Before we compare, let's place these services within the broader context of a streaming data architecture. A typical pipeline involves ingestion, storage, processing, and a destination. Both Kinesis and MSK primarily serve as the durable, scalable stream storage layer.
The core choice you'll face is between an AWS-native, fully managed service (Kinesis) and a managed open-source platform (MSK).
2. The Fundamental Difference: Philosophy and Architecture
At a high level, the choice between Kinesis and MSK comes down to a trade-off between simplicity/integration and power/flexibility.
- Amazon Kinesis Data Streams is an AWS-native, fully managed, and opinionated service. AWS handles almost all the operational complexity. You provision capacity in terms of shards and interact with the service using AWS SDKs.
- Amazon MSK is a managed service for running Apache Kafka. It gives you the power of the vast Kafka ecosystem and its open-source APIs, while AWS manages the underlying infrastructure (like patching servers and replacing failed brokers). You work with topics, partitions, and brokers.
The following image provides a great visual summary of their differing architectures.
3. A Detailed Comparison Across Key Dimensions
To make an informed decision, we need to compare the services across several critical dimensions. The following video provides an excellent, comprehensive overview of these differences.
Find the Right Data Streaming Architecture for Your Data Needs - AWS Online Tech Talks
This clip from the AWS Online Tech Talk 'Find the Right Data Streaming Architecture for Your Data Needs' provides a detailed comparison of Kinesis (Data Streams and Firehose) and MSK (Provisioned and Serverless) across key dimensions like provisioning, scalability, retention, and latency.
Watch the segment from 13:27 to 24:03. As you watch, focus on the differences between the services in these areas: Provisioning Model: How do you provision capacity? (Shards vs. Brokers) Scalability: How does the service scale to handle more traffic? Data Retention: What are the limits on how long data can be stored? Latency: What are the typical latencies for data processing?
Let's consolidate the key points from the video and other resources into a comparative table. This is the kind of information that is essential for the exam.
| Factor | Amazon Kinesis Data Streams | Amazon MSK (Provisioned & Serverless) |
|---|---|---|
| Management Model | Fully managed ("serverless" feel). Minimal operational overhead. | Managed open-source. Provisioned requires broker sizing and scaling. Serverless abstracts this away. |
| API & Ecosystem | Proprietary AWS APIs and SDKs. Tight integration with other AWS services (Lambda, Firehose). | Standard Apache Kafka APIs. Unlocks a vast open-source ecosystem (Kafka Connect, Debezium, ksqlDB, etc.). |
| Data Organization | Stream composed of one or more shards. | Topic composed of one or more partitions, distributed across brokers. |
| Data Retention | Up to 365 days. | Provisioned: Effectively unlimited with Tiered Storage. Serverless: Up to 365 days. Also supports Log Compaction. |
| Scaling | Change shard count (manual or API-driven). On-Demand mode provides automatic scaling based on throughput. | Provisioned: Add brokers or storage (manual). Serverless: Fully automatic scaling. |
| Delivery Guarantees | At-least-once delivery is standard. | Can support exactly-once semantics (with appropriate client and broker configuration). |
| Fan-Out (Consumers) | Limited number of standard consumers per shard. Enhanced Fan-Out (EFO) provides dedicated throughput for each consumer at extra cost. | Natively supports a large number of independent consumer groups, each reading from the topic at its own pace. |
4. The Cost Dimension
Cost is a critical and complex factor. The pricing models are fundamentally different, making direct comparisons tricky.
- Kinesis Data Streams (Provisioned): You pay per shard-hour and per 1 million PUT payload units (25KB chunks). This is predictable for stable workloads.
- Kinesis Data Streams (On-Demand): You pay per GB of data written and read, plus a per-stream hourly charge. This is better for variable or unknown workloads.
- MSK Provisioned: You pay per broker-hour (based on EC2 instance type), for the provisioned storage (EBS), and for data transfer between brokers in different availability zones.
- MSK Serverless: You pay per GB of data ingested and delivered, and per GB-hour of data stored.
The following video provides an excellent, worked-out cost analysis that illustrates the trade-offs.
The video 'Kinesis versus MSK' from cloudonaut offers a clear cost comparison using a specific throughput scenario. This will help you understand the economic implications of choosing one over the other.
Watch the segment from 6:42 to 14:06. You don't need to memorize the exact numbers, but focus on understanding the conclusion: Kinesis is generally more cost-effective for smaller workloads due to MSK's fixed broker costs, while MSK Provisioned can become more economical at very high, sustained throughput levels.
5. Making the Right Choice: A Decision Framework
For the exam, you'll need to analyze a scenario and select the most appropriate service. Here’s a decision framework to guide you.
Amazon Kinesis vs Amazon MSK: The Complete Guide for ...
The article 'Amazon Kinesis vs Amazon MSK: The Complete Guide' offers a fantastic decision-making framework. The 'Enhanced Decision Matrix' and 'Decision Flow Chart' are particularly useful for exam preparation.
Read the section titled 'Part 8: Making the Right Choice'. Pay close attention to the lists of questions under 'Choose Kinesis if...', 'Choose MSK Serverless if...', and 'Choose MSK Provisioned if...'. This will help you internalize the decision criteria.
Here is a summary of the key questions to ask yourself when evaluating a scenario:
Choose Amazon Kinesis if:
- You want the simplest solution with the lowest operational overhead.
- Your architecture is primarily built on other AWS services like Lambda, S3, and Redshift.
- You don't have existing Kafka expertise or applications.
- You don't need Kafka-specific features like log compaction or connectors to non-AWS systems.
Choose Amazon MSK if:
- You are migrating an existing Apache Kafka workload to AWS.
- You need to leverage the rich Kafka ecosystem (e.g., Kafka Connect for Change Data Capture with Debezium, Kafka Streams for complex stream processing).
- You have a hybrid-cloud strategy and want a consistent streaming platform on-premises and in AWS.
- You require data retention longer than 365 days (using MSK Provisioned with Tiered Storage).
- You need fine-grained control over broker configuration or require features like exactly-once semantics.
Test your understanding!
A financial services company is building a new real-time fraud detection system on AWS. Their existing on-premises systems use Apache Kafka extensively for event sourcing, and their development team has deep Kafka expertise. The new system must be able to replay events from the last 2 years for auditing and model retraining. The workload is expected to be high and relatively consistent.
Which AWS streaming service should they choose and why?
Show answer
They should choose Amazon MSK Provisioned.
Reasoning:
- Existing Expertise & Ecosystem: The team already has deep Kafka expertise, which eliminates the learning curve and allows them to leverage their existing tools and knowledge.
- Long-Term Retention: The requirement to store and replay data from the last 2 years exceeds the 365-day limit of Kinesis. MSK Provisioned with Tiered Storage can support this "unlimited" retention requirement cost-effectively.
- Consistent High Throughput: For high, sustained workloads, MSK Provisioned can be more cost-effective than usage-based models once you cross a certain threshold, as the fixed broker costs are amortized over a large volume of data.
Conclusion
You now have a clear framework for comparing Amazon Kinesis and Amazon MSK. While both are powerful services for stream storage, they serve different needs and philosophies.
Key Takeaways:
- Kinesis is the go-to for simplicity, tight AWS integration, and minimal operational burden. It's the "AWS-native" path.
- MSK is your choice when you need the power, flexibility, and rich ecosystem of Apache Kafka. It's the "managed open-source" path.
- The decision hinges on factors like existing team expertise, required ecosystem integrations, data retention policies, and cost profile at scale.
- MSK Serverless is an excellent middle-ground, offering the Kafka API with the operational simplicity closer to Kinesis On-Demand, making it ideal for new projects or variable workloads.
In our next lesson, we'll move from "why" to "how." You will learn to configure an MSK cluster with appropriate sizing, broker specifications, and storage, taking the first practical step into the world of managed Kafka on AWS.