CloudWatch Metrics for MSK Cluster Monitoring

Hello! Welcome back.

In our last lesson, we focused on ensuring data integrity and evolvability within your MSK data streams by using the AWS Glue Schema Registry. We saw how it acts as a gatekeeper to prevent malformed data from disrupting your pipelines. Now, we'll shift our focus from the data itself to the health and performance of the infrastructure carrying it.

Today's lesson is all about how to monitor MSK cluster performance and health using CloudWatch metrics. A well-monitored cluster is a reliable one. For the Data Engineer certification, you'll be expected to know not just how to build data systems, but also how to operate, troubleshoot, and optimize them. This lesson provides the foundational knowledge for those critical skills.

1. MSK Monitoring Levels and Integration with CloudWatch

Amazon MSK is deeply integrated with Amazon CloudWatch, automatically pushing a wide range of metrics that give you visibility into the state of your cluster. However, not all metrics are enabled by default. MSK offers several monitoring levels, which allow you to balance the granularity of insights against monitoring costs.

This is a key concept to understand, as choosing the right level is a common operational decision.

Amazon MSK metrics for monitoring Standard brokers with CloudWatch

To begin, let's explore the different monitoring levels available. The official AWS documentation provides a comprehensive list of metrics for each level.

First, read the introductory section to understand the four monitoring levels: DEFAULT, PER_BROKER, PER_TOPIC_PER_BROKER, and PER_TOPIC_PER_PARTITION. Then, quickly scan the tables for each level. You don't need to memorize every metric. Instead, focus on understanding the type of information that becomes available as you increase the monitoring level.

As you've just seen, the levels provide progressively more detail:

DEFAULT: This is the free, base level of monitoring. It provides essential cluster-wide and broker-level metrics, including CPU, disk, memory, and high-level consumer lag. For many use cases, this is sufficient for general health monitoring.
PER_BROKER: This level incurs a cost and adds more detailed metrics for each broker. It's useful for diagnosing performance issues on a specific broker, such as network bottlenecks or request processing latency.
PER_TOPIC_PER_BROKER: This level provides metrics broken down by topic on each broker. It's useful when you suspect a specific "noisy neighbor" topic is impacting a broker's performance.
PER_TOPIC_PER_PARTITION: This is the most granular (and most expensive) level. It provides metrics like consumer lag for each individual partition, which is invaluable for fine-grained troubleshooting of consumer performance.

For the exam, and for general operations, you should be very familiar with the most critical metrics from the DEFAULT level. Here are some of the most important ones:

CPU: CpuUser, CpuSystem, CpuIdle — Measures the CPU load on your brokers.
Disk: KafkaDataLogsDiskUsed — The percentage of disk space used for your Kafka data logs. Running out of disk is a critical failure mode.
Memory: HeapMemoryAfterGC — The percentage of JVM heap memory in use after a garbage collection cycle. This is a key indicator of memory pressure.
Cluster Health: ActiveControllerCount (should be 1), OfflinePartitionsCount, UnderReplicatedPartitions — These metrics give you a quick check on the overall health and availability of the cluster and its data.
Consumer Health: MaxOffsetLag — This is one of the most important metrics. It tells you the largest lag (in number of messages) for any consumer group on a topic. A consistently increasing lag means your consumers can't keep up with your producers.

2. Best Practices: Key Metrics to Alarm On

Knowing which metrics exist is the first step. The second, more crucial step is knowing what "good" and "bad" look like. AWS provides specific best practices for monitoring key resources. Let's turn the metrics we've just learned about into actionable monitoring strategies.

Best practices for Standard brokers

The 'Best practices for Standard brokers' documentation provides clear, actionable thresholds for some of the most important MSK metrics. This is exactly the kind of information that is valuable for both real-world operations and certification questions.

Please read the sections titled 'Monitor CPU usage', 'Monitor disk space', and 'Monitor Apache Kafka memory'. Focus on the specific metric names and the recommended alarm thresholds (e.g., CPU < 60%, Disk < 85%, Memory < 60%). Also, note the recommended actions to take when an alarm is triggered.

Let's summarize these critical best practices:

CPU Utilization

Metric: The sum of CpuUser and CpuSystem. You can create this composite metric in CloudWatch using Metric Math.
Threshold: Keep utilization below 60%.
Why?: This headroom is vital for handling operational events. When a broker fails or is taken offline for patching, its workload (partition leadership) is shifted to other brokers in the cluster. Without sufficient CPU headroom, the remaining brokers can become overloaded, leading to a cascading failure.
Action: If the alarm triggers, you can scale up (use larger broker instances) or scale out (add more brokers and rebalance partitions).

Disk Space

Metric: KafkaDataLogsDiskUsed.
Threshold: Alarm when usage exceeds 85%.
Why?: If a broker's disk fills up, it can no longer accept writes for the partitions it leads, causing producers to fail. This is a critical failure.
Action: Enable auto-scaling for storage, manually increase broker storage, or reduce the data retention period for your topics.

Memory Usage

Metric: HeapMemoryAfterGC (Percentage of heap memory used after garbage collection).
Threshold: Alarm when usage exceeds 60%.
Why?: High memory usage after garbage collection indicates that the JVM is under constant pressure, which can lead to long garbage collection pauses, increasing latency and potentially causing the broker to become unresponsive.
Action: Actions are application-specific. The document gives an example of reducing the transactional.id.expiration.ms setting if you use Kafka transactions. Another common approach is to scale up to instances with more memory.

Test your understanding!

You are the data engineer for a growing e-commerce platform. Your team notices that during peak shopping events, producer latency to your MSK cluster increases significantly. A new CloudWatch alarm you configured for (CpuUser + CpuSystem) > 60% on your brokers has started firing during these peaks. However, your KafkaDataLogsDiskUsed and HeapMemoryAfterGC metrics remain well within safe limits.

Based on this information, what is the most appropriate first step to resolve the issue?

Show answer

The most appropriate first step is to scale the cluster's compute capacity. Since the bottleneck is clearly CPU, and not disk or memory, increasing the CPU resources is the direct solution. Following the best practices, the recommended approach would be to update the broker type to the next larger size (scaling up, e.g., from kafka.m5.large to kafka.m5.xlarge). This provides the needed CPU headroom to handle the increased load during peak events.

3. Visualizing Health with CloudWatch Dashboards

While alarms are essential for notifying you of problems, CloudWatch Dashboards are invaluable for at-a-glance monitoring, correlation of metrics, and historical analysis. A well-designed dashboard can immediately tell you the story of your cluster's health.

The image below shows an example of a comprehensive Kafka monitoring dashboard in CloudWatch.

AWS CloudWatch Dashboard for Apache Kafka Cluster Monitoring — This CloudWatch dashboard provides a consolidated view of an MSK cluster's health. It's organized into logical sections: a **Cluster overview** showing critical state metrics like `UnderReplicatedPartitions` and `OfflinePartitionsCount`; **Producer** and **Consumer** overviews showing latency and `Consumer Lag`; and a **Brokers** section. This allows an operator to quickly correlate events, for instance, seeing if a spike in producer latency corresponds with high CPU on a broker or an increase in consumer lag.

This dashboard brings together many of the metrics we've discussed. You can see how Consumer Lag is prominently displayed, as it directly impacts data freshness for your downstream applications. You can also see cluster-level health indicators like OfflinePartitionsCount. Building a dashboard like this is a standard practice for operating any production MSK cluster.

Conclusion

This lesson completes our module on Amazon MSK. You have learned how to provision, secure, integrate, govern, and now monitor a managed Kafka cluster. Mastering monitoring is non-negotiable for a data engineer, as it's the key to maintaining the reliability and performance of your streaming pipelines.

Key Takeaways:

Monitoring Levels: MSK provides tiered monitoring levels (DEFAULT, PER_BROKER, etc.) allowing you to trade cost for granularity.
The Big Three: For cluster health, always monitor and set alarms on CPU Utilization (< 60%), Disk Usage (KafkaDataLogsDiskUsed < 85%), and JVM Memory (HeapMemoryAfterGC < 60%).
Consumer Lag is King: The MaxOffsetLag metric is your primary indicator of consumer health. A rising lag is an early warning that your downstream processing is falling behind.
Dashboards for Visibility: Use CloudWatch Dashboards to create a single pane of glass for your cluster's key metrics, enabling rapid troubleshooting and analysis.

Next Up:
We will now move from data ingestion and streaming to data transformation and cataloging. The next module begins our deep dive into AWS Glue, the serverless ETL and data catalog service. Our first lesson will focus on its central component: how to "Configure the AWS Glue Data Catalog as a central metadata repository for a data lake." This is the foundation upon which scalable ETL and data lake governance are built.

Recommended resources