Understanding High Cardinality and Its Observability Impact

What is High Cardinality?

First, let's define the term. In data, cardinality refers to the number of unique values in a set.

• Low cardinality: A field with a small, predictable number of unique values. For example, an HTTP status metric might have labels for status_code with values like 200, 404, 500, 503. There are only a few possibilities.
• High cardinality: A field with a very large, often unpredictable number of unique values. Think of fields like user_id, session_id, request_id, or container_id. In a popular application, these can run into the millions or even billions.

To get a solid definition, please start by reading the first part of the following article.

The core issue arises when these high-cardinality attributes are attached as labels or tags to your telemetry data (metrics, logs, traces). Each unique combination of a metric name and its labels creates a unique time series.

Consider this metric: http_requests_total{service="api", region="us-east-1"}. If you add a high-cardinality label like user_id, you get:

• http_requests_total{service="api", region="us-east-1", user_id="123"}
• http_requests_total{service="api", region="us-east-1", user_id="124"}
• ...and millions more...

You've just created millions of unique time series. This problem has become exponentially worse with the shift to modern cloud-native architectures.

This image illustrates the explosion of cardinality when moving from traditional virtual machine (VM) based architectures to dynamic, cloud-native environments. The number of potential unique time series skyrockets from thousands to millions due to the sheer number of ephemeral components like pods and containers.

As you can see, the move from a few hundred VMs to thousands of ephemeral pods multiplies the cardinality of your system, creating a data explosion that many observability tools are not equipped to handle.

The High-Cardinality Trap: Why It's a Major Problem for Your Customers

High cardinality is not just a technical curiosity; it is the source of significant pain for engineering teams and a major driver of cost. Understanding this pain is critical for your sales conversations. When an observability platform cannot handle high cardinality, it leads to a cascade of failures.

Let's summarize these critical business and technical impacts:

1. Massive Cost Increases: Every unique time series needs to be ingested, indexed, and stored. For platforms that price based on data volume or the number of time series, uncontrolled cardinality leads directly to spiraling cloud bills. This is a primary concern for any economic buyer.

*This graph shows how costs can increase as data cardinality grows for different observability solutions. Solutions not architected for high cardinality see costs rise exponentially, making them financially unsustainable at scale.*

2. Cripplingly Slow Performance: When a developer or SRE tries to run a query (e.g., "show me the error rate for all users in Germany"), the system has to sift through millions or billions of time series. This leads to slow dashboards, query timeouts, and an inability to investigate issues during an outage. This directly increases Mean Time to Resolution (MTTR).

3. Forced Compromises & Blind Spots: This is perhaps the most critical impact. To keep their systems from crashing under the load, teams are forced to make painful choices. They intentionally stop collecting high-cardinality data. They drop the user_id, tenant_id, or request_id labels. While this stabilizes the platform, it creates a massive blind spot. When an issue occurs, they can see that there's a problem, but they have no way of knowing who is affected or tracing the problem back to a specific action. They are flying blind.

The Architectural Divide: Why Traditional Tools Fail

These problems aren't solved by simply adding more servers. They are the result of fundamental architectural limitations in older observability tools. Your ability to explain this positions you as a knowledgeable advisor.

Here is a summary of how different architectures handle—or fail to handle—high cardinality. This is a powerful competitive differentiator.

(This table is adapted from the ClickHouse article.)

Given your experience with Grafana and Kibana, you've likely seen tools that sit on top of these different architectures. Grafana often visualizes data from TSDBs like Prometheus, while Kibana is the 'K' in the ELK stack, sitting on Elasticsearch. Understanding these underlying databases is key to understanding their limitations.

Conclusion

Today you've explored one of the most important technical concepts in modern observability. A platform's approach to cardinality is not just a feature; it is a fundamental design choice with massive consequences for cost, performance, and a team's ability to actually solve problems.

Key Takeaways:

• High Cardinality is the explosion of unique identifiers (like user_id, container_id) in modern, distributed systems.
• It is a major source of pain, causing runaway costs, slow performance, and forcing engineering teams to drop the very data they need for debugging.
• Traditional observability architectures (Time-Series Databases and Search Engines) were not designed for high cardinality and force painful compromises.
• A platform's ability to handle high cardinality natively is a crucial technical differentiator that unlocks the true value of observability—asking any question of your data and getting a fast answer.

In our next lesson, we will directly connect this concept to a key business metric by focusing on the learning outcome: Describe the relationship between observability and mean time to resolution (MTTR). You'll see precisely how the ability to query high-cardinality data allows teams to slash the time it takes to find and fix problems.