Metrics and Logging for Large-Scale Distributed Systems

The Twin Crises: Cardinality and Cost

At the heart of any metrics strategy lies a fundamental tension. We want to attach as much context as possible to our measurements. For example, knowing the total number of errors is useful, but knowing the number of errors per customer, per service version, and per datacenter is far more powerful for debugging. Each piece of unique context we add (a customer ID, a version hash, a location) is a new dimension, and the total number of unique combinations of these dimensions is the cardinality.

As cardinality explodes, so does the number of unique time series you must store, index, and query. This leads directly to an explosion in cost—for memory, storage, CPU, and network. This problem is so central that it shapes the entire architecture of modern monitoring systems.

To get a sense of the current industry thinking on this challenge, the following talk by Charity Majors, a prominent voice in the observability space, frames the problem as a generational shift.

Majors argues that the traditional approach—pre-aggregating data into low-cardinality metrics—discards the very context needed for deep debugging. The alternative she proposes is to treat wide, structured log events as the source of truth. This is a powerful idea, but it comes with its own set of engineering challenges. How do you build a system that can handle this firehose of raw data? Google's answer is Monarch.

Case Study: Google's Monarch

Monarch is Google's planet-scale, in-memory time series database, built to replace its predecessor, Borgmon. The Monarch paper is a masterclass in designing for scale, and it directly addresses the cardinality and cost problems we've discussed.

First, let's understand Monarch's data model. It isn't just a key-value store for metrics; it's a schematized, relational model.

The key insight here is schematization. A time series in Monarch is not an arbitrary string; it is a point in a well-defined table formed by joining a target schema and a metric schema.

This structured approach allows Monarch to validate, optimize, and efficiently route queries. But how does it handle the raw volume and cardinality? The answer lies in its collection architecture.

Taming Cardinality with Collection Aggregation

Monarch's most powerful weapon against cardinality is collection aggregation. For extremely high-cardinality data sources, it's simply not feasible to store a unique time series for every possible combination of labels. The paper gives the example of disk I/O, where you might want to track usage per user across millions of disk servers. This would generate tens of billions of time series.

Instead of storing this raw data, Monarch aggregates it during ingestion.

This technique is a critical design pattern. Instead of every disk server writing a cumulative value for "IOPs for user X," it writes a delta: "IOPs for user X in the last 10 seconds." Routers and leaves can then efficiently sum these deltas into a single, lower-cardinality time series (e.g., sum(IOPs) by user), drastically reducing the storage and query cost. As the paper notes, this is not just a theoretical optimization; on average, Monarch aggregates 36 input time series into one, and in extreme cases, over a million to one.

The video below gives a great walkthrough of the architecture and lessons learned, including the importance of this pre-aggregation.

Designing a Holistic Strategy

Monarch provides a powerful blueprint for a metrics-focused system. However, as a Principal Engineer, you need to design a comprehensive strategy for both metrics and logs. We can synthesize the lessons from Monarch with the broader industry trends and practical cost management techniques.

A robust strategy can be broken down into a few key design pillars. The Google Cloud blog post, "Cloud Logging cost management best practices," though focused on a specific product, provides an excellent general-purpose framework.

Let's integrate these ideas into a design framework.

1. Define What to Collect (and What Not To)

• Metrics: Use a Monarch-style system for your core operational health signals (the "golden signals": latency, traffic, errors, saturation). These are typically high-volume but have predictable, relatively low cardinality.
  • Action: Define strict schemas for your core metrics.
  • Action: Aggressively use collection aggregation for any metric where cardinality threatens to explode (e.g., per-user statistics, per-request-ID metrics).
• Structured Logs: Reserve detailed, high-cardinality logging for two primary use cases:
  1. Critical business events: An order_placed event that contains user_id, cart_id, item_skus, total_value, etc.
  2. Debug/Trace events: One wide event per service per request, containing all relevant local context, as advocated by Charity Majors. This is invaluable for debugging unknown-unknowns.
• Action: Implement exclusion rules to drop noisy, low-value logs at the source (e.g., verbose health checks, successful 2xx load balancer logs). Consider sampling for logs where volume is high but individual events are less critical.

2. Configure the Lifecycle: Retention and Downsampling

Not all data has the same value over time.

• Hot Tier: High-resolution metrics and logs needed for real-time alerting and debugging might only need to be retained for a few hours or days.
• Warm/Cold Tier: You can downsample older data to save costs. For example, metrics might go from 10-second resolution to 1-minute resolution after 24 hours, and 10-minute resolution after 30 days. Logs can be moved from fast, expensive storage to cheaper, slower object storage (like Google Cloud Storage or Amazon S3) for long-term compliance or historical analysis.
• Action: Define tiered retention policies for different classes of data. Monarch's configuration system allows users to do this explicitly.

3. Instrument for Correlation

A metric spike is a signal; a log line is context. The real power comes from linking them.

• Action: Ensure that your application logs include identifiers that can correlate them to metrics. For instance, if a metric has a service and version label, your logs should contain the same information.
• Action: Propagate trace and span IDs through your entire stack—RPCs, logs, and even as exemplars in your metrics. Monarch's ability to attach a Dapper trace ID as an exemplar to a high-latency distribution is a prime example of this. When you see a latency spike, you can immediately jump to the exact trace that caused it.

4. Monitor the Monitors

Finally, your observability platform is itself a critical distributed system.

• Action: Apply the same principles to monitoring your monitoring system. Track ingestion volume, query latency, and cardinality of your metrics.
• Action: Set up billing and usage alerts to catch runaway costs before they become a major problem. A bug that accidentally adds a high-cardinality label like a timestamp to a metric can cause costs to skyrocket.

Conclusion

Designing a metrics and logging strategy for a large-scale system is a constant balancing act between visibility, cost, and performance. There is no single "right" answer, only a series of well-reasoned trade-offs.

Here are our key takeaways:

• Cardinality is the core challenge: The number of unique time series you generate directly dictates the cost and complexity of your monitoring system.
• Monarch's playbook is powerful: Techniques like schematization to structure data, and especially collection aggregation to reduce cardinality during ingestion, are essential patterns for building scalable metrics platforms.
• Structured logs are the future of debugging: While metrics are excellent for summarizing known system behaviors, high-cardinality structured logs are unparalleled for exploring unknown issues and preserving critical context.
• A hybrid strategy is pragmatic: A successful design will likely use a metrics-first approach for well-understood operational signals and a structured-logging-first approach for deep debugging and business analytics, with strong correlation between them.
• Strategy is an active process: You must continuously analyze usage, optimize collection and retention policies, and monitor costs to keep the system effective and efficient.

In our next lesson, we will build directly upon this foundation. Now that we have a strategy for collecting high-quality observability data, we will ask: what do we do with it? We will explore how to translate raw data into formal guarantees of service health by addressing the learning outcome: Define SLOs, SLIs, and error budgets and explain their role in data-driven reliability management.