Micrometer & Prometheus: Custom Application Metrics

1. The Metrics Stack: Micrometer & Prometheus

Before writing code, let's understand the tools. In a modern Spring Boot ecosystem, collecting metrics involves a few key components that work together.

• Micrometer: Think of Micrometer as the SLF4J for metrics. It's a facade or vendor-neutral API that lets you instrument your code without tying it to a specific monitoring system. You write your code against the Micrometer API, and then you can plug in a backend like Prometheus, Datadog, or New Relic by just adding a dependency.
• Prometheus: This is a powerful, open-source monitoring and alerting toolkit. Its core feature is a time-series database. It operates on a pull model, meaning the Prometheus server periodically sends an HTTP request to your application's metrics endpoint to collect the latest data.

2. Setting Up Your Application

Let's get your Spring Boot application ready to produce metrics for Prometheus. This involves adding dependencies and exposing the correct Actuator endpoint.

Based on the article, here are the steps:

1. Add Dependencies to pom.xml:
   You likely already have spring-boot-starter-actuator from the last lesson. The key new dependency is micrometer-registry-prometheus.

   <!-- Already present for observability -->
   <dependency>
       <groupId>org.springframework.boot</groupId>
       <artifactId>spring-boot-starter-actuator</artifactId>
   </dependency>
   
   <!-- This dependency tells Micrometer how to format metrics for Prometheus -->
   <dependency>
       <groupId>io.micrometer</groupId>
       <artifactId>micrometer-registry-prometheus</artifactId>
   </dependency>
   

   With these dependencies, Spring Boot auto-configuration will create a PrometheusMeterRegistry bean for you.

2. Expose the Prometheus Endpoint in application.properties:
   By default, Spring Boot doesn't expose all Actuator endpoints over HTTP. You need to explicitly include prometheus.

   # Expose health, info, metrics, and the prometheus-specific endpoint
   management.endpoints.web.exposure.include=health,info,metrics,prometheus
   

After adding these, restart your application. You can now access http://localhost:8080/actuator/prometheus. You'll see a lot of text output—these are the default JVM, system, and web server metrics that Spring Boot provides out of the box, all formatted for Prometheus to scrape.

3. Implementing Custom Application Metrics

Default metrics are useful, but the real power comes from creating metrics that are specific to your business logic. For example, how many orders have been processed? How long does it take to generate a report?

Micrometer's central interface for this is the MeterRegistry. You simply inject it into any Spring bean and use it to create and update meters. Let's focus on the two most common types: Counter and Timer.

Counter

A Counter is a metric that only ever goes up. It's perfect for counting occurrences of an event.

Let's imagine you have a service that processes incoming messages. You want to count how many messages are processed successfully versus how many fail.

@Service
public class MessageProcessingService {

    private final Counter messagesProcessedSuccess;
    private final Counter messagesProcessedFailed;

    public MessageProcessingService(MeterRegistry meterRegistry) {
        // Create a counter for successful messages
        this.messagesProcessedSuccess = Counter.builder("myapp.messages.processed")
                .tag("status", "success")
                .description("The number of successfully processed messages")
                .register(meterRegistry);

        // Create a counter for failed messages
        this.messagesProcessedFailed = Counter.builder("myapp.messages.processed")
                .tag("status", "failure")
                .description("The number of failed messages")
                .register(meterRegistry);
    }

    public void processMessage(Message message) {
        try {
            // Your message processing logic here...
            System.out.println("Processing message: " + message.getId());
            // Simulate some failures
            if (message.getId() % 3 == 0) {
                throw new RuntimeException("Simulated processing error");
            }
            // If successful, increment the success counter
            messagesProcessedSuccess.increment();
        } catch (Exception e) {
            // If an error occurs, increment the failure counter
            messagesProcessedFailed.increment();
        }
    }
}

Key Concepts:

• MeterRegistry: Injected via the constructor.
• Builder Pattern: Counter.builder(...) provides a fluent API to define the metric.
• Naming Convention: Micrometer uses lowercase.dot.notation. This will be automatically converted to snake_case for Prometheus (e.g., myapp_messages_processed_total).
• Tags (Labels): This is a crucial concept. Instead of creating two metrics (messages.success and messages.failed), we create one metric (myapp.messages.processed) and use a status tag to differentiate. This makes querying much more powerful in Prometheus (e.g., you can easily get the total count by summing across all statuses).

Timer

A Timer measures both the frequency and duration of events. It's ideal for tracking the latency of method executions or API calls. A Timer automatically creates a counter for you, so you don't need a separate Counter if you're already timing an event.

Let's time a method that performs a potentially slow operation.

@Service
public class ReportGenerationService {

    private final Timer reportGenerationTimer;

    public ReportGenerationService(MeterRegistry meterRegistry) {
        this.reportGenerationTimer = Timer.builder("myapp.report.generation")
                .description("Time taken to generate a user report")
                .tag("type", "financial")
                .register(meterRegistry);
    }

    public void generateReport() {
        // The record() method takes a lambda and times its execution.
        reportGenerationTimer.record(() -> {
            try {
                // Simulate a slow operation
                Thread.sleep((long) (Math.random() * 1500));
            } catch (InterruptedException e) {
                Thread.currentThread().interrupt();
            }
        });
    }
}

When you use a Timer, Micrometer will generate multiple time series in Prometheus:

• myapp_report_generation_seconds_count: The total number of times the event occurred.
• myapp_report_generation_seconds_sum: The total cumulative time spent on this operation.
• myapp_report_generation_seconds_max: The maximum time recorded for a single event.
• (Optionally) Percentile histograms if configured.

4. Integrating with Prometheus

Now that your application is instrumented, you need to set up Prometheus to collect the data. This involves running Prometheus (typically in Docker) and giving it a configuration file that tells it where to find your application's metrics endpoint.

The key piece of configuration is the prometheus.yml file. Here’s a minimal example for a local development setup:

# prometheus.yml
global:
  scrape_interval: 15s # How often to scrape targets

scrape_configs:
  - job_name: 'spring-boot-app'
    metrics_path: '/actuator/prometheus' # The path on your app where metrics are exposed
    static_configs:
      - targets: ['host.docker.internal:8080'] # The address of your Spring Boot app

Note: host.docker.internal is a special DNS name that allows a Docker container (like Prometheus) to connect to services running on the host machine. If your Spring app is also running in Docker on the same network, you would use its container name (e.g., 'my-app:8080').

With Prometheus running and configured, it will start scraping your application every 15 seconds, storing the custom metrics you've just created.

Interview Corner

Being able to articulate not just how to implement something, but also the best practices and reasoning behind your choices, is what separates a senior candidate.

Question: "You've joined a team where developers are creating metrics like api_requests_get_users and api_requests_post_orders. What's the problem with this approach, and how would you refactor it using Micrometer and Prometheus best practices?"

A Strong Answer Structure:

1. Identify the Problem: "The main problem here is high metric cardinality and poor query flexibility. The developers are embedding dimensional data (the HTTP method and the endpoint path) directly into the metric name. This creates a separate metric for every single combination, which is inefficient and hard to manage."
2. Explain the Drawbacks: "This approach makes aggregation difficult. For example, to get the total number of all API requests, you'd have to manually sum up every individual metric. If you want to see all requests for the /users endpoint, regardless of method, it's again a complex query. The power of a dimensional time-series database like Prometheus is lost."
3. Propose the Solution: "The correct approach is to use tags (or labels in Prometheus). I would refactor this to a single metric name, for example api.requests. Then, I would add tags for the dimensions."
4. Provide a Code Example: "Using Micrometer, the implementation would look like this:

   meterRegistry.counter("api.requests", 
                         "method", "GET", 
                         "path", "/api/users")
                .increment();
   
   meterRegistry.counter("api.requests", 
                         "method", "POST", 
                         "path", "/api/orders")
                .increment();
   

   This single api_requests_total metric can now be easily queried in Prometheus. For instance, sum(rate(api_requests_total[5m])) gives you the total request rate, and sum(rate(api_requests_total{path="/api/users"}[5m])) by (method) gives you the request rate for the users API, broken down by HTTP method. This is far more flexible and scalable."

Conclusion

Great work! You've now learned how to create and expose custom, business-relevant metrics from your Spring Boot application. This moves you beyond basic application health checks and into the realm of true system observability, where you can quantitatively measure the performance and behavior of your specific services.

Key Takeaways:

• Micrometer is the standard instrumentation facade for JVM applications.
• Prometheus is a pull-based monitoring system that scrapes metrics from an HTTP endpoint.
• Custom metrics are created using the MeterRegistry. The most common types are Counter (for counting events) and Timer (for measuring duration and frequency).
• Tags are essential for creating flexible, queryable metrics. They allow you to add dimensions (like status, method, region) to your data without creating an explosion of metric names.

Preview of the Next Lesson:
We have now covered metrics and traces. In our next lesson, we will complete our deep dive into the three pillars of observability by focusing on the first: logs. You will learn to implement structured, correlated logging (including trace IDs) for effective debugging. This will tie everything together, showing you how to link your logs directly to the traces you implemented in the previous lesson for a seamless debugging experience.