Distributed Tracing with Micrometer and Zipkin

1. Understanding the Architecture

Before we start, let's visualize our goal. We have multiple microservices that communicate with each other to fulfill a request. We want to instrument each service so that it sends trace data to a centralized server, Zipkin, which will then allow us to see the entire request journey.

The core technology we'll use is Micrometer Tracing, the standard tracing facade in Spring Boot 3 and beyond. It replaced the older Spring Cloud Sleuth project. Micrometer provides a vendor-neutral API, and we'll use the Brave implementation, which is the tracer library compatible with Zipkin.

2. Setting Up the Tracing Backend: Zipkin

First, we need a tracing backend to receive and display our traces. We'll use Zipkin, a popular and easy-to-use open-source distributed tracing system. The most common way to run services like Zipkin in a development environment is with Docker.

We will add Zipkin as a service to our project's docker-compose.yml file. This ensures it starts up alongside our other infrastructure, like the database and service discovery server.

After following the video, your docker-compose.yml should include a new service definition for Zipkin, like the one shown in the video. It will typically expose port 9411.

  zipkin:
    image: openzipkin/zipkin
    container_name: zipkin
    ports:
      - "9411:9411"

Once you run docker-compose up, you should be able to access the Zipkin UI at http://localhost:9411. For now, it will be empty because none of our services are sending data to it. Let's fix that.

3. Instrumenting Your Spring Boot Microservices

Now for the main event: configuring our Spring Boot applications to generate and export traces. This involves two steps: adding dependencies and setting configuration properties.

Step 1: Add the Required Dependencies

As you learned in the official Spring blog post "Observability with Spring Boot 3," Micrometer Tracing requires a few key pieces to work with Zipkin.

Based on what you've read and what the video demonstrates, you'll need to add the following dependencies to the pom.xml of each microservice you want to trace:

1. spring-boot-starter-actuator: This is the foundation for all observability features in Spring Boot, including tracing.
2. micrometer-tracing-bridge-brave: This is the "bridge" library. It connects Micrometer's generic tracing API to Brave, the actual tracing implementation library that Zipkin understands.
3. zipkin-reporter-brave: This is the "reporter." Its job is to take the finished spans from Brave and send them over the network to the Zipkin server.

Here is the XML you'll need to add to your <dependencies> section in each pom.xml:

<!-- For Observability -->
<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-actuator</artifactId>
</dependency>

<!-- For Distributed Tracing with Zipkin -->
<dependency>
    <groupId>io.micrometer</groupId>
    <artifactId>micrometer-tracing-bridge-brave</artifactId>
</dependency>
<dependency>
    <groupId>io.zipkin.reporter2</groupId>
    <artifactId>zipkin-reporter-brave</artifactId>
</dependency>

Step 2: Configure Application Properties

With the dependencies in place, Spring Boot's auto-configuration will handle most of the setup. We just need to provide one crucial piece of configuration in application.properties for each service: the sampling rate.

# Send 100% of traces to Zipkin. Good for development, but not for production.
management.tracing.sampling.probability=1.0

# Optional: Set the application name to identify the service in Zipkin
spring.application.name=your-service-name

The management.tracing.sampling.probability property tells Micrometer what percentage of requests to trace.

• A value of 1.0 means 100% of requests will be traced. This is perfect for development and debugging.
• A value of 0.1 means 10% of requests will be traced.
• In a high-traffic production environment, you would use a lower value (e.g., 0.01 for 1%) to reduce the performance overhead and storage costs associated with tracing.

Now, go ahead and apply these dependency and property changes to all the microservices in your project.

4. Visualizing the End-to-End Trace

With everything configured, it's time for the payoff.

After making a request, head to the Zipkin UI. Click the "Run Query" button. You should now see a list of traces. Find the trace that corresponds to your API call (it will likely be the longest one involving multiple services).

When you click on a trace, you'll see a waterfall diagram. Pay attention to:

• The total duration of the request at the top.
• The individual spans for each service. The length of the bar represents the time taken.
• The service names on the left.
• The dependency graph in the "Dependencies" tab, which shows how your services call each other.

Try stopping one of your downstream services and making the API call again. You'll see the trace show up in red in Zipkin, instantly telling you where the failure occurred.

Interview Corner

Implementing distributed tracing is a common task, and being able to troubleshoot it is a key senior-level skill.

Question: "You've just deployed a new microservice, but its operations are not showing up in your Zipkin traces. What are the most likely causes?"

A Strong Answer Structure:

1. "First, I'd check the basics: dependencies and configuration. Did I remember to add spring-boot-starter-actuator, micrometer-tracing-bridge-brave, and zipkin-reporter-brave to the new service's pom.xml? A missing Actuator is a very common mistake."
2. "Next, I'd verify the application.properties. Is management.tracing.sampling.probability set to a non-zero value? Forgetting this means no traces will be sent. I'd also check that spring.application.name is set, so I can easily identify the service."
3. "Then, I'd look at networking. Can the new service container resolve and connect to the Zipkin container's address and port (http://zipkin:9411 in a Docker network)? Firewall rules or incorrect network configuration can block the trace data."
4. "Finally, I'd confirm traffic. Has the new service actually received any requests since it was deployed? Traces are only generated for actual operations. If it's idle, there will be no traces."

Conclusion

Congratulations! You have successfully implemented distributed tracing in a multi-service Spring Boot application. You now have a powerful tool to visualize request flows, pinpoint performance bottlenecks, and diagnose errors in a distributed environment.

Key Takeaways:

• Distributed tracing in Spring Boot 3 is powered by Micrometer Tracing.
• The essential dependencies for Zipkin integration are the actuator, the micrometer-tracing-bridge-brave, and the zipkin-reporter-brave.
• The management.tracing.sampling.probability property is critical for controlling trace volume and performance overhead.
• The Zipkin UI provides invaluable insights into request latency, service dependencies, and error analysis.

Preview of the Next Lesson:
We've now implemented the third pillar of observability: traces. In the next lesson, we will turn our attention back to the second pillar. You will learn to implement custom application metrics (e.g., counters, timers) using Micrometer and expose them for Prometheus. This will complete your practical knowledge of the core observability pillars.