Distributed Tracing with Spring Cloud Sleuth and Zipkin

1. The "Why" and "What" of Distributed Tracing

In a microservices architecture, a single user action can trigger a chain reaction across multiple services. If any part of that chain is slow or fails, the entire request can be affected. Distributed tracing provides a way to follow that request from the moment it enters the system until the final response is sent.

To do this, we rely on two key concepts:

• Trace: Represents the entire end-to-end journey of a request. It is identified by a single, unique Trace ID.
• Span: Represents a single unit of work or operation within a trace, such as an HTTP call to a specific service or a database query. Each span has its own unique Span ID. Spans can also have parent-child relationships, forming a tree structure that shows which operation triggered another.

This information (the Trace ID and current Span ID) is passed between services, typically in HTTP headers. This is called context propagation. Each service then reports its timing data for its part of the job to a central server.

To get a clear visual explanation of this process, watch the first part of the following video.

2. The Tools: Micrometer Tracing and Zipkin

To implement distributed tracing, we need two types of components:

1. Instrumentation Library: A library that lives inside our application. It's responsible for generating trace and span IDs, adding them to logs, and propagating them in outgoing network requests.
2. Tracing Backend: A separate server responsible for collecting, storing, and visualizing the trace data sent by our services.

For our stack, we will use:

• Micrometer Tracing: Our instrumentation library.
• Zipkin: Our tracing backend.

A Note on Spring Cloud Sleuth vs. Micrometer Tracing

You will find many tutorials and articles online that mention Spring Cloud Sleuth. For a long time, Sleuth was the standard for integrating distributed tracing in Spring. However, with the release of Spring Boot 3, the tracing instrumentation has been moved into the Micrometer project, which you might already know as a library for collecting application metrics.

Micrometer Tracing is the successor to Spring Cloud Sleuth. It provides the same core functionality but is more flexible and better integrated with the wider observability features in Spring. We will be using this modern approach.

3. Hands-On: Implementing Distributed Tracing

Now, let's add tracing to the microservices you've been building.

Step 1: Run the Zipkin Server

First, we need to get our tracing backend, Zipkin, up and running. The easiest way to do this is with Docker. Open your terminal and run the following command:

docker run -d -p 9411:9411 openzipkin/zipkin

This command downloads the latest Zipkin image and starts it in a container. It will listen on port 9411. You can open your web browser to http://localhost:9411 to see the Zipkin UI. It will be empty for now, but we'll fix that soon.

The following resources walk through this simple setup process.

Step 2: Add Dependencies to Your Microservices

Next, you need to add the necessary dependencies to the pom.xml file of each microservice you want to trace (e.g., your API Gateway, Order Service, Inventory Service, etc.).

<dependencies>
    <!-- Enables observability features -->
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-actuator</artifactId>
    </dependency>

    <!-- Micrometer Tracing bridge for the Brave tracer -->
    <dependency>
        <groupId>io.micrometer</groupId>
        <artifactId>micrometer-tracing-bridge-brave</artifactId>
    </dependency>

    <!-- Reporter that sends traces from Brave to Zipkin -->
    <dependency>
        <groupId>io.zipkin.reporter2</groupId>
        <artifactId>zipkin-reporter-brave</artifactId>
    </dependency>
</dependencies>

Let's break down what these do:

• spring-boot-starter-actuator: A foundational dependency for monitoring and management features in Spring Boot.
• micrometer-tracing-bridge-brave: This is the core dependency. It integrates Micrometer Tracing with Brave, a popular and robust tracing instrumentation library.
• zipkin-reporter-brave: This is the "connector" that knows how to take the trace data collected by Brave and send it to a Zipkin server.

Step 3: Configure Your Applications

Now, add the following configuration to the application.yml file of each microservice.

spring:
  application:
    name: order-service # IMPORTANT: Change this for each service!

management:
  tracing:
    sampling:
      # Trace 100% of requests. Lower this in production (e.g., 0.1 for 10%).
      probability: 1.0 
  zipkin:
    tracing:
      # The URL of your Zipkin server
      endpoint: "http://localhost:9411/api/v2/spans" 

Key configuration points:

• spring.application.name: This is crucial! Zipkin uses this name to identify which service each span came from. Make sure it's unique for each of your services.
• management.tracing.sampling.probability: This controls what percentage of requests get traced. A value of 1.0 means 100% of requests are traced, which is perfect for development. In a high-traffic production environment, you would lower this to reduce overhead.
• management.zipkin.tracing.endpoint: This tells Micrometer where to send the trace data.

Step 4: See It in Action!

You're all set up. Now, follow these steps:

1. Restart all of your microservices to apply the new dependencies and configuration.
2. As the services start, look at the console logs. You should now see log entries formatted with the application name, a trace ID, and a span ID, like this:
   INFO [order-service,2a8b...,2a8b...] 2316 --- [main] c.e.orderservice.OrderServiceApplication : Starting OrderServiceApplication...
3. Use Postman or your browser to make a request that travels through your microservice chain (e.g., a request to the API Gateway that gets routed to the Order Service, which then calls the Inventory Service).
4. Go back to the Zipkin UI at http://localhost:9411 and click the Run Query button.

You should now see your request listed as a trace!

Example of the Zipkin UI. You can see the services involved, the total duration of the request, and a timeline showing how long each step took.

Click on a trace to see the detailed view. It will show you the full hierarchy of spans, the time spent in each service, and any associated tags or logs. This gives you an incredibly powerful view of your system's behavior.

The video below demonstrates this entire process, from adding dependencies to visualizing the trace in Zipkin.

4. Customizing Traces for Better Debugging

You can enrich your traces with custom information that is specific to your application's domain. For example, you could tag a trace with the ID of the user who made the request or the order number being processed. This makes it much easier to find specific traces when debugging a problem reported by a customer.

To do this, you can inject the Tracer object into your service and interact with the current span.

import brave.Span;
import brave.Tracer;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.PathVariable;
import org.springframework.stereotype.Service;

@Service
public class OrderService {

    @Autowired
    private Tracer tracer;

    // ... other dependencies and methods

    public Order getOrderDetails(String orderId) {
        // Add a custom tag to the current span
        Span currentSpan = tracer.currentSpan();
        if (currentSpan != null) {
            currentSpan.tag("order.id", orderId);
            // You can add any key-value pair you find useful
            currentSpan.tag("business.process", "FetchOrderDetails"); 
        }

        // ... existing logic to fetch order from database ...
        return orderRepository.findById(orderId);
    }
}

When you view this trace in Zipkin, you will now see order.id and business.process in the tags for the order-service span, making your traces searchable and much more informative.

For more examples of customization, including how to exclude certain endpoints from being traced, the following article is a great resource.

Conclusion

Congratulations on completing the final lesson of our Spring Cloud module! You've tackled one of the most significant challenges in a distributed architecture: observability. By instrumenting your services with Micrometer Tracing and visualizing the data in Zipkin, you've moved from working with a collection of isolated black boxes to managing a transparent, understandable system.

Key Takeaways:

• Distributed tracing is essential for debugging and monitoring microservices by tracking a request's journey across service boundaries.
• A Trace represents the entire request, and a Span represents a single operation within that request.
• In modern Spring Boot (3+), Micrometer Tracing is the standard for instrumenting your applications, replacing the older Spring Cloud Sleuth.
• Zipkin is a popular tracing backend used to collect and visualize trace data.
• Implementation involves running Zipkin, adding dependencies to each service, and configuring the application name and Zipkin endpoint.

What's Next?

You now have a complete, resilient, and observable microservices system running on your local machine. The next logical step is to learn how to package this system for deployment and automate the process of getting it from your code repository to a running environment.

In our next module, Containerization and CI/CD, we will begin by diving into Docker. You'll learn how to create a portable, self-contained "box" for your Spring Boot applications, ensuring they run the same way anywhere. This is the first step towards professional, repeatable deployments. See you there