Resilient Service Calls: Retries and Timeouts

1. The First Line of Defense: The Retry Pattern

Imagine you're calling a friend, but the line is busy. You don't give up immediately; you hang up and try again a minute later. The Retry pattern works on the same principle. It's designed to handle transient failures—temporary issues like a brief network hiccup, a momentary database deadlock, or a service restarting. Instead of failing immediately, the calling service automatically re-attempts the operation a few times.

To understand the scenarios where this is useful, let's watch a short conceptual explanation.

1.1. Implementing Retry with Resilience4j

Just like the Circuit Breaker, Resilience4j makes implementing the Retry pattern straightforward. The required dependencies (resilience4j-spring-boot2, spring-boot-starter-aop) are the same, so we can jump straight to configuration.

Step 1: Configure the Retry Instance

In your application.yml, you define a named retry instance.

# In application.yml

resilience4j:
  retry:
    instances:
      # This is the name of our retry instance
      inventoryService:
        maxAttempts: 3
        waitDuration: 2s
        retryExceptions:
          - java.util.concurrent.TimeoutException
          - java.io.IOException
        ignoreExceptions:
          - com.myapp.exceptions.InvalidRequestException

Let's break this down:

• maxAttempts: 3: The operation will be attempted a maximum of 3 times (1 initial call + 2 retries).
• waitDuration: 2s: It will wait for 2 seconds between each attempt.
• retryExceptions: A list of specific exceptions that should trigger a retry. If this is not specified, it defaults to retrying on all exceptions. It's a good practice to be explicit.
• ignoreExceptions: A list of exceptions that should not trigger a retry. For example, a 400 Bad Request error indicates a problem with the client's request, and retrying it will just yield the same error.

Step 2: Apply the @Retry Annotation

Now, you apply the annotation to the method you want to make resilient. The name must match the instance you configured in application.yml. You should also specify a fallbackMethod to execute if all retry attempts fail.

import io.github.resilience4j.retry.annotation.Retry;
import org.springframework.stereotype.Service;

@Service
public class OrderService {

    private final InventoryClient inventoryClient;
    
    // ... constructor ...

    @Retry(name = "inventoryService", fallbackMethod = "fallbackForCheckAvailability")
    public boolean checkProductAvailability(String sku) {
        // This call might fail due to a temporary network issue
        return inventoryClient.isInStock(sku);
    }

    // Fallback method executed after all retries fail
    public boolean fallbackForCheckAvailability(String sku, Throwable t) {
        System.err.println("All retries failed for SKU: " + sku + ". Error: " + t.getMessage());
        // Assume unavailability as a safe default
        return false;
    }
}

The following article provides a clear, textual walkthrough of these steps.

1.2. Advanced Retry: Exponential Backoff

Constantly retrying every 2 seconds might not be the best strategy. If a downstream service is struggling under heavy load, repeated rapid-fire retries can make the problem worse (this is known as the "thundering herd" problem).

A better approach is exponential backoff. With this strategy, the wait time between retries increases exponentially (e.g., 1s, 2s, 4s, 8s...). This gives the struggling service progressively more time to recover.

You can enable this easily in your configuration:

resilience4j:
  retry:
    instances:
      inventoryService:
        maxAttempts: 4
        waitDuration: 500ms # Initial wait duration
        enableExponentialBackoff: true
        exponentialBackoffMultiplier: 2 # Multiplies the wait duration by 2 for each subsequent retry

With this configuration, the retries would occur after approximately 500ms, then 1000ms (1s), then 2000ms (2s).

2. Guarding Against Slowness: The Timeout Pattern

Retries handle transient failures, but what about a service that isn't down but is just very slow? Without a timeout, the thread making the call could be blocked indefinitely, consuming resources and impacting the performance of your entire service.

A timeout sets a maximum duration you are willing to wait for a response. If the response doesn't arrive in time, the call is aborted, and an exception is thrown, freeing up the thread.

2.1. Implementing Timeout with @TimeLimiter

Resilience4j provides the @TimeLimiter annotation for this purpose. However, there's a crucial requirement: the decorated method must be asynchronous and return a CompletableFuture. This is because terminating a synchronous, blocking call from another thread is complex and risky in Java.

Step 1: Configure the Timeout Instance

# In application.yml

resilience4j:
  timelimiter:
    instances:
      inventoryService:
        timeoutDuration: 3s
        cancelRunningFuture: true # Tries to cancel the underlying task on timeout

Step 2: Apply the @TimeLimiter Annotation

You must wrap your potentially slow, synchronous call inside a CompletableFuture.supplyAsync() block.

import io.github.resilience4j.timelimiter.annotation.TimeLimiter;
import java.util.concurrent.CompletableFuture;

@Service
public class OrderService {
    // ...

    @TimeLimiter(name = "inventoryService", fallbackMethod = "fallbackForCheckAvailabilityFuture")
    public CompletableFuture<Boolean> checkProductAvailabilityAsync(String sku) {
        return CompletableFuture.supplyAsync(() -> inventoryClient.isInStock(sku));
    }

    public CompletableFuture<Boolean> fallbackForCheckAvailabilityFuture(String sku, Throwable t) {
        System.err.println("Timeout calling inventory for SKU: " + sku + ". Error: " + t.getMessage());
        return CompletableFuture.completedFuture(false);
    }
}

The reflectoring.io article on timeouts provides a great example of this asynchronous implementation.

2.2. A Simpler Alternative: Client-Level Timeouts

While @TimeLimiter is powerful, requiring CompletableFuture can add complexity. A more common and often simpler approach is to configure timeouts directly on the HTTP client itself (e.g., RestTemplate, WebClient, or RestClient).

This approach is synchronous and doesn't require any changes to your service method signatures. Here's how you can do it with Spring's RestClient, which you might use to build a client.

import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.web.client.RestClient;
import java.time.Duration;

@Configuration
public class MyRestClientConfig {

    @Bean
    public RestClient inventoryRestClient() {
        return RestClient.builder()
                .baseUrl("http://inventory-service")
                .setConnectTimeout(Duration.ofSeconds(2)) // Max time to establish a connection
                .setReadTimeout(Duration.ofSeconds(5))    // Max time to wait for data after connection
                .build();
    }
}

In this scenario, if the inventory service doesn't respond within 5 seconds, the RestClient will throw an exception. This failure can then be handled by the @Retry mechanism or counted by the @CircuitBreaker.

The following video demonstrates this practical approach.

3. The Complete Resilience Strategy

Retry, Timeout, and Circuit Breaker are not isolated patterns; they work together to create a robust, multi-layered defense. Here’s the typical order of operations for a single remote call:

1. The Call is Made: A RestClient attempts to call a downstream service.
2. Timeout Check: The HTTP client's readTimeout is active. If the service doesn't respond within the configured duration (e.g., 3 seconds), the client throws a TimeoutException. This is counted as one failure.
3. Retry Logic: If the call fails (either from a timeout or another error like IOException), the @Retry mechanism kicks in. It waits for the waitDuration and re-attempts the call. This happens up to maxAttempts.
4. Circuit Breaker Monitoring: The @CircuitBreaker is watching all of this. It counts each definitive failure (i.e., a failure after all retries are exhausted, or a failure from a non-retriable exception).
5. Opening the Circuit: If the rate of these definitive failures exceeds the failureRateThreshold, the circuit breaker trips to the OPEN state, and all subsequent calls are immediately rejected, invoking the fallback logic without even attempting a network call.

This combination ensures you can handle everything from a momentary glitch (retry) to a slow service (timeout) to a complete outage (circuit breaker).

Conclusion

You have now added two more essential tools to your microservice resilience toolkit. By combining these patterns, you can build applications that are not just functional but are also robust and can gracefully handle the inevitable failures of a distributed environment.

Key Takeaways:

• The Retry pattern is your first line of defense against transient, intermittent failures. Use @Retry with a sensible number of attempts and consider exponential backoff.
• The Timeout pattern is crucial for preventing your application from getting stuck on slow dependencies. You can implement it directly on your HTTP client (e.g., RestClient) or use @TimeLimiter for asynchronous operations.
• Retry, Timeout, and Circuit Breaker work together as a multi-layered resilience strategy. Retries handle glitches, timeouts handle slowness, and circuit breakers handle major outages.

What's Next?

With patterns like circuit breakers and retries, a single user request can trigger a complex series of events across multiple services. How do you trace and debug a request that failed after three retries and then tripped a circuit breaker? In our next and final lesson of this module, you will learn how to implement distributed tracing with Spring Cloud Sleuth and Zipkin, giving you the visibility needed to monitor and understand these complex interactions.