Skip to main content
Back to path

Welcome to a Grasp lesson

Create a free account to keep studying and create your own study path.

Lesson 58

Choosing Communication Patterns for Service Interactions

Recommended resources

Introduction

In our previous lessons, we've focused on the low-level mechanics of reliable service communication, covering topics from gRPC internals to backpressure patterns. You've learned how individual services can protect themselves and how feedback loops like exponential backoff and circuit breakers prevent localized issues from becoming system-wide cascading failures.

Today, we move from the tactical to the strategic. We'll elevate our perspective from how services talk, to how they collaborate to achieve a business outcome. This is a fundamental aspect of system design and a key area of responsibility for a Principal Engineer. Our goal is to equip you with a robust framework to select and justify communication patterns for key service interactions in any given scenario. We'll explore the core trade-offs between different collaboration styles and establish a vocabulary for analyzing and designing these critical workflows.

The Three Forces of Service Interaction

When designing a workflow that spans multiple services—like an e-commerce checkout—you are constantly balancing three fundamental forces. Understanding these forces, as articulated by Mark Richards and Neal Ford, is the first step toward making deliberate and defensible architectural choices.

  1. Communication: How do services exchange information? The primary choice is between synchronous (blocking) and asynchronous (non-blocking) styles.
  2. Coordination: Who directs the workflow? The choice is between orchestration (a central controller) and choreography (decentralized, event-driven reactions).
  3. Consistency: What are the data guarantees across services? The choice is between atomic (all-or-nothing, ACID-like) and eventual (data converges over time, BASE-style) consistency.

These three forces are not independent; a choice in one dimension heavily constrains the others. For example, atomic consistency is very difficult to achieve with asynchronous, choreographed communication. We'll now examine each of these in more detail.

Diving Deeper: Communication and Coordination

While you are already familiar with synchronous (e.g., REST, gRPC) and asynchronous (e.g., Kafka, Pub/Sub) communication, the distinction between orchestration and choreography is more nuanced and critical for system design.

A common but overly simplistic view defines orchestration as a central "spider in the web" and choreography as services just talking to each other. A more powerful definition, proposed by expert Bernd Rücker, frames the distinction around intent.

Balancing Choreography and Orchestration • Bernd Rücker • GOTO 2020

This GOTO conference talk by Bernd Rücker provides a masterclass in the nuances of orchestration and choreography. It will help you move beyond superficial definitions to a deeper understanding of the underlying principles.

Please watch the following key segments: Common Definitions: The speaker critiques the popular, yet flawed, definitions of orchestration and choreography. The Event Chain Problem: This section explains the "pinball machine architecture," a major risk of pure choreography where understanding and changing business flows becomes incredibly difficult. This directly relates to the maintainability and evolution of a system. The Essence of the Difference: Here, Rücker presents his core thesis: Orchestration is command-driven communication (expressing intent), while Choreography is event-driven communication (reacting to facts). Making the Distinction: The talk concludes with practical heuristics for deciding between the two, focusing on coupling, responsibility, and the fact that most good architectures are a mix of both.

As the video emphasizes, the choice is not about technology (you can send a command over a message queue) but about which service holds the responsibility for the business process.

  • In Orchestration (Command-driven), a controller service is accountable for the end-to-end workflow. It issues explicit commands like ProcessPayment or ShipOrder. This centralizes the business logic, making the process explicit, visible, and easier to manage, but can introduce a potential bottleneck and tighter coupling.

  • In Choreography (Event-driven), services are decoupled and react to events. A service emits a factual event like OrderPlaced. Other services subscribe to this event and perform their work independently, possibly emitting new events. This promotes autonomy and scalability but can obscure the overall workflow, making it difficult to monitor, debug, and change.

The following architecture for an e-commerce system clearly shows a hybrid approach, which is typical for real-world systems.

In this e-commerce architecture, synchronous patterns like gRPC and REST are used for interactions requiring immediate response or tight control (likely orchestration), while asynchronous Kafka is used for processes where decoupling and resilience are paramount (choreography).

A Framework for Decision-Making

With this deeper understanding, how do you decide which pattern to use? The choice depends on the specific requirements of the interaction. A formal decision framework can help structure this analysis.

Microservices Orchestration vs. Choreography: A Decision ...

This research paper proposes a formal framework for choosing between orchestration and choreography. It provides a set of factors to evaluate, which is precisely the kind of structured thinking required for a Principal-level design role.

Please read the following sections: Start with Section III, "MICROSERVICE COLLABORATION PATTERNS". Read the subsections on Choreography and Orchestration. Pay close attention to the listed strengths and weaknesses, as these form the basis of the decision framework. Then, read Section IV, "DECISION FRAMEWORK". Focus on the six collaboration factors the authors identify. These are the concrete criteria you can use to evaluate your design choices.

The six factors from the paper give us a powerful checklist:

  1. Distribution (Chattiness): Are services deployed across a WAN? If so, asynchronous, less chatty patterns (choreography) are favored.
  2. Predictability of Response Time: Does a user need an immediate, predictable response? This strongly favors synchronous patterns (orchestration).
  3. Loose Coupling: Is independent deployability critical? Asynchronous patterns provide looser temporal coupling.
  4. Reusability (Atomicity): Are services intended to be atomic, reusable components? Orchestration often leads to more reusable "worker" services, as the process logic is externalized.
  5. Complexity (Number of services): For complex flows with many steps, orchestration provides better visibility and manageability than a tangled web of event subscriptions.
  6. Runtime Visibility: Is it critical to know the state of a workflow at any time? Orchestration makes this trivial; choreography requires sophisticated distributed tracing.

A Taxonomy of Patterns: The Saga Model

Now we can synthesize all three forces—Communication, Coordination, and Consistency—into a practical set of patterns known as Sagas. A saga is a sequence of local transactions where each transaction updates the database within a single service. When one step completes, it triggers the next. If a step fails, the saga executes compensating transactions to undo the preceding work.

The following article provides an excellent taxonomy of eight saga patterns, which are simply the combinations of the three forces we've discussed.

Designing Distributed Systems: Sagas and Trade-Offs

This article provides a clear and practical guide to sagas, framing them as combinations of Communication, Coordination, and Consistency. The named patterns give you a valuable shorthand for discussing design options.

First, quickly read the introductory sections on the three forces to reinforce the concepts: Communication, Coordination, and Consistency. Then, focus on the "Saga Patterns" section. You don't need to memorize all eight. Instead, focus on understanding these four key patterns and their trade-offs: Fairy-Tale Saga (Synchronous, Eventual, Orchestrated): This is the classic REST-based orchestration. It's simple and provides good control but offers lower availability. Anthology Saga (Asynchronous, Eventual, Choreographed): The canonical event-driven pattern. Highly scalable and decoupled, but lacks visibility and makes error handling complex. Parallel Saga (Asynchronous, Eventual, Orchestrated): A powerful modern pattern using tools like AWS Step Functions or Netflix Conductor. It offers the control of orchestration with the scalability and resilience of asynchronous messaging. Horror-Story Saga (Asynchronous, Atomic, Choreographed): Note this as an anti-pattern to be avoided due to its extreme complexity.

Application: Designing a Returns Workflow

Let's apply this framework to a concrete problem. Imagine you're designing a new returns processing workflow for the e-commerce system pictured earlier.

The high-level process is:

  1. A customer initiates a return from their order history page.
  2. The system verifies the order is eligible for return (Order Service).
  3. A shipping label is generated and presented to the customer (Shipping Service).
  4. A notification is sent to the customer with the label and instructions (Notification Service).
  5. When the returned package arrives at the warehouse, it is scanned and inspected (Warehouse Service).
  6. If approved, a refund is issued to the customer's original payment method (Payment Service).
  7. The item is returned to stock (Inventory Service).
  8. A final confirmation is sent to the customer (Notification Service).

Your Task:
Think about the interactions between these services. For each of the three main phases below, select and justify a communication pattern. Use the vocabulary we've established (Orchestration/Choreography, Command/Event-driven, Saga Patterns) and the six decision factors.

  • Phase 1: Return Initiation (Steps 1-4)
  • Phase 2: Item Processing (Step 5)
  • Phase 3: Refund and Restock (Steps 6-8)
Click here to see a sample analysis

Here is one possible way to analyze this problem:

  • Phase 1: Return Initiation (Steps 1-4)

    • Interaction: Customer request -> Order Service -> Shipping Service -> Customer response -> Notification Service.
    • Analysis: This is a user-facing interaction. Predictable response time and runtime visibility are critical. The user needs to know immediately if their return is accepted and must receive a shipping label. This points strongly towards a synchronous, orchestrated flow.
    • Pattern Selection: Orchestration (Fairy-Tale Saga). A Returns Service would act as the orchestrator.
      1. It receives the synchronous request from the UI.
      2. It sends a synchronous command (VerifyReturnEligibility) to the Order Service.
      3. It sends a synchronous command (GenerateReturnLabel) to the Shipping Service.
      4. It returns the label URL to the UI.
      5. In parallel (or after the response), it publishes an asynchronous event ReturnInitiated to a message broker. The Notification Service subscribes to this event to send the email, decoupling it from the main flow. This is a classic example of a hybrid approach.

  • Phase 2: Item Processing (Step 5)

    • Interaction: A warehouse employee scans the package, triggering the Warehouse Service.
    • Analysis: This is an internal, asynchronous trigger. The key is to communicate the result of the inspection to the rest of the system.
    • Pattern Selection: Choreography (Event-driven). The Warehouse Service, upon completion of inspection, publishes a factual event, either ReturnApproved or ReturnRejected. This is a perfect use for choreography because the Warehouse Service shouldn't be responsible for what happens next (refunds, notifications, etc.). It simply states a fact.

  • Phase 3: Refund and Restock (Steps 6-8)

    • Interaction: The ReturnApproved event triggers the refund, restock, and notification steps.
    • Analysis: This is a critical business process involving money and inventory. While a pure choreography approach (having Payment and Inventory services both react to ReturnApproved) is possible, it poses risks. What if the refund succeeds but the inventory update fails? Recovering from this distributed state is complex and lacks visibility. An orchestrator that is accountable for the "refund settlement" process is a more robust choice.
    • Pattern Selection: Orchestration (Parallel Saga). The Returns Service (the same one from Phase 1) subscribes to the ReturnApproved event. It then orchestrates the final steps asynchronously:
      1. It sends a IssueRefund command to the Payment Service.
      2. Upon successful confirmation, it sends an UpdateStock command to the Inventory Service.
      3. It publishes a ReturnCompleted event, which the Notification Service consumes.
        This pattern gives us the control and visibility of orchestration for a critical process, while the asynchronous communication provides resilience and scalability.

Conclusion

In this lesson, we established a comprehensive framework for selecting and justifying service communication patterns. You've learned that this is not a simple technical choice but a strategic design decision with far-reaching implications for your system's scalability, resilience, and maintainability.

Key takeaways include:

  • Service collaboration can be analyzed through the three forces of Communication, Coordination, and Consistency.
  • Orchestration vs. Choreography is best understood as Command-driven vs. Event-driven, a distinction rooted in business responsibility and accountability.
  • A structured decision process, using factors like response time, coupling, and visibility, allows you to make defensible architectural choices.
  • Real-world systems are almost always hybrids, mixing different patterns to meet the specific needs of each interaction.

In our next lesson, we will build upon this foundation. Having chosen our communication patterns, we will then explore how to propose a load balancing and reliability strategy to ensure these interactions perform correctly and efficiently at scale.