Distributed Communication Patterns for AI Systems

Welcome back to our series on Distributed Systems Foundations. In our last lesson, we examined data replication, establishing how synchronous and asynchronous strategies create a trade-off between durability and performance. We saw that replicating data from a leader to its followers requires some form of communication. But what do those communication mechanisms actually look like?

This lesson moves up one layer in the stack to answer that question. We will analyze the fundamental communication patterns that serve as the nervous system for any distributed architecture: Remote Procedure Calls (RPC), message queues, and publish-subscribe (pub-sub) systems. For each pattern, we'll dissect its characteristics, trade-offs, and typical use cases. As a senior engineer, you'll recognize these as scaled-out, distributed versions of familiar programming paradigms. Our goal is to equip you to analyze a complex AI system requirement—like serving real-time recommendations or orchestrating model training—and select the most appropriate communication pattern for each part of the design.

The Two Philosophies: Synchronous vs. Asynchronous

At the highest level, communication between services can be divided into two philosophies: synchronous and asynchronous. This distinction is the most critical first step in designing any interaction.

Synchronous Communication: The sender makes a request and then waits for a response. The sender is blocked, and its thread of execution cannot proceed until the receiver replies or a timeout occurs. This is like making a phone call: you ask a question and wait for the answer before you hang up and move on.
Asynchronous Communication: The sender dispatches a message and immediately continues with its own work, without waiting for a response. The receiver processes the message at its own pace. If a response is needed, it's sent back as a separate, independent message. This is like sending an email: you send your message and then go about your day, checking your inbox for a reply later.

The image below provides a great visual summary of this core difference, and also previews some of the key protocols we'll be discussing.

This diagram contrasts the blocking nature of synchronous calls with the non-blocking, queue-based approach of asynchronous communication. It also introduces several key protocols (REST, gRPC, AMQP, MQTT) that implement these patterns.

The Direct Approach: Remote Procedure Call (RPC)

RPC is the most common implementation of the synchronous, request-response pattern. Its goal is to make a function call on a remote service feel just like calling a local function in your code. The client code calls a "stub" method, which handles the complex work of serializing the request, sending it over the network to the server, waiting for the response, deserializing it, and returning the result to the caller.

From a developer's perspective, it looks simple:
prediction = model_service.predict(feature_vector)

Under the hood, model_service is a client stub that handles the network communication.

Characteristics of RPC:

Coupling: RPC creates a tight coupling between the client and server. The client must know the server's network location (address and port) and the precise signature of the remote procedure it wants to call. If the server is down, the client's call will fail.
Latency: For a simple request-response, RPC generally offers lower latency than brokered messaging systems because there is no intermediary. The request goes directly from client to server.
Simplicity: The mental model is very straightforward for developers, as it mirrors the familiar function-call paradigm.

A modern, high-performance evolution of this pattern is gRPC (Google RPC). It uses the efficient Protocol Buffers format for serializing data and leverages the HTTP/2 protocol for transport, making it a popular choice for internal communication between microservices in high-performance systems, including many AI platforms.

AI System Use Case: The most classic use case for RPC in AI infrastructure is model serving for online inference. When a user-facing application needs a real-time prediction, it makes a synchronous call to a model serving endpoint, passing the input data (e.g., text, image, feature vector) and blocking until it receives the prediction. The application needs the result now to render the next page or action for the user, making the synchronous nature of RPC a perfect fit.

The Indirect Approach: Asynchronous Messaging

Asynchronous messaging introduces an intermediary component called a message broker (e.g., RabbitMQ, Apache Kafka, AWS SQS) between the communicating services. Instead of talking to each other directly, services communicate by sending and receiving messages through the broker. This approach fundamentally decouples the services.

The following video provides an excellent overview of the core concepts behind asynchronous messaging.

From REST to gRPC to Messaging: SCALING Microservices the Right Way!

This video from ByteMonk clearly explains the shift from direct communication (like RPC) to brokered, asynchronous messaging.

Please watch the following segments: Start with the introduction to asynchronous communication, which defines the core idea. Then, jump to the comparison between brokerless and broker-based architectures. This directly contrasts the RPC-style approach with messaging. Pay close attention to the benefits a broker provides, such as reliability and loose coupling. Finally, watch the summary of why brokers are beneficial despite the added operational overhead.

As the video explains, using a broker provides several key advantages:

Loose Coupling: The message producer doesn't need to know the location, or even the identity, of the message consumer. It just needs to know where the broker is.
Temporal Decoupling: The producer and consumer don't need to be running at the same time. The broker can store messages in a queue until the consumer becomes available to process them.
Resilience and Buffering: The broker acts as a buffer, absorbing spikes in load. If an application suddenly generates a large number of messages, the broker can queue them, allowing the consumer to process them at a steady rate without being overwhelmed.

There are two primary patterns for asynchronous messaging.

1. Message Queues (Point-to-Point)

In the message queue, or point-to-point, pattern, a message is sent by a producer to a specific queue. A consumer, or a pool of competing consumers, listens on that queue. The broker ensures that each message is processed by only one consumer. If multiple consumers are listening, they effectively compete to pull messages from the queue, which provides a natural way to parallelize work and load-balance.

Characteristics of Message Queues:

Intended for: Distributing tasks to a pool of workers.
Delivery: One message is delivered to one consumer.
Use Case: Ideal for imperative commands, like "process this image," "train this model," or "generate this report."

AI System Use Case: A system for fine-tuning LLMs. A user uploads a dataset and configuration, and the web application submits a START_FINETUNING_JOB message to a queue. A fleet of GPU worker machines are configured as consumers for this queue. One of the available workers will pick up the message, provision resources, download the data, run the training job (which could take hours), and upon completion, report the status back (often via another message queue). This decouples the long-running, resource-intensive training task from the responsive web front-end.

2. Publish-Subscribe (Pub/Sub)

In the publish-subscribe pattern, a producer, now called a publisher, sends a message to a logical channel known as a topic. Any number of consumers, called subscribers, can express interest in that topic. When a message is published to the topic, the broker delivers a copy of the message to all active subscribers. The publisher has no knowledge of who the subscribers are, or if there are any at all.

Characteristics of Pub/Sub:

Intended for: Broadcasting events or notifications.
Delivery: One message is delivered to many consumers.
Use Case: Ideal for notifying different parts of a system that "something happened." This is the foundation of event-driven architecture.

AI System Use Case: A real-time data pipeline for an e-commerce site. A service publishes user interaction events (e.g., product-viewed, item-added-to-cart, order-placed) to different topics on a pub/sub system like Apache Kafka or AWS Kinesis. Multiple, independent downstream systems subscribe to these topics:

A real-time personalization service consumes product-viewed events to update a user's recommendations.
An inventory management service subscribes to order-placed events to decrement stock counts.
A business analytics dashboard subscribes to all events to update metrics in real-time.
A data lake ingestion service also subscribes to everything to create a historical archive for offline model training.

All these systems react to the same events in parallel without any knowledge of each other's existence. Adding a new system (e.g., a fraud detection service) just requires creating a new subscriber; no existing services need to be changed.

This article provides more context on these patterns, specifically in the world of multi-agent systems, which aligns with your interest in more complex AI architectures.

Agent Communication Protocols

This article discusses communication in multi-agent systems, but the patterns are universal. The descriptions here provide a solid reinforcement of what we've covered.

Please read the section on Communication Patterns. It clearly defines and contrasts the synchronous Request-Reply pattern (our RPC) with the asynchronous Publish-Subscribe pattern, which are the main focus of our lesson.

Choosing the Right Pattern: A Hybrid Approach

No single pattern is universally best. Professional-grade distributed systems are almost always hybrids, using the right pattern for the right job.

Let's analyze a real-world AI system architecture to see this in practice.

This diagram shows the architecture for a real-time recommendation system, which combines multiple communication patterns to achieve its goal.

Let's break down the communication patterns in this diagram:

Step 1 → 2 (Pub/Sub): User events are sent from the client to an Amazon Kinesis Data Stream. Kinesis is a classic pub/sub system. The client publishes events without knowing what will consume them. A Spark Streaming job subscribes to this stream to process the events. This is a perfect use case for pub/sub, as it decouples the application from the real-time processing pipeline.
Step 6 → SageMaker Model Endpoint (RPC): The Prediction Service makes a request to a SageMaker Model Endpoint to get a prediction. This is a classic synchronous, request-response interaction. The service needs the prediction now to return it to the user. This is almost certainly implemented as a REST API call, which is a form of RPC.
Step 5 → Sagemaker Online Feature Store (RPC): The Lambda function fetches features from the Online Feature Store. Again, this is a direct, synchronous request for data needed to serve the prediction. It's a request-response call, likely a high-performance RPC or API call.
Step 3 (Prediction queue) & Step 4 (Lambda invocation): This is an interesting asynchronous pattern. The Stream Receiver sends a message to a prediction queue. An AWS Lambda function is invoked in response to messages on this queue. This could be a message queue pattern used to buffer requests to the prediction Lambda, preventing the stream receiver from being blocked and allowing the system to handle bursts of activity.

This single architecture beautifully illustrates how different parts of a system have different communication needs, leading to a hybrid design where RPC, Pub/Sub, and Message Queues all play a vital role.

Pattern	Coupling	Interaction Style	Key Advantage	AI Infra Example
RPC	Tight	Synchronous Request-Response	Low latency, simple mental model	Online model inference (e.g., calling a SageMaker endpoint)
Message Queue	Loose	Asynchronous Command	Load balancing, task distribution, buffering	Submitting model training jobs to a worker pool
Pub/Sub	Very Loose	Asynchronous Event Notification	Decoupling, parallel processing	Streaming user logs to multiple services (analytics, personalization, etc.)

Conclusion

In this lesson, we've dissected the three primary distributed communication patterns that form the building blocks of AI infrastructure.

Here are the key takeaways:

RPC provides a simple, low-latency, synchronous request-response model, ideal for when a client needs an immediate answer from a specific service, like in online model serving.
Message Queues enable asynchronous, point-to-point communication through a broker, perfect for distributing tasks to a pool of workers and ensuring each task is handled once.
Publish-Subscribe systems allow a single message to be broadcast to multiple interested subscribers, providing maximum decoupling for event-driven architectures.
Real-world systems are hybrids, selecting the optimal pattern for each specific interaction to balance performance, scalability, and resilience.

You now have a framework for analyzing the flow of information in a distributed system. The next logical question is: when you have multiple instances of a service to handle the load (like the workers listening to a queue or multiple servers behind an RPC endpoint), how do you distribute requests or data among them intelligently and efficiently?

Our next lesson will introduce consistent hashing, a crucial algorithm that solves this exact problem, enabling scalable load distribution and data partitioning in the dynamic environments typical of modern cloud infrastructure.

Recommended resources