Building a Resilient Notification Pipeline

1. From Synchronous Pain to Asynchronous Scale

Let's first consider the simplest, most naive approach: the notification service receives an API call and directly invokes the third-party provider's API (e.g., Twilio for SMS) before returning a response to the client.

What's wrong with this?

• High Latency: The client has to wait for our service and the third-party provider to respond. This can be seconds, which is unacceptable for a responsive API.
• Poor Reliability: If the provider's service is temporarily down or slow, our API call fails. The notification is lost unless the client implements complex retry logic.
• Tight Coupling: Our notification service is tightly coupled to the availability and performance of every third-party provider it integrates with.

The solution is to decouple the ingestion of a notification request from the processing and delivery of that notification. We achieve this by introducing a message queue as a buffer.

This fundamental shift to an asynchronous, message-driven architecture is the first step toward a scalable and resilient system.

2. The Core Pipeline: Queues and Workers

With a message queue in place, our system now has two distinct parts: producers and consumers.

• Producer (Notification Service): Its only job is to validate the incoming API request, create a notification event message, and publish it to the queue. It can then immediately return a 202 Accepted response to the client, acknowledging the request has been accepted for processing.
• Consumer (Worker/Dispatcher): A separate, stateless service (or group of services) that polls the queue for new messages. When it receives a message, it performs the actual work: fetches templates and preferences, renders the content, and calls the appropriate third-party provider.

This design naturally allows for horizontal scaling. If we have a surge of notifications, we can simply add more consumer instances to process the queue faster, without affecting the performance of the main API.

Designing for Priority

Not all notifications are created equal. A one-time password (OTP) needs to be delivered in seconds, while a promotional newsletter can tolerate a few minutes of delay. Processing all notifications in a single First-In-First-Out (FIFO) queue is a recipe for disaster. A large batch of low-priority marketing emails could delay a critical password reset notification.

To learn how a real-world, high-stakes company solves this, let's look at Razorpay's architecture.

The solution, as Razorpay implemented, is to use multiple queues or topics based on priority. A common setup is:

• Critical Queue (P0): For OTPs, security alerts. Workers for this queue are provisioned to ensure near-real-time processing.
• Standard Queue (P1): For transactional notifications like order confirmations and shipping updates.
• Promotional Queue (P2): For marketing campaigns, newsletters, and other non-urgent communication.

This ensures that high-priority messages can "jump the line" and are processed by dedicated workers, meeting their strict latency requirements.

3. Architecting for Reliability: Retries, DLQs, and Durability

Decoupling is a great start, but what happens when things inevitably fail? A robust pipeline must handle failures gracefully without losing data.

Retry Strategy and Dead-Letter Queues (DLQs)

Imagine a worker tries to send an SMS via Twilio, but Twilio's API returns a 503 Service Unavailable error. The worker shouldn't just give up. This is where a retry strategy comes in.

• Exponential Backoff with Jitter: Instead of retrying immediately, the worker should wait for a progressively longer period between attempts (e.g., 1s, 2s, 4s, 8s...). Adding a random "jitter" to this backoff time prevents multiple workers from retrying in a thundering herd.

But what if a notification repeatedly fails? Perhaps the phone number is invalid, or there's a persistent issue. After a configured number of retries (e.g., 3-5 attempts), we must stop trying to prevent an infinite loop. The failed message is then moved to a Dead-Letter Queue (DLQ).

The DLQ is a critical component for observability and recovery. It's a separate queue that collects messages that could not be processed. An on-call engineer can then inspect the DLQ to diagnose the problem, fix the underlying issue (e.g., correct a malformed template), and potentially re-queue the messages for processing.

Let's see how these concepts are integrated into a Kafka-based pipeline.

Durability: The Outbox Pattern

We've handled failures in the consumer, but what if the producer fails? Consider this scenario:

1. The Notification Service receives an API request.
2. It attempts to publish a message to Kafka.
3. Kafka is down for a brief moment, and the publish call fails.

What should the service do? If it returns an error to the client, the notification is lost unless the client retries. If we retry on our end, the client is stuck waiting. The core issue is that the notification event is not yet durably stored.

This is a classic distributed systems problem solved by the Outbox Pattern.

Here is the flow of the Outbox pattern:

1. The Notification Service receives the API request.
2. It starts a database transaction.
3. Inside that transaction, it writes the notification details to its primary data table (e.g., notifications) AND to a special outbox table.
4. It commits the transaction. Only if this is successful does it return a 202 Accepted to the client. The notification is now durably stored in our database.
5. A separate, asynchronous process (like a Change Data Capture service, e.g., Debezium) monitors the outbox table. When a new row appears, it reliably publishes the corresponding message to the message queue.
6. Once successfully published, the message in the outbox table can be marked as published or deleted.

This pattern ensures that even if the message queue is unavailable, the notification request is never lost. It guarantees at-least-once publishing to the queue.

At-Least-Once Delivery and Idempotency

Between retries and the outbox pattern, our system is designed for at-least-once delivery. However, this introduces a new risk: duplicate notifications. A worker might successfully send a notification but crash before it can acknowledge the message from the queue. The queue will then redeliver the same message to another worker, causing a duplicate.

To prevent this, our workers must be idempotent. This means processing the same message multiple times has the same effect as processing it once. We achieve this by checking if a notification has already been successfully sent before attempting to send it again.

The consumer workflow becomes:

1. Consume message containing notification_id.
2. Check a shared store (like Redis): Is notification_id already in the 'processed' set?
3. If yes, discard the message and acknowledge it.
4. If no, attempt to send the notification.
5. If sending succeeds, add notification_id to the 'processed' set (with a TTL) and then acknowledge the message.

4. The Complete Architecture

By combining these patterns, we arrive at a robust, scalable, and reliable architecture.

The flow is as follows:

1. A client sends a request to the API Gateway.
2. The Notification Service validates the request and uses the Outbox Pattern to durably store the notification and publish it to a priority-based Kafka topic (e.g., critical.sms, standard.email).
3. Channel-specific Worker Services (e.g., SMS Worker, Email Worker) consume messages from their respective topics.
4. Each worker checks for idempotency using Redis.
5. If the message is new, the worker attempts to send the notification via an external provider.
6. If sending fails, the worker uses exponential backoff to retry. After max retries, the message is sent to a DLQ.
7. The entire pipeline is monitored with dashboards and alerts for queue depth, processing latency, and failure rates.

Conclusion

In this lesson, we designed the complex machinery behind our simple notification API. We moved from a brittle synchronous design to a resilient, asynchronous pipeline capable of handling massive scale and component failures.

Key Takeaways:

• Decouple with Queues: Use message queues (like Kafka or SQS) to decouple request ingestion from processing, enabling scalability and resilience.
• Prioritize for Performance: Implement multiple priority queues to ensure critical notifications (like OTPs) meet low-latency SLAs, even under heavy load from low-priority tasks.
• Build for Failure: A reliable pipeline requires a robust retry strategy (exponential backoff with jitter) and a Dead-Letter Queue (DLQ) for messages that persistently fail.
• Guarantee Durability: Use the Outbox pattern to ensure that once a notification request is accepted, it is never lost, even if the message broker is temporarily unavailable.
• Ensure Idempotency: To prevent duplicate notifications in an at-least-once delivery system, workers must be idempotent, typically by using a shared store like Redis to track processed message IDs.

Next Lesson Preview:

We have now designed two major communication systems. In our next lesson, we will shift domains and tackle another classic system design problem: storage and media. We will begin by designing a file synchronization service like Dropbox, focusing on how clients and servers coordinate to keep files in sync efficiently and reliably.