Asynchronous Inference with SQS Queuing

Recommended resources

The freely available resources this lesson will direct you to.

Hello! In our last lesson, we established a robust strategy for ensuring your LLM inference pods are scheduled exclusively on the correct GPU-equipped nodes using taints, tolerations, and node affinity. This gives us control over where our expensive workloads run. Now, we'll shift our focus to managing how work gets to them.

This lesson addresses a fundamental challenge in serving large models: long inference times. A synchronous API, where a client sends a request and waits for the full generation to complete, is often impractical. It can lead to client timeouts, poor user experience, and inefficiently tied-up server resources. To build a truly scalable and resilient LLM service, we need to decouple the client from the workload.

We will achieve this by implementing an asynchronous processing pattern. By the end of this lesson, you will be able to implement request queuing using a message broker like SQS to enable asynchronous inference processing. This is a cornerstone of production-grade MLOps and leverages your extensive background in distributed systems and cloud infrastructure.

1. The Asynchronous Inference Pattern

In an asynchronous system, instead of waiting for the full result, the client submits a request and immediately receives a confirmation that the job has been accepted, usually with a unique task ID. The heavy lifting of inference happens in the background. The client can then use the task ID to poll for the result later or be notified upon completion.

This architecture offers several key advantages:

Decoupling: The client and the inference workers are not directly connected. The client doesn't need to know which worker will process its request or if the workers are currently busy.
Resilience: If an inference worker crashes mid-task, the request is not lost. It remains safely in the queue and can be picked up by another worker.
Load Leveling: The queue acts as a buffer. During sudden traffic spikes, it can absorb a large number of requests, which the workers can then process at a sustainable pace without being overwhelmed.

At the heart of this pattern is a message broker, a dedicated service for managing queues. For this lesson, we'll focus on AWS Simple Queue Service (SQS), a highly scalable and managed message queuing service.

The overall workflow is illustrated below. A user request is placed into a queue, from which ML instances (our LLM worker pods) consume jobs.

This diagram shows the end-to-end flow of asynchronous inference. A request is placed in a queue, processed by workers, with results stored in S3 and notifications sent via SNS.

A critical design consideration in this pattern is what data goes into the message queue. For LLM inference, where prompts and context can be very large, putting the entire payload into the message is inefficient. Instead, we use the Claim Check pattern:

The client first uploads the large payload (e.g., the prompt, documents for RAG) to an object store like S3.
It then sends a small "claim check" message to the queue. This message contains a reference to the data in S3 (e.g., the object key) and any other metadata.
The worker retrieves the lightweight message from the queue, uses the reference to download the full payload from S3, and begins processing.

The following paper formally defines this architecture as the "Claim Check + Semaphore-Queued Inference" (CCSQI) pattern. It's a fantastic resource that aligns perfectly with your goal of mastering the system layer.

[PDF] A Fault- Tolerant Pattern for Distributed OCR/LLM Inference

This paper introduces and formalizes the exact pattern we are building. It provides the "why" behind the architectural choices.

Focus on these key ideas: Introduction (Section 1): Understand the problem statement—why naive synchronous APIs fail for heavy workloads and why in-memory queues are insufficient. Formal Specification (Section 3.2): Pay close attention to the definition of the CCSQI pipeline and the roles of the Ingestion Gateway (our API), Object Store (O), Message Broker (B), and Inference Worker (W). This formalizes the Claim Check pattern. When to Use CCSQI (Section 8.1): This section confirms why this pattern is perfectly suited for LLM inference workloads that are resource-intensive and can be temporally decoupled.

The diagram below shows where the request queue fits within a detailed inference runtime architecture, acting as the entry point for work before it's scheduled for execution on the GPU.

This detailed architecture of an LLM inference runtime highlights the 'Request Queue' as a central component that feeds jobs to the scheduler and downstream hardware.

2. Implementation with Python, Celery, and SQS

Now, let's move from theory to a practical implementation. A popular and powerful stack for this pattern in the Python ecosystem is combining FastAPI (as the web frontend), Celery (as the distributed task queue framework), and SQS (as the message broker).

FastAPI: Receives the user's request and is responsible for publishing the job to the queue.
Celery: Provides the worker processes that consume messages from the queue and execute the corresponding tasks (i.e., run the LLM inference). It handles the complexities of connecting to the broker, fetching messages, and managing task execution.
SQS: The durable, scalable message broker that decouples FastAPI from the Celery workers.

The following video provides an excellent, step-by-step guide to setting up this entire stack. Given your proficiency in Python and infrastructure automation, the code and configuration should be very familiar.

Asynchronous Tasks with FastApi, Celery and SQS

The "Asynchronous Tasks with FastAPI, Celery and SQS" video from the Describly channel is a complete walkthrough of our target architecture.

Follow these segments to build and understand the system: Celery Configuration for SQS (Celery Config): This is the most critical part. Observe how the broker_url is set to sqs:// and how Celery is configured to use SQS as the transport. Note the definition of task_queues—this allows you to route different types of jobs to different queues if needed. Creating the Celery App (Celery App): See how the Celery application instance is created and configured. The broker_connection_retry_on_startup setting is a good example of a reliability feature. Defining a Celery Task (Defining Tasks): Focus on the @celery.task decorator. This is how you designate a Python function as a background job. In our case, this function would contain the LLM inference logic. Pay attention to parameters like ack_late=True, which ensures a task is acknowledged (and removed from the queue) only after it completes successfully. Triggering Tasks from FastAPI (Triggering Tasks): This section shows how the FastAPI endpoint receives a request and uses task.apply_async() to send the job to the Celery queue. This call returns immediately, decoupling the web server from the worker. End-to-End Demonstration (E2E Demo): Watch the full loop in action. A request hits the FastAPI endpoint, a message appears in the SQS queue, and the Celery worker picks it up and processes it.

3. Reliability: Dead-Letter Queues and Retries

A key benefit of using a message broker is improved reliability. What happens if a worker repeatedly fails to process a message, perhaps due to a bug or corrupted data? Without a proper mechanism, the "poison pill" message could be retried indefinitely, wasting resources and blocking other jobs.

The standard solution is a Dead-Letter Queue (DLQ). You configure your main queue so that after a message fails a certain number of times, SQS automatically moves it to a separate DLQ. This gets the problematic message out of the main workflow, allowing other jobs to proceed. You can then inspect the DLQ to diagnose the failures without halting the entire system.

The following video, though implemented in Go, provides an excellent conceptual overview of SQS queue mechanics, including the DLQ configuration. The principles are language-agnostic.

How to Use Message Queues in Golang (SQS, Docker, Terraform)

This video from devtool, "How to Use Message Queues in Golang," clearly explains and demonstrates core SQS reliability features.

Focus on these two parts: Terraform Setup: Observe how the redrive_policy is configured in Terraform. This policy links the main queue to a DLQ and specifies the maxReceiveCount—the number of failed processing attempts before a message is moved to the DLQ. Failure Demo: The presenter intentionally creates a timeout, causing the task to fail. Watch how the message is received again (a retry) and, after the second failure, is moved to the DLQ. This is a perfect demonstration of the redrive policy in action.

4. Preview: Scaling Workers Based on Queue Depth

Now that we have a queue acting as a work buffer, a new question arises: how many worker pods do we need? If the queue is empty, we might not need any. If it has thousands of jobs, we need to scale up our worker deployment to process them quickly.

This is a perfect use case for event-driven autoscaling. Instead of scaling based on CPU or memory, we can scale based on the length of the SQS queue. A tool called KEDA (Kubernetes Event-driven Autoscaling) is designed for exactly this. KEDA integrates with the Kubernetes Horizontal Pod Autoscaler (HPA) to scale deployments based on metrics from various event sources, including SQS.

The following article gives a great overview of this architecture.

Deploy scalable, cost-effective event-driven workloads with Amazon ...

This article introduces KEDA and shows how it's used to scale a deployment based on an SQS queue.

Read the section "Understanding KEDA" to get a high-level idea of what KEDA is and how it works with HPA using ScaledObject resources. Then, examine the YAML manifest in the section "Keda and Karpenter autoscaling in action". Focus on the ScaledObject definition. Notice the scaleTargetRef (which points to our worker deployment), the minReplicaCount: 0 (scale to zero when idle), and the triggers section, which configures the SQS queue to monitor.

This ScaledObject is the key piece we'll be implementing in our next lesson to create a fully elastic, on-demand LLM inference workforce.

Conclusion

In this lesson, you've learned how to architect and implement a scalable, resilient, asynchronous inference system—a critical pattern for any production LLM service. By decoupling your API frontend from your GPU workers with a message broker, you can handle variable loads gracefully and ensure no request is lost during transient failures.

Key Takeaways:

Asynchronous is Key: For long-running tasks like LLM inference, an asynchronous pattern using a message queue is superior to a synchronous API.
Message Brokers Provide Resilience: Services like AWS SQS offer durability (persisting requests) and load leveling (buffering traffic spikes).
The Claim Check Pattern: Keep messages lightweight by storing large payloads in an object store (like S3) and putting only a reference (the "claim check") in the queue.
Celery + SQS: This is a powerful and common combination in the Python world for building distributed task-processing systems.
Reliability with DLQs: Use Dead-Letter Queues to isolate "poison pill" messages that repeatedly fail, preventing them from halting your entire processing pipeline.

In the next lesson, we will build directly on this foundation. We will use KEDA to configure a Horizontal Pod Autoscaler to scale our LLM worker replicas automatically based on the depth of the SQS inference queue.