Skip to main content

Welcome to a Grasp lesson

Create a free account to keep studying and create your own study path.

Lesson 15

Asynchronous Queues for Rate-Limited API Processing

Recommended resources

Introduction

In our previous lesson, you mastered handling API pagination, enabling you to design workflows that can retrieve complete datasets of any size. However, fetching hundreds or thousands of records in a rapid, synchronous loop introduces a critical point of failure: API rate limits. A single "429 Too Many Requests" error can halt your entire data pull, rendering the workflow unreliable for production use.

This lesson directly confronts that challenge. Your objective is to implement an asynchronous queue pattern to process API requests in batches while respecting service rate limits. You will learn to decouple the generation of tasks from their execution, moving from a brittle, monolithic loop to a resilient, scalable system. This is a foundational pattern for any serious automation engineer, as it ensures your workflows are both robust and well-behaved citizens of the API ecosystems they interact with.


Part 1: The Architectural Solution: Queueing Systems

When a synchronous process fails, the entire chain breaks. If you're fetching 1,000 records and the 501st call fails due to a rate limit or transient network error, the remaining 499 records are never processed. The professional solution is to not process items in a single, long-running execution. Instead, we break the problem down:

  1. Producer: A process that generates a list of "jobs" to be done (e.g., a list of 1,000 user IDs to fetch).
  2. Queue: A message broker (like Redis or RabbitMQ) that holds these jobs.
  3. Consumer (Worker): One or more separate processes that pull jobs from the queue one by one and execute them.

This asynchronous architecture provides three major advantages:

  • Resilience: If a single job fails, it only affects that one item. The other 999 jobs continue processing independently. The failed job can be logged or retried later.
  • Scalability: You can add more workers to process the queue in parallel, dramatically increasing throughput.
  • Rate Limit Control: The producer can add jobs to the queue at a controlled rate, or the workers can be configured to pull jobs at a pace that respects API limits.

To understand the "gold standard" for this architecture in a production n8n environment, let's review how n8n's built-in Queue Mode works.

Configuring queue mode | n8n Docs

The official n8n documentation provides the definitive overview of its queue-based architecture. This will give you the formal model of how a scalable n8n system is structured.

Read the sections 'How it works', 'Configuring workers', 'Start Redis', 'Running n8n with queues', and 'Configure worker concurrency'. Focus on the roles of the main instance (producer), Redis (queue), and the workers (consumers). Pay close attention to the flow diagram.

The article "n8n Performance Optimization for High-Volume Workflows" further contextualizes why this is critical for high-volume systems.

n8n Performance Optimization for High-Volume Workflows

This article explains the engineering principles behind queueing and its importance for performance.

Read the sections 'Queue architecture and backpressure' and 'Batching versus single-item processing'. Note the concepts of backpressure and how batching relates to throughput and error handling.

This diagram illustrates the full architecture of a production-ready, scalable n8n deployment. This is the target state for a high-load freelance project.

n8n Queue Mode Architecture with Monitoring
This diagram shows a complete production n8n deployment using queue mode. It features Traefik for routing, a main n8n instance, multiple n8n workers, Redis for the job queue, PostgreSQL for storage, and an observability stack for monitoring.

Part 2: Simulating a Queue on the Free Tier

While a full-scale deployment with dedicated workers and a Redis instance is the ideal, the principles of queueing can be implemented within a standard n8n instance using a webhook-based pattern. This is a highly effective technique for your current setup and a common professional pattern when a full queue mode isn't available or necessary.

The pattern consists of two workflows:

  1. Workflow A: The Producer

    • Fetches the initial list of items to be processed (e.g., all users from the reqres.in API).
    • Loops through this list.
    • Inside the loop, it does not perform the heavy processing. Instead, it makes an HTTP Request to call the webhook of Workflow B, passing the data for a single item (e.g., a user ID).
    • It includes a Wait node in the loop to explicitly throttle the creation of new "jobs," thereby controlling the rate of API calls.
  2. Workflow B: The Consumer

    • Is triggered by a Webhook node.
    • Receives the data for one item from Workflow A.
    • Performs the actual API call and processing for that single item.
    • Saves the result (e.g., to a Google Sheet).

Each call to Workflow B's webhook creates a separate, independent execution in n8n. This effectively simulates a job queue, giving you the resilience and control needed for batch processing.

This sequence diagram illustrates the logical flow, where "n8n Main" is your producer workflow and "n8n Worker(s)" are the individual executions of your consumer workflow.

Asynchronous Queue Pattern with RabbitMQ, Redis, and n8n
This sequence diagram shows an external producer (your main workflow) enqueuing a job in a Redis queue. An n8n worker (your webhook workflow) then pulls and executes the job. This visualizes the decoupling of job creation and execution.

Part 3: Practical Task - Implement the Asynchronous Processor

Scenario: You need to build a system that fetches detailed profiles for a list of users. The source API for user details has a strict rate limit of 5 requests per 10 seconds. You must design a system that can process an arbitrary number of users without being blocked.

Your Task: Using the two-workflow pattern, build an asynchronous batch processor that:

  1. Retrieves the list of 12 users from reqres.in.
  2. For each user, calls a separate API endpoint to get their detailed profile.
  3. Respects the rate limit.
  4. Logs the results to a Google Sheet.

Execution Steps:

1. Set up the Consumer Workflow (User_Detail_Processor)

  • Objective: To process a single user ID, fetch its details, and log the result.
  • Create a new workflow and name it User_Detail_Processor.
  • Trigger: Add a Webhook node. When the dialog appears, copy the Test URL. You will need it for the Producer workflow. Keep this node's settings open for now.
  • Fetch Details: Add an HTTP Request node.
    • Set the URL to https://reqres.in/api/users/{{ $json.body.id }}. This expression will dynamically insert the user ID received by the webhook.
  • Log to Sheet: Add a Google Sheets node (Append or Update).
    • Authenticate and select a Sheet and a worksheet (e.g., "User Details").
    • Map the columns: ID from {{ $json.body.id }}, Email from {{ $node["HTTP Request"].json["data"]["email"] }}, FirstName from {{ $node["HTTP Request"].json["data"]["first_name"] }}, etc.
  • Test the Consumer:
    • Go back to the Webhook node and click "Listen for Test Event".
    • In a separate terminal or API tool (like Postman/Insomnia), send a POST request to your webhook's Test URL with a JSON body like: {"id": 2}.
    • Verify the workflow runs successfully and adds user 2's data to your Google Sheet.
    • Save and activate this workflow. Copy the Production URL from the webhook node.

2. Set up the Producer Workflow (Batch_Request_Producer)

  • Objective: To get the list of all users and send a "job" for each one to the consumer workflow at a controlled rate.
  • Create a new workflow and name it Batch_Request_Producer.
  • Trigger: Use a Manual trigger.
  • Get User List: Add an HTTP Request node configured as in the previous lesson to fetch all 12 users from https://reqres.in/api/users using pagination. Ensure its JSON/RAW Path for Items is set to data.
  • Loop through Users: The HTTP Request node will output 12 separate items. The subsequent nodes will now run once for each item.
  • Call Consumer Webhook: Add a new HTTP Request node after the first one.
    • Set Method to POST.
    • Set URL to the Production URL of your User_Detail_Processor webhook.
    • Under Body Content Type, select JSON.
    • In the Body Parameters, add a parameter named id with the value {{ $json.id }}. This expression takes the id from the current user item in the loop and sends it to the consumer.
  • Throttle the Requests: Add a Wait node after the webhook call.
    • The rate limit is 5 requests per 10 seconds, which means we can make one request every 10 / 5 = 2 seconds.
    • Set Wait Time (seconds) to 2. This ensures you never exceed the rate limit.

3. Execute and Verify

  • Execute the Batch_Request_Producer workflow.
  • Observe its execution. You should see it loop 12 times, pausing for 2 seconds on each iteration.
  • Go to the User_Detail_Processor workflow's execution log. You will see 12 separate successful executions.
  • Check your Google Sheet. It should now be populated with the details for all 12 users.

Deliverables

  1. System Diagram: Create a simple box-and-arrow diagram illustrating your two-workflow solution. Label "Workflow A (Producer)" and "Workflow B (Consumer)", showing how data flows between them and the external services (reqres.in API, Google Sheets).
  2. Producer Configuration (Screenshot): A screenshot of your Batch_Request_Producer workflow showing the four key nodes: HTTP Request (Get List), HTTP Request (Call Webhook), and the Wait node.
  3. Consumer Execution Log (Screenshot): A screenshot of the User_Detail_Processor's execution list showing at least 5-10 successful, independent executions triggered by the producer.

Evaluation Criteria

  • Architectural Soundness: Your diagram must correctly depict the producer/consumer pattern and data flow.
  • Correct Implementation: The producer must correctly loop and call the consumer's webhook, and the consumer must correctly process the data.
  • Rate Limit Compliance: The presence and correct configuration of the Wait node (2 seconds) is mandatory.
  • Successful End-to-End Execution: The final Google Sheet must contain the data for all 12 users, proving the system worked as designed.


Conclusion

You have successfully engineered an asynchronous processing system. By separating the producer and consumer logic and introducing a controlled delay, you have built a workflow that is not only resilient to single-point failures but also compliant with external API rate limits. This pattern is a significant step up from simple, linear workflows and is fundamental to building scalable, production-grade automations.

Key Takeaways:

  • Decouple to Scale: Separate task generation (producer) from task execution (consumer) to build resilient systems.
  • Webhooks as a Queue: A webhook-triggered workflow is an effective way to process jobs asynchronously, with each trigger creating an independent execution.
  • Throttle with Wait: Use the Wait node to explicitly control the rate of outgoing requests to respect API limits.
  • Design for Failure: This architecture inherently contains failures. If one consumer execution fails, it doesn't stop the others.

In this lesson, we created a system that spawns many independent executions. What happens when one of them fails? In our next lesson, we will focus on implementing robust try-catch blocks for error handling using the Error Trigger node, giving you the tools to manage and react to these inevitable failures gracefully.