Introduction
In our last session, you engineered a Dead-Letter Queue (DLQ), creating a critical safety net to prevent data loss from terminal processing failures. This completed a key piece of our resilient system: the ability to retry transient errors and safely capture non-recoverable ones.
However, the power to retry and reprocess introduces a new, critical risk: duplicate operations. What happens if a webhook is sent twice by the source system? Or if a network timeout causes your client to retry a request that actually succeeded? Without a protective mechanism, you could create duplicate customer records, send redundant notifications, or charge a credit card twice.
This lesson directly addresses that challenge. Your objective is to apply idempotency patterns to webhook-triggered workflows to prevent duplicate processing of events. Idempotency ensures that performing an operation multiple times has the same effect as performing it once. For systems that must be reliable, this is not an optional feature; it is a core architectural requirement.
Part 1: The Architectural Mandate for Idempotency
Before we build, we must understand the "why". A common mistake is to pack business logic directly into the webhook-receiving endpoint. This creates brittle, slow, and unreliable systems. A production-grade webhook receiver has only three jobs:
- Verify the request's authenticity (security).
- Acknowledge receipt of the event immediately.
- Queue the event for later, asynchronous processing.
This separation of concerns is fundamental. The sender of the webhook only needs to know their message was received, not whether your entire business process succeeded. Duplicate events are handled in the asynchronous processing stage.
Stop Doing Business Logic in Webhook Endpoints. I Don't Care ...
The article 'Stop Doing Business Logic in Webhook Endpoints' provides a sharp, real-world argument for this architectural separation. It perfectly frames the problem we are solving.
Read the sections 'The Setup' and 'How do you handle duplicates then?'. Focus on the clear distinction between the webhook endpoint's responsibility (acknowledging receipt) and the worker's responsibility (handling idempotency and processing). The author's core argument in the 'How do you handle duplicates then?' section is precisely the pattern we will implement.
This architecture is visually represented in professional system diagrams. Notice how idempotency is a distinct, critical step in the processing pipeline, separate from the initial event reception.

To solidify the core concept, let's get a formal definition.
Idempotency - What it is and How to Implement it
The video 'Idempotency - What it is and How to Implement it' by Alex Hyett offers a clear and concise technical definition, which is essential for our work.
Watch from the beginning to 05:23. Pay close attention to: The formal definition of idempotency. Why POST requests are typically not idempotent and why this is a problem. The concept of the 'Idempotency-Key' as the primary method for making non-idempotent operations safe to retry.
Part 2: Practical Implementation of an Idempotency Check
We will now implement an idempotency pattern within your [WIP] Webhook Lead Intake workflow. The core principle is known as a local deduplication step.
The pattern is as follows:
- Receive a webhook containing a unique identifier for the event (e.g.,
event_id,transaction_id). We will call this our Idempotency Key. - Before any processing, query a persistent datastore to check if this Idempotency Key has been seen before.
- If the key exists, the event is a duplicate. Acknowledge receipt with a
200 OKstatus and terminate the workflow. - If the key does not exist, it's a new event. Immediately write the key to the datastore to "claim" it, then proceed with the rest of the workflow logic.
For our implementation, we will use a Google Sheet as our persistent datastore. This is a simple, no-cost way to simulate a key-value store like Redis or DynamoDB.
Your Task: Engineer an Idempotent Webhook Receiver
Scenario: The service that sends leads to your [WIP] Webhook Lead Intake workflow occasionally sends duplicate webhook events. The webhook payload contains a unique identifier for each submission, located at body.responseId. We will use this as our idempotency key.
1. Prepare the Idempotency Datastore:
- Create a new Google Sheet named
[SYS] Idempotency Store. - Create a sheet/tab named
ProcessedEvents. - Define the following headers in the first row:
EventID,Timestamp,Status.
2. Modify the [WIP] Webhook Lead Intake Workflow:
This entire block of logic must be inserted immediately after your Webhook trigger node and before any other processing steps (like your existing retry logic).
-
Step A: Check for Duplicates
- Add a Google Sheets node. Set the Operation to
Get Many Rows. - Connect to your
[SYS] Idempotency Storesheet. - In the 'Filters' section, add a filter where the column
EventIDmust equal the idempotency key from the webhook.- Value:
={{ $json.body.responseId }}
- Value:
- Rename this node to
Check for Existing EventID.
- Add a Google Sheets node. Set the Operation to
-
Step B: Route Based on Existence
- Add an IF node immediately after
Check for Existing EventID. - Set a condition to check if the previous step found a matching event. A simple way is to check the number of items returned.
- Value 1:
={{ $items.length }} - Operation:
Larger Than - Value 2:
0
- Value 1:
- Rename this node to
Is Duplicate?.
- Add an IF node immediately after
-
Step C: Handle the Duplicate Path (IF node's
trueoutput)- Add a Respond to Webhook node.
- Set the
Response Codeto200. - Set the
Response DatatoJSONand enter the following in theJSONfield:{ "status": "accepted", "info": "Duplicate event ignored. EventID: {{ $json.body.responseId }}" } - This path ends here. You have successfully acknowledged and ignored the duplicate.
-
Step D: Handle the New Event Path (IF node's
falseoutput)- Add a new Google Sheets node. Set the Operation to
Append or Update Rows. - Connect to the
[SYS] Idempotency Storesheet. - Key Column:
EventID. This tells the node how to identify rows for updating. - Map the columns to record the new event:
EventID:={{ $json.body.responseId }}Timestamp:={{ $now.toFormat("yyyy-MM-dd HH:mm:ss") }}Status:Processing
- Rename this node to
Claim EventID. - Crucially, connect the output of
Claim EventIDto the start of your existing workflow logic (i.e., the first node of your retry/DLQ pipeline).
- Add a new Google Sheets node. Set the Operation to
This implementation ensures that only new, unique events proceed to the resource-intensive parts of your workflow.
For further reference on this pattern of using a unique event ID, review this specific segment.
n8n Webhook Security: A Deep Dive Into Real Workflow Protection
The video 'n8n Webhook Security' by Bart Slodyczka, while focused on security, contains a perfect explanation of using an 'event ID' for deduplication, which is exactly our idempotency pattern.
Watch from 26:50 to 28:12. The speaker explains the 'Event ID' concept: taking a unique ID from a payload, logging it, and checking against that log to prevent duplicate processing. This directly mirrors the logic you are building with Google Sheets.
Deliverables
- Idempotency Store Screenshot: A screenshot of your
[SYS] Idempotency StoreGoogle Sheet showing the headers (EventID,Timestamp,Status) and at least one test entry. - Workflow Diagram Screenshot: A screenshot of your
[WIP] Webhook Lead Intakeworkflow canvas. It must clearly show the new logic block after the Webhook trigger:Check for Existing EventID->Is Duplicate?-> and the two resulting branches (Respond to WebhookandClaim EventID).
Evaluation Criteria
- Architectural Correctness: Is the idempotency check placed at the very beginning of the workflow, before any other significant processing?
- Deduplication Logic: Does the
IFnode correctly route based on whether theEventIDwas found in the Google Sheet? - Graceful Rejection: Does the "duplicate" path correctly use a
Respond to Webhooknode to send a200 OKand terminate gracefully? - State Management: Does the "new event" path correctly "claim" the
EventIDby writing it to the Google Sheet before proceeding to the main business logic?
Conclusion
You have now implemented an idempotency pattern, elevating your workflow from a simple automation to a resilient, production-grade process. By separating acknowledgment from processing and implementing a deduplication check, you have protected your system against one of the most common and damaging failure modes in distributed systems. This ability to make operations safely repeatable is a hallmark of a systems architect, not a workflow beginner.
Key Takeaways:
- Idempotency is Non-Negotiable: For any critical operation triggered by an external event or subject to retries, idempotency is mandatory.
- The Idempotency Key is the Core Pattern: Identify a unique value in the incoming data and use it to track processing state.
- Acknowledge First, Process Later: Webhook endpoints should be fast and simple. Heavy lifting and stateful checks belong in an asynchronous worker process.
- State Must Be Persistent: Your idempotency check is only as reliable as its datastore. While we used Google Sheets, in high-volume scenarios, this would be a dedicated database or key-value store like Redis.
In our next lesson, we will formalize the concept of synchronous vs. asynchronous processing. The pattern you built today is inherently asynchronous. We will explore when to choose this pattern versus a synchronous one, and how to justify that architectural decision based on client requirements, system constraints, and performance trade-offs.