Skip to main content

Welcome to a Grasp lesson

Create a free account to keep studying and create your own study path.

Lesson 22

Automated Alerting for Anomaly Detection

Recommended resources

Introduction

In the previous lesson, you engineered a crucial piece of infrastructure: an execution monitoring dashboard. This gives you a reactive, historical view of your system's health, allowing you to analyze performance and reliability over time. It's the foundation of professional-grade observability.

However, a dashboard requires you to pull information. A truly robust system pushes critical information to you when your attention is needed. Today, we build that proactive layer. The objective is to create automated alerting workflows that trigger notifications based on detected anomalies or threshold breaches.

We will move beyond simple "alert on every failure" logic, which quickly leads to alert fatigue—the single biggest reason developers ignore notifications. Instead, you will design and implement an intelligent alerting engine that understands context, respects thresholds, and delivers high-signal, actionable alerts. This is a system that a client will value because it finds problems before they do.


The Principles of an Effective Alerting Strategy

Before we build, we must design. A flood of meaningless alerts is worse than no alerts at all. The goal is to create a system that signals genuine problems that require intervention.

To understand the professional mindset behind this, let's review the core principles of a production-ready alerting strategy.

n8n Workflow Design Patterns: Monitoring and Alerting Strategies

The following article, 'n8n Workflow Design Patterns', contains a section dedicated to Monitoring and Alerting. It perfectly outlines the strategic thinking required before writing any code.

Read the section titled 'Monitoring and Alerting Strategies'. Focus on three key concepts: The distinction between 'Workflow Health', 'System Health', and 'Business Metrics'. The difference between 'Critical Alerts' and 'Warning Alerts'. The strategies for 'Alert Fatigue Prevention'. This will provide the framework for the system you are about to build.

The key takeaway is that we need to define rules. Our alerting workflow will act as a "smart router," analyzing data from our logs and deciding if an issue is critical, a warning, or just noise.

This pattern of analyzing and routing alerts is fundamental in system operations. The workflow we build today is a practical implementation of the principles used in more advanced systems, like the incident response automation described in the "Automated Incident Response with n8n & Monitoring Tools" article from Bubobot, which you can review later for more advanced ideas.


Practical Task: Engineer the Proactive Alerting Engine

Our data source is the executions_log Google Sheet you created in the last lesson. We will build a new, scheduled workflow that periodically analyzes this log data.

Phase 1: Define Alerting Thresholds

This is a system design exercise. Based on the data you're collecting (timestamp, workflowName, status, etc.), define two specific, measurable, and actionable alert thresholds.

Your Task:
Define one "Critical" and one "Warning" threshold. Document them. For example:

  • Critical Alert Rule: Trigger if the total number of FAILURE statuses in the last hour is greater than 5. This indicates a potential system-wide outage.
  • Warning Alert Rule: Trigger if the failure rate for any single workflow exceeds 20% over the last 24 hours. This points to a problem with a specific process, not necessarily the entire system.

These are just examples. Define rules that you believe provide a good balance between sensitivity and noise reduction.


Phase 2: Build the Alerting Workflow

Now, let's construct the workflow that enforces these rules.

Your Task:

  1. Create a new n8n workflow named System - Anomaly Detector.

  2. Trigger: Use a Schedule node. For now, set it to run every hour.

  3. Read Log Data: Add a Google Sheets node with the operation Get Rows. Configure it to read all data from your executions_log sheet.

  4. Process and Calculate Metrics: This is where the core logic resides. Add a Code node (previously Function node). Your background in development makes this the most efficient way to process the data.

    Use the following JavaScript to get started. Your task is to complete the logic to calculate the metrics needed for the thresholds you defined in Phase 1.

    // This code runs inside the n8n Code node.
    // 'items' contains the data from the Google Sheets node.
    const logs = items.map(i => i.json);
    
    const now = new Date();
    const oneHourAgo = new Date(now.getTime() - 60 * 60 * 1000);
    const twentyFourHoursAgo = new Date(now.getTime() - 24 * 60 * 60 * 1000);
    
    // --- YOUR LOGIC HERE ---
    
    // 1. Filter logs for the last hour
    const recentFailures = logs.filter(log => {
      const logDate = new Date(log.timestamp);
      return log.status === 'FAILURE' && logDate > oneHourAgo;
    });
    const recentFailureCount = recentFailures.length;
    
    // 2. Calculate failure rate per workflow for the last 24 hours
    const logsLast24h = logs.filter(log => new Date(log.timestamp) > twentyFourHoursAgo);
    const workflowStats = {};
    
    for (const log of logsLast24h) {
      if (!workflowStats[log.workflowName]) {
        workflowStats[log.workflowName] = { successes: 0, failures: 0 };
      }
      if (log.status === 'SUCCESS') {
        workflowStats[log.workflowName].successes++;
      } else if (log.status === 'FAILURE') {
        workflowStats[log.workflowName].failures++;
      }
    }
    
    const workflowsExceedingThreshold = [];
    for (const wfName in workflowStats) {
      const stats = workflowStats[wfName];
      const total = stats.successes + stats.failures;
      const failureRate = total > 0 ? (stats.failures / total) : 0;
      
      // Assumes a 20% threshold as an example. Adjust to your rule.
      if (total > 5 && failureRate > 0.20) { 
        workflowsExceedingThreshold.push({
          name: wfName,
          failureRate: (failureRate * 100).toFixed(2) + '%',
          failures: stats.failures,
          totalExecutions: total
        });
      }
    }
    
    // --- END YOUR LOGIC ---
    
    // Output the calculated metrics for the next nodes to use.
    return [{
      json: {
        criticalFailureCount: recentFailureCount,
        workflowsToWarn: workflowsExceedingThreshold
      }
    }];
    
  5. Route Based on Thresholds: Add a Switch node connected to the Code node. Configure it to route based on the output.

    • Routing Rule 1 (Critical):
      • Condition: Number > {{ $json.criticalFailureCount }} > 5 (adjust to your rule).
      • Output: critical_alert
    • Routing Rule 2 (Warning):
      • Condition: Is Not Empty > {{ $json.workflowsToWarn }}.
      • Output: warning_alert
  6. Construct and Send Alerts: You'll now create the notification actions. We'll use email as the channel. The mechanics of setting up the Gmail node and composing a message are shown well in the video resource below. While the video alerts on every error, we will apply its techniques to our specific threshold outputs.

Master n8n Error Workflows: Get Instant Email Alerts on Failures

This video from Leon van Zyl, 'Master n8n Error Workflows', demonstrates how to dynamically construct and send an email alert. We'll adapt this method for our threshold-based alerts.

Watch the segment from 06:31 to 09:16. Pay close attention to how he uses the Expression Editor to combine static text with dynamic data from a trigger to create a detailed email subject and body. You will use the same technique with data from your Code node.

  • Critical Alert: Connect a Gmail node to the critical_alert output of the Switch node.
    • Configure your Gmail credentials if you haven't already.
    • Subject: CRITICAL ALERT: System-Wide Failures Detected
    • Body (HTML): Craft a message that clearly states the problem.
      <h1>Critical System Alert</h1>
      <p>A high number of workflow failures has been detected, indicating a potential system-wide issue.</p>
      <p><strong>Failures in last hour:</strong> {{ $json.criticalFailureCount }}</p>
      <p>Immediate investigation is required.</p>
      <p><em>Alert triggered at: {{ new Date().toLocaleString('en-US', { timeZone: 'UTC' }) }} UTC</em></p>
      
  • Warning Alert: Connect another Gmail node to the warning_alert output. Since workflowsToWarn is an array, you'll need to process it. The easiest way is to use an Items Lists node before the Gmail node to aggregate the array into a single item, or simply reference the first item {{ $json.workflowsToWarn[0] }} in your email if you want to alert per-workflow. For a summary alert, we'll use a simple loop inside the email body expression.
    • Subject: WARNING: High Failure Rate Detected for Workflows
    • Body (HTML Expression):
      const workflows = $json.workflowsToWarn;
      let body = '<h1>Workflow Performance Warning</h1><p>The following workflows have exceeded their failure rate threshold in the last 24 hours:</p><ul>';
      for (const wf of workflows) {
        body += `<li><strong>${wf.name}</strong>: ${wf.failureRate} failure rate (${wf.failures}/${wf.totalExecutions} executions).</li>`;
      }
      body += '</ul><p>Please review for potential issues.</p>';
      return body;
      

Your final workflow should look conceptually like this:

n8n Workflow Monitoring and Alerting Concept
This diagram illustrates the concept: data is gathered and analyzed, and when an anomaly is detected (as shown by the alert icon), a notification is triggered. Our workflow implements this with a Schedule trigger, Google Sheets reader, Code node for analysis, a Switch for routing, and Gmail for alerting.

Phase 3: Test the Alerting System

A monitoring system that hasn't been tested is a broken system.

Your Task:

  1. Trigger a Critical Alert: Go to your executions_log Google Sheet. Manually add 6+ rows with a FAILURE status and a timestamp within the last hour.
  2. Go to your System - Anomaly Detector workflow in n8n and click Test workflow. Verify that it takes the critical_alert path and that you receive the corresponding email.
  3. Trigger a Warning Alert: Clear the test data. Now, add ~10 rows for a single workflow (e.g., System - Health Check), with 3 marked FAILURE and 7 SUCCESS. Ensure their timestamps are within the last 24 hours.
  4. Run the test again. Verify it now takes the warning_alert path and you receive the warning email detailing the specific workflow's high failure rate.

Conclusion

You have successfully designed and deployed a proactive, threshold-based alerting engine. This is a significant step up from basic error logging and represents a core competency of a systems engineer. You have built a system that respects your attention by distinguishing between noise and signal, using rules to escalate issues intelligently.

Key Takeaways:

  • Proactive alerting is about thresholds, not volume. The goal is to prevent alert fatigue by only notifying on events that breach pre-defined rules.
  • The architecture of a scheduled analysis workflow (Schedule -> Get Data -> Analyze -> Route -> Notify) is a powerful and reusable pattern for monitoring.
  • Using a Code node for data manipulation is highly efficient and leverages your existing development skills for complex calculations like time-window filtering and rate calculation.
  • Actionable alerts are critical. Your notifications must provide enough context for the recipient to immediately understand the problem's nature and severity.

In our next lesson, we will focus on what to do after an alert identifies a problem. We will shift from observation to action by learning how to optimize workflow performance by consolidating data manipulation nodes into single, efficient expressions. This will equip you to fix the very performance bottlenecks your new monitoring system can now detect.