Designing a Findings Pipeline for Automated Remediation

Hello! Welcome to the next lesson in our course on architecting security products.

In our previous lesson, we designed a sophisticated hybrid drift detection system. This system provides our Cloud Security Posture Management (CSPM) tool with a real-time stream of findings, alerting us the moment a resource's configuration violates one of our security policies.

This is a huge step forward, but it introduces a new set of system design challenges. A large cloud estate can generate thousands of findings per day. How do we process this firehose of information without overwhelming security teams? How do we distinguish a critical, five-alarm fire from a minor compliance issue? And most importantly, how can we build a system that not only detects problems but also safely fixes them?

Today, we will design the answer to these questions. Our learning outcome is to: Design a findings pipeline including severity classification, notification routing, and a secure auto-remediation framework. We will architect the data pipeline that takes a raw finding as input and orchestrates its entire lifecycle, from initial triage to final resolution.

---

1. High-Level Architecture of a Findings Pipeline

At its core, a findings pipeline is a specialized event-processing system. Your experience with data pipelines will be directly applicable here. The goal is to take a raw finding event, enrich it, make decisions based on its content, and route it to the correct destination—which could be a human, a ticketing system, or an automated remediation bot.

Let's look at a real-world example of this architecture.

As the video illustrates, a modern findings pipeline consists of three main stages:

1. Ingestion & Normalization: Collecting findings from various detectors and converting them into a single, standardized format.
2. Processing & Routing: Enriching the finding with business context, classifying its severity, and routing it based on predefined rules.
3. Disposition & Remediation: Delivering notifications to the appropriate channels and, when possible, automatically resolving the underlying issue.

Let's use this architectural pattern as our blueprint.

---

2. Pipeline Stage 1: Ingestion and Normalization

The pipeline begins the moment our drift detection system from the last lesson generates a finding.

• Input: A raw event indicating a policy violation for a specific resource.

• Decoupling: The first architectural principle is to decouple the finding producers (our scanners and drift detectors) from the consumers (the rest of the pipeline). As with any robust distributed system, we use a message queue for this. Findings are published to a central topic in a service like Apache Kafka or AWS SQS. This provides buffering against traffic spikes and ensures reliable delivery.

• Normalization: Different detection sources may have different output formats. The first service in our pipeline consumes from the queue and normalizes each finding into a canonical schema for our CSPM. This ensures that downstream components work with a consistent data structure. This is analogous to the AWS Security Finding Format (ASFF) mentioned in the video.

A normalized finding might look like this:

{
  "findingId": "uuid-v4-...",
  "source": "DriftDetector",
  "resourceId": "arn:aws:s3:::my-production-bucket",
  "policyId": "S3-001",
  "policyDescription": "S3 buckets should not allow public read access",
  "status": "OPEN",
  "firstSeen": "2023-10-27T10:00:00Z",
  "lastSeen": "2023-10-27T10:00:00Z",
  "details": { ... }
}

---

3. Pipeline Stage 2: Processing and Routing

Once a finding is normalized, we need to decide how important it is and who needs to know about it. This stage acts as the intelligent brain of the pipeline.

Severity Classification

A static severity on a policy (e.g., "S3 public access is HIGH severity") is insufficient. The true risk depends on context. Our processing layer must enrich the finding with business context to calculate a dynamic severity score.

It should ask questions like:

• Is the resource tagged with env=prod or env=dev?
• Does the resource have a data classification tag like pii=true?
• Who is the owner of the application associated with this resource?

By enriching the finding with this data (often from a CMDB or resource tagging system), a public S3 bucket in a development account might be classified as Medium, while the exact same finding for a production bucket containing PII would be classified as Critical.

Notification Routing

With a classified finding, we now face a classic notification system design problem. How do we deliver this information to the right entity?

As the article suggests, we can design a Notification Processor that uses a rules engine to determine the delivery "channel". The logic would look something like this:

• IF severity is Critical AND resource_owner is Team-Alpha, THEN create a PagerDuty incident for Team-Alpha's on-call rotation.
• IF severity is High AND resource_owner is Team-Beta, THEN create a Jira ticket and post a message to the #team-beta-security Slack channel.
• IF severity is Low, THEN add the finding to a weekly digest email sent to the resource owner.

The Chain of Responsibility pattern is an excellent fit here. We can construct a pipeline of handlers (e.g., SeverityHandler -> OwnerLookupHandler -> ChannelRoutingHandler) to process each finding.

---

4. Pipeline Stage 3: Secure Auto-Remediation

This is the most powerful—and most dangerous—part of our system. Automatically fixing a misconfiguration can prevent a breach, but a flawed automation can cause a major outage. Therefore, we must design a secure framework, not just a collection of scripts.

Drawing from these best practices, our Secure Auto-Remediation Framework must include four key components.

1. Playbook Library

A playbook is a version-controlled piece of code that defines a precise, tested procedure for fixing a specific finding. Examples:

• Finding: EC2 Security Group open to 0.0.0.0/0 on port 22
• Playbook: remediate-ssh-access.py
  • Identifies the overly permissive rule.
  • Replaces it with a rule allowing access only from the corporate VPN IP range.
  • Adds a tag remediated-by=cspm to the security group.

These playbooks are our unit of automation. They must be peer-reviewed, tested in staging environments, and treated as production code.

2. Execution Engine

This is the mechanism that runs the playbooks. As we saw in the AWS video, a common and effective pattern is to use a serverless function orchestrator:

• Amazon EventBridge acts as the router.
• A rule for a specific finding (e.g., policyId == 'S3-001') triggers an AWS Step Function or Lambda Function that encapsulates the playbook logic.

3. Safety Controls

This is what makes the framework secure. Blindly running a script is reckless. A secure framework provides guardrails.

• Idempotency: Remediation actions must be idempotent. Running a fix twice should have the same result as running it once. This prevents race conditions or duplicate events from causing problems.
• Dry-Run Mode: The engine must support a "dry run" or "audit" mode where it logs the actions it would take without actually performing them.
• Approval Gates: For high-risk actions (e.g., modifying a production database network policy), the framework shouldn't act autonomously. It should execute the plan up to the point of change, then pause and create a ticket (e.g., in Jira or ServiceNow) with "approve" and "reject" buttons for a human operator.
• Circuit Breaker: If a specific playbook fails repeatedly (e.g., due to a change in a cloud provider API), the framework must automatically "open the circuit"—disable that automation and escalate to a human—to prevent cascading failures.
• Automatic Rollback: If a remediation action is applied but post-verification checks fail, the engine should attempt to automatically revert the change to its last known good state.

4. The Feedback Loop

The pipeline doesn't end when a fix is applied. The remediation engine must call back to the CSPM's API to update the finding's status.

• Execution successful? The finding status is updated from OPEN to RESOLVED.
• Execution failed? The status is updated to REMEDIATION_FAILED and an alert is sent to the security team.

This feedback loop is critical for tracking the efficacy of the automation and ensuring auditability.

---

Conclusion

In this lesson, we have designed a comprehensive findings pipeline, moving from raw detection to intelligent and safe resolution. We've transformed our CSPM from a simple reporting tool into an active defense system.

Key Takeaways:

• Pipeline Architecture: A findings pipeline follows a standard data processing pattern: Ingest/Normalize, Process/Route, and Dispose/Remediate.
• Context is King: Severity classification must be dynamic, enriching findings with business context to accurately assess risk.
• Routing as a Design Pattern: Use established patterns like Chain of Responsibility to build a flexible and scalable notification routing engine.
• Remediate with Guardrails: Auto-remediation is powerful but requires a robust framework built on safety principles like playbooks, dry-runs, approval gates, circuit breakers, and feedback loops.

Preview of the Next Lesson

We have spent the last several lessons deep in the cloud, architecting systems that monitor configuration and posture. Now, we are going to shift our focus dramatically—from the control plane to the data plane, and from the data center to the network edge.

Our next module is "Architecting Unified Edge Security Platforms." In the first lesson, we will tackle the learning outcome: Architect a unified security platform at the network edge, integrating WAF, Bot Management, and DDoS mitigation. We will explore the unique architectural challenges of building systems that must inspect and make decisions on billions of user requests in real time, all over the world.