Cloud Resource Discovery: Polling vs. Event-Driven APIs

Hello! Welcome back to our course on designing security products.

In the previous lesson, we designed the foundational abstraction layer for our Cloud Security Posture Management (CSPM) tool. We established the what: a canonical data model that normalizes resource representations from AWS, Azure, and GCP into a single, unified view. This gives us a consistent target for querying and analysis.

Today, we'll address the critical next question: how and when do we populate and update this data model? This brings us to our learning outcome for this lesson: Design a cloud resource discovery system, comparing polling vs. event-driven approaches via cloud provider APIs.

We will explore two fundamental architectural patterns for discovering cloud resources, analyze their trade-offs, and design a robust, real-world system that combines the best of both.

A Quick Recap: The ETL Foundation

Remember the Extract-Transform-Load (ETL) pattern we discussed?

• Extract: Pull raw configuration data from cloud provider APIs.
• Transform: Normalize this data into our canonical model.
• Load: Store the normalized data in a central database (e.g., a graph database).

The last lesson focused on the "Transform" and "Load" stages. This lesson dives deep into the "Extract" stage, specifically the architectural patterns that trigger and execute the data extraction process.

---

1. The Polling-Based Discovery Architecture

The most straightforward approach to resource discovery is polling. The concept is simple: on a recurring schedule, our system systematically queries the cloud provider APIs to get a complete list of all resources and their configurations.

Think of it as taking a full census of your entire cloud environment at regular intervals (e.g., every 6, 12, or 24 hours).

Architectural Components

A polling-based system requires several key components, as illustrated in the architecture below.

Let's break down the roles of these components in our CSPM context:

1. Scheduler: This component initiates the entire discovery process. It's essentially a distributed cron job. In AWS, this could be an Amazon EventBridge Scheduler. It triggers the scan at a configured interval (e.g., "run every 24 hours at midnight UTC").

2. Orchestrator: A full scan of a large enterprise with thousands of accounts and millions of resources is a complex, long-running, and failure-prone process. The orchestrator manages this complexity. It breaks the job down into smaller tasks (e.g., "scan EC2 in account A," "scan S3 in account B") and manages their parallel execution. AWS Step Functions is a perfect fit for this role, as it can coordinate thousands of parallel tasks, handle retries, and manage state.

3. Workers: These are the components that do the actual work of calling the cloud provider APIs (e.g., ec2:DescribeInstances, s3:ListBuckets). These are typically implemented as serverless functions (AWS Lambda, Azure Functions, Google Cloud Functions) that receive a task from the orchestrator, execute it, and return the results.

4. State Management: The orchestrator needs a database to track the status of all tasks. A key-value store like Amazon DynamoDB is often used to store job state, ensuring that if the orchestrator itself fails, it can resume from where it left off.

Trade-offs of the Polling Model

The AWS blog post "Enhancing multi-account activity monitoring with event-driven architectures" highlights the core issue with polling at scale.

Based on this, let's summarize the trade-offs:

Pros	Cons
Simplicity: The logic is straightforward to understand.	High Latency: Creates a significant "detection blind spot." If a scan runs every 24 hours, a misconfiguration could exist for nearly a full day before being detected.
Guaranteed Completeness: When a scan completes successfully, you have a comprehensive snapshot of your environment at that point in time.	High Cost & Throttling: Generates an enormous number of API calls, which can be expensive and lead to API rate limiting, causing scans to fail or slow down.
Data Consistency: Easier to reason about data consistency, as you're dealing with a complete dataset from a single point in time.	Poor Scalability: The cost and duration of a scan grow linearly (or worse) with the number of resources.
	Misses Ephemeral Resources: Short-lived resources (like a serverless function that runs for a few minutes) that are created and destroyed between scans will never be seen.

---

2. The Event-Driven Discovery Architecture

The polling model pulls data on a schedule. The event-driven model flips this: the cloud provider pushes a notification to our system whenever a resource state changes. This enables a near real-time view of our security posture.

Core Concepts of Event-Driven Architecture (EDA)

Before diving into the cloud security specifics, let's refresh our understanding of EDA. Since you value a first-principles approach, the following video provides a concise overview of the core components.

In our CSPM's event-driven discovery system:

• Producers: The cloud services themselves (e.g., AWS IAM, Azure Networking, GCP Storage).
• Events: Notifications that something happened (e.g., "An IAM policy was updated," "A firewall rule was changed").
• Broker: A managed service that ingests and routes these events (e.g., AWS EventBridge, Azure Event Grid, GCP Pub/Sub).
• Consumers: Our CSPM's processing pipeline, which reacts to the events.

Architectural Components

The article "Event-Driven Cloud Security Architecture: Implementation Guide" provides an excellent blueprint for this model.

Based on the reading, a robust event-driven architecture looks like this:

1. Event Sources: These are the native logging services in each cloud that record all API calls and configuration changes.

   • AWS: AWS CloudTrail, AWS Config
   • Azure: Azure Activity Log, Microsoft Defender for Cloud
   • GCP: Cloud Audit Logs, Security Command Center

2. Event Bus / Router: This is the central nervous system. All event sources are configured to send their logs to a central, managed event bus like AWS EventBridge. This service can then filter and route events to different targets based on rules.

3. Event Processing Pipeline: This is a series of serverless functions that act as our consumers.

   • Collection & Normalization: An initial function receives the raw event, parses it, and transforms it into our canonical data model (as designed in the previous lesson).
   • Enrichment: Subsequent functions might enrich the event with additional context. For example, if an event contains an IP address, an enrichment function could look up that IP in a threat intelligence database.
   • Loading: The final, enriched, and normalized event is then loaded into our central data store.

Trade-offs of the Event-Driven Model

Pros	Cons
Near Real-time Detection: Reduces the detection blind spot from hours to seconds.	Implementation Complexity: Asynchronous systems are inherently more complex to design, debug, and reason about. You must handle out-of-order events, retries, and potential event loss.
High Scalability & Efficiency: A "pay-per-event" model that scales seamlessly with your workload. Far more cost-effective than constant polling.	Eventual Consistency: The state of your inventory is "eventually consistent" with reality. There's a slight delay, and managing this requires careful design.
Captures Ephemeral Resources: Since every creation event is captured, even short-lived resources are recorded in the inventory.	The "Day Zero" Problem: An event-driven system only knows about changes. It doesn't know the state of resources that already exist. How do you perform the initial population of your inventory?
Reduced API Usage: Drastically reduces the number of API calls made to cloud providers, avoiding throttling issues.	Risk of Drift: If your processing pipeline is down or has a bug, you might miss events, causing your inventory's state to "drift" from reality.

---

3. The Hybrid Approach: The Architect's Solution

As you've likely deduced, neither polling nor event-driven architecture is a complete solution on its own. A production-grade CSPM requires a hybrid architecture that combines the strengths of both to mitigate their respective weaknesses.

Addressing the Gaps

1. Solving the "Day Zero" and "Drift" Problems: The pure event-driven model's biggest flaws—initial bootstrapping and state drift—are perfectly solved by the polling model. The hybrid system uses a full polling scan for two purposes:

   • Initial Bootstrap: When a new cloud account is onboarded, the system immediately triggers a full scan to get a complete baseline inventory.
   • Periodic Reconciliation: The system runs a full scan at a much lower frequency (e.g., once every 24 hours) to find and correct any discrepancies that may have arisen due to missed events, ensuring the inventory doesn't drift over time.

2. Dealing with "Thin" vs. "Fat" Events: CloudTrail events are often notification events (or "thin" events). They tell you that a resource changed, but may not contain the complete new configuration in the event payload itself.

Since many cloud events are "thin," our event-driven pipeline must perform a **targeted, on-demand poll**. When it receives an event like `UpdateSecurityGroupRuleDescriptionsEgress`, it uses the resource ID from the event to make a single, precise API call (e.g., `ec2:DescribeSecurityGroups`) to fetch the full, updated configuration of *only that one resource*.

The Hybrid System Design

The final, robust architecture for a cloud resource discovery system is therefore:

1. Onboarding: Run a full polling scan to establish the baseline inventory for a new account.
2. Real-time Updates: Subscribe to real-time event streams (CloudTrail, etc.). When a change event is received:
   a. Parse the event.
   b. Make a targeted API call to fetch the full, current state of the affected resource.
   c. Normalize the data and update the resource in the central inventory.
3. Reconciliation: Run a full polling scan periodically (e.g., daily) to true-up the inventory and correct any potential drift.

This hybrid model gives you the near real-time speed of an event-driven system while retaining the correctness and completeness guarantees of a polling system.

---

Conclusion

In this lesson, we designed the data collection engine for our CSPM, moving from simple concepts to a sophisticated, production-ready architecture.

Key Takeaways:

• Polling-based discovery is simple and provides complete snapshots but is slow, inefficient, and misses ephemeral activity.
• Event-driven discovery offers near real-time updates and is highly scalable but introduces complexity around eventual consistency and requires a mechanism for bootstrapping and reconciliation.
• The optimal architecture is a hybrid model. It uses polling for initial bootstrapping and periodic reconciliation, while leveraging an event-driven pipeline for continuous, low-latency updates. This pipeline often performs targeted, on-demand polling to enrich thin notification events.

Preview of the Next Lesson

We have now designed a complete system for maintaining a near real-time, normalized inventory of all our multi-cloud resources. The inventory is the source of truth. The next logical question is: what do we do with it?

In the next lesson, we will design the core evaluation logic of a policy engine that runs rules against this discovered cloud resource configuration. We will move from data collection to data analysis, building the "brains" of the CSPM that actually identifies security risks.