RASP Data Output & Integration with SIEM/WAF

Hello! Welcome back to our course on architecting security products.

In our previous lessons, we've meticulously designed the core of a Runtime Application Self-Protection (RASP) agent. We've architected its instrumentation, its context-aware detection engine, and critically, the performance optimization strategies—like selective instrumentation and adaptive sampling—that make it viable in a production environment.

An agent that detects an attack but tells no one is a silent observer, not a protector. This brings us to the final, crucial step in the agent's workflow: communication. Our learning outcome for this lesson is to design the data output and integration points for a RASP agent to interoperate with SIEM and WAF systems.

We'll design the "what" (the event payload), the "how" (the transport mechanisms), and the "where" (the integration points with the broader security ecosystem). This is fundamentally a system design challenge involving data pipelines, APIs, and distributed systems—areas where your software engineering background will be invaluable.

---

1. Designing the RASP Event Schema: The "What"

A RASP agent's primary output is a security event. Unlike a WAF, which only sees network traffic, a RASP has deep application context. Our event schema must reflect this unique advantage. A well-designed event is the foundation for effective threat detection, incident response, and forensic analysis.

A common, flexible format for such events is JSON. Let's design a comprehensive schema for a SQL Injection event.

{
  "eventId": "a1b2c3d4-e5f6-7890-1234-567890abcdef",
  "timestamp": "2023-10-27T10:00:05.123Z",
  "product": "RASP Agent",
  "agentVersion": "2.5.1",
  "actionTaken": "BLOCK",
  "severity": "CRITICAL",
  
  "application": {
    "name": "payment-processor",
    "host": "prod-app-server-04.internal",
    "ip": "10.1.2.34"
  },

  "attack": {
    "type": "SQL_INJECTION",
    "ruleId": "SQLI-001",
    "description": "SQL injection attempt detected in prepared statement."
  },

  "source": {
    "ip": "203.0.113.55",
    "port": 54321,
    "headers": {
      "User-Agent": "sqlmap/1.6.8",
      "X-Forwarded-For": "198.51.100.10"
    }
  },

  "context": {
    "http": {
      "method": "POST",
      "url": "/api/v1/user/search",
      "requestId": "d9e8f7a6-c5b4-32a1-b0c9-d8e7f6a5b4c3"
    },
    "user": {
      "id": "user-12345",
      "tenantId": "acme-corp"
    },
    "runtime": {
      "language": "Java",
      "vmVersion": "17.0.8",
      "vulnerableMethod": "com.example.dao.UserDAO.findUserByName",
      "lineNumber": 88,
      "stackTrace": [
        "com.example.dao.UserDAO.findUserByName(UserDAO.java:88)",
        "com.example.service.UserService.search(UserService.java:42)",
        "com.example.controller.UserController.searchUsers(UserController.java:25)",
        "..."
      ],
      "parameters": [
        { "name": "username", "value": "admin' OR '1'='1" }
      ]
    }
  }
}

This schema provides a complete narrative: what happened (attack), who did it (source, user), where it happened (application, runtime), and how it was handled (actionTaken). The runtime object is RASP's superpower, providing ground-truth evidence like the exact method and stack trace, which is invaluable for both developers and security analysts. The requestId is crucial for correlating this event with logs from other systems, like a WAF or load balancer.

Key Design Considerations for Data Output

Creating a rich event is only half the battle. We must also manage the data's volume and sensitivity. The article "What is RASP Agent?" highlights several failure modes and metrics related to this challenge.

Based on these considerations, we must build two critical features into our data output design:

1. Data Redaction: The parameters field in our event is powerful but dangerous. It could contain Personally Identifiable Information (PII), passwords, or API keys. A production-grade RASP must have a configurable redaction engine that masks sensitive patterns (e.g., credit card numbers, social security numbers) before the event is transmitted. This is a non-negotiable requirement for privacy and compliance (GDPR, CCPA).

2. Volume Control: A busy application could generate millions of events, leading to a "telemetry flood" that overwhelms the SIEM and incurs massive costs. We can use techniques from our previous lesson:

   • Sampling: Send 100% of BLOCK events but only a 1% sample of low-severity DETECT events.
   • Aggregation: Instead of sending 1,000 identical events, send one summary event every minute: {"event": "...", "count": 1000}.
   • Tiered Verbosity: Send the full, rich JSON for critical events, but a lightweight, summarized version for informational ones.

---

2. Integration with SIEM Systems: The "How" and "Where"

A Security Information and Event Management (SIEM) system (like Splunk, Elastic SIEM, or Microsoft Sentinel) is the primary consumer of RASP events. It aggregates logs from across the enterprise, correlates them to find broader attack campaigns, and provides tools for long-term storage and investigation.

How do we get our beautifully crafted JSON events to the SIEM? Let's explore three common architectural patterns.

Pattern 1: Log File Forwarding (Decoupled & Traditional)

This is a classic and robust pattern.

• Architecture:

  1. The RASP agent writes its JSON events to a local file (e.g., /var/log/app/rasp-events.log).
  2. A separate log forwarding agent (like Fluentd, Logstash, or a vendor-specific forwarder) is installed on the same host or as a sidecar.
  3. The forwarder "tails" this log file, reads new events, buffers them, and securely transmits them to the SIEM's ingestion endpoint.

• Trade-offs:

  • Pro: Excellent decoupling. The RASP agent's only job is to write to a file, a simple and low-latency operation. It doesn't need to know about network issues, SIEM endpoints, or authentication. Backpressure is handled by the forwarder.
  • Con: Requires deploying and managing a second agent (the forwarder), which adds operational overhead.

The "Sidecar Pattern" image below illustrates this perfectly. Imagine the "Main Application" is your app with the RASP agent inside, and the "Sidecar Support" container is the log forwarder.

Pattern 2: Direct API Push (Simple & Coupled)

• Architecture: The RASP agent itself formats an HTTP request and sends the event directly to the SIEM's data ingestion API (e.g., Splunk HTTP Event Collector).

• Trade-offs:

  • Pro: Simple to implement, with no extra agents to manage.
  • Con: Tightly couples the RASP agent to the SIEM. If the SIEM is down or the network is slow, the agent must handle retries, buffering, and potential data loss, adding complexity and performance risk to the application process itself.

Pattern 3: Message Broker (Scalable & Resilient)

This is the most scalable and flexible approach, leveraging your expertise in distributed systems.

• Architecture:

  1. The RASP agent sends its events to a message broker topic (e.g., Apache Kafka, AWS SQS, Google Pub/Sub).
  2. One or more consumers subscribe to this topic. One consumer is the SIEM connector, which reads from the broker and forwards events to the SIEM.
  3. Other consumers can be added for different purposes without affecting the RASP agent: a real-time alerting engine, a long-term archival process to a data lake, etc.

• Trade-offs:

  • Pro: Extremely decoupled and resilient. The message broker acts as a massive buffer, absorbing spikes in volume and allowing consumers to be offline without data loss.
  • Con: Introduces another major piece of infrastructure (the message broker) that must be managed and secured.

The WAF log pipeline shown below, while for a different product, exemplifies a robust, cloud-native version of this pattern. A RASP pipeline could follow a similar flow: RASP Agent -> Kinesis/SQS (Broker) -> Lambda (Processor/Forwarder) -> SIEM/S3.

---

3. Interoperation with WAF Systems: Creating a Feedback Loop

WAF and RASP are complementary technologies that form a powerful "defense-in-depth" tandem. The WAF handles high-volume, less sophisticated attacks at the edge, while RASP provides deep, contextual protection inside the application. Effective integration makes the whole system smarter.

Integration Pattern: RASP-to-WAF Feedback Loop

This is the most valuable integration. RASP can act as a high-fidelity "sensor" that tells the WAF when to block a malicious actor at the network edge, protecting all applications behind it.

• Architecture:

  1. A RASP agent detects a persistent attacker (e.g., an IP address 203.0.113.55 makes 5 failed SQLi attempts in 60 seconds).
  2. The agent sends a special "block signal" event to a central Management/Control Plane (we will design this in our next lesson). This event is distinct from a standard log event.
  3. The Control Plane authenticates the signal and, using its administrative privileges, calls the WAF's provider API (e.g., Cloudflare API, AWS WAF API).
  4. The API call dynamically creates a rule in the WAF to block all traffic from IP 203.0.113.55 for a specified duration (e.g., 1 hour).

• Why this architecture? The RASP agent should never have the credentials to modify WAF rules directly. That would be a huge security risk. The Control Plane acts as a trusted, centralized broker for such actions.

This feedback loop allows the system to automatically escalate its defenses—moving from in-app monitoring (RASP) to perimeter blocking (WAF) based on high-confidence intelligence.

---

Conclusion

Today, we've designed the critical communication pathways that allow our RASP agent to function as part of a larger security ecosystem. We've moved beyond simple detection to architecting a system that produces actionable, context-rich intelligence and shares it effectively.

Key Takeaways:

• RASP Event Schema: The power of RASP is in its context. A well-designed event schema (e.g., in JSON) must include runtime details like stack traces and method names, but also be designed with data redaction and volume control in mind.
• SIEM Integration Patterns: We have multiple architectural choices, each with trade-offs. Log forwarding offers good decoupling, direct API calls are simple but create coupling, and using a message broker provides the most scalability and resilience for large-scale systems.
• WAF Interoperation: RASP and WAF are a powerful combination. The key integration is a RASP-to-WAF feedback loop, where a high-confidence RASP detection triggers a block rule at the WAF via a central control plane and the WAF's API.

Preview of the Next Lesson

We've mentioned a "Management Console" or "Control Plane" several times today. This centralized system is the brain of the entire RASP deployment. In our final lesson on RASP, we will design a centralized management console for configuring RASP policies and deploying them to a fleet of agents. This will complete our architecture, providing the mechanism to control and orchestrate the agents we've designed.