Building a Security Event Pipeline

Hello! Welcome to the seventh lesson in our course on architecting security products.

In our previous lesson, we designed the collection and transport layer of a security telemetry pipeline. We ended with a robust system capable of ingesting high volumes of raw logs from diverse sources and reliably delivering them to a central, durable message queue like Apache Kafka.

However, these raw logs are a chaotic mix of different formats, schemas, and levels of detail. In their raw state, they are of limited use for security analysis. This brings us to our current learning outcome: Design a security event processing pipeline for normalization, enrichment with threat intelligence, and routing to consumers like SIEMs.

Today, we'll architect the crucial next stage of the pipeline. We'll design the "refinery" that transforms this crude oil of raw telemetry into the high-octane fuel of actionable security intelligence. This involves three core functions:

• Normalization: Standardizing disparate event formats into a single, coherent schema.
• Enrichment: Adding valuable context to events to increase their analytical value.
• Routing: Intelligently sending processed events to the right downstream systems.

1. The Rationale: Why Process Events Before Analysis?

Before diving into the architecture, it's essential to understand why this processing layer is so critical in modern security operations. Simply forwarding all raw logs directly to a Security Information and Event Management (SIEM) system is often inefficient, costly, and leads to poor outcomes.

The industry has developed a category of tools to solve this problem, often called Security Data Pipeline Platforms (SDPPs). Let's start by understanding the problems they solve.

As the article highlights, the primary drivers for a dedicated processing pipeline are:

• Cost Management: Raw logs are voluminous. Filtering and reducing them before they hit expensive, volume-based SIEMs can lead to massive cost savings.
• Data Quality & Complexity: With dozens of security tools, each with a unique log format, it's an engineering nightmare to perform analysis without first standardizing the data.
• Performance: A SIEM can become overwhelmed trying to parse, normalize, and enrich a firehose of raw data. Offloading this work to a dedicated, scalable pipeline improves the performance of the entire security stack.

2. A Blueprint for a Security Event Processing Pipeline

Now, let's outline an architecture for this processing layer. It's helpful to think of it as a multi-stage data plane that consumes events from the transport layer we designed in the last lesson.

The following article describes a real-world architecture for a detection system. Its "Preprocessor" stage is a perfect model for the system we need to design.

This blueprint gives us a clear, microservices-based architecture. Raw events are ingested, passed to a Preprocessor for normalization and enrichment, and then forwarded to an Evaluator. For our purposes, the "Evaluator" and "Alert Service" are examples of consumers, alongside others like SIEMs or data lakes.

Our focus today is on designing that Preprocessor component. Let's break down its core functions: normalization, enrichment, and routing.

3. Core Function 1: Normalization and Structuring

The first task is to tame the chaos. A log event from a Palo Alto Networks firewall looks completely different from an AWS CloudTrail event or a Windows logon event. Normalization is the process of parsing these varied formats and mapping them to a common, standardized schema.

Popular standard schemas in the industry include:

• Elastic Common Schema (ECS)
• Open Cybersecurity Schema Framework (OCSF)
• Splunk's Common Information Model (CIM)

A normalization service would perform the following steps:

1. Consume a raw event from the message queue (e.g., a Kafka topic).
2. Detect the format (e.g., JSON, Syslog, CEF, XML).
3. Parse the raw text string into key-value fields.
4. Map the source-specific fields to the common schema. For example, src_ip from one log source and source.ip from another both get mapped to source.ip in ECS.
5. Standardize critical values, like converting timestamps to a consistent UTC format and action fields (permit, allow, allowed) to a single value (allowed).
6. Publish the structured, normalized event to a new "normalized-events" topic in the message queue.

This diagram illustrates the architecture of such a service.

4. Core Function 2: Enrichment

A normalized event is structured and consistent, but it still only contains the information from the original source. Enrichment is the process of adding external context to make the event more valuable for analysis.

As the "Matters AI" article noted, this turns a simple event like "user downloaded a file" into a high-fidelity signal: "user from a flagged IP downloaded sensitive PII data, which they’ve never done before, and they already have 3 alerts this week."

An enrichment service would typically:

1. Consume a normalized event.
2. Extract key indicators from the event (e.g., IP address, domain name, file hash, username).
3. Query various internal and external data sources for context about these indicators.
4. Append this context to the event object.
5. Publish the enriched event to an "enriched-events" topic.

Common enrichment sources include:

• GeoIP Databases: Map an IP address to a physical location.
• Threat Intelligence Feeds: Check if an IP, domain, or hash is on a list of known malicious indicators.
• Asset Databases (CMDB): Add information about a host, such as its owner, function, and data sensitivity.
• Identity Systems (e.g., Active Directory): Add user context like department, role, and manager.
• Behavioral Baselines: Note if the observed activity is anomalous for this specific user or entity.

To see this in action, let's look at a tool purpose-built for threat intelligence enrichment.

The MISP demo provides a concrete visualization of taking a simple indicator and layering on external context to build a richer picture—the very essence of enrichment.

5. Core Function 3: Routing and Tiering

Now we have a stream of normalized, enriched, high-fidelity security events. The final step is to route them to the appropriate consumer. Not all data has the same value or requires the same destination. A routing strategy allows you to optimize for both cost and speed.

A routing service would:

1. Consume an enriched event.
2. Inspect the event's content, priority, or tags.
3. Make a routing decision based on pre-configured rules.
4. Publish the event to the designated destination.

This process is sometimes called "tiering" because you route data to different storage/analysis tiers based on its value. Here is a typical routing strategy:

• Hot Path (Real-time Alerting):

  • Events: High-severity alerts, confirmed malicious activity.
  • Destination: SIEM (e.g., Splunk, QRadar) or a SOAR (Security Orchestration, Automation, and Response) platform.
  • Goal: Immediate visibility and automated response.

• Warm Path (Investigation & Hunting):

  • Events: All enriched security events (alerts, notable activities).
  • Destination: A Security Data Lake or specialized analytics database (e.g., ClickHouse, Elasticsearch).
  • Goal: Retain data for several months for threat hunting and incident investigation.

• Cold Path (Compliance & Archival):

  • Events: All raw or normalized logs.
  • Destination: Cheap object storage (e.g., AWS S3, Google Cloud Storage).
  • Goal: Long-term retention (1+ years) for compliance and forensic purposes at minimal cost.

This approach is detailed in the "Market Guide" article.

By architecting a pipeline with intelligent routing, you decouple your data sources from your analysis tools. You can swap out a SIEM vendor or add a new analytics platform just by changing the routing rules in your pipeline, without having to reconfigure hundreds or thousands of data sources.

Conclusion

We have now designed the critical middle layer of our security data pipeline. By taking raw telemetry from the transport layer and passing it through a dedicated processing pipeline, we transform low-value logs into high-fidelity, context-rich security events ready for analysis.

Key Takeaways:

• A security event processing pipeline consists of three main stages: Normalization, Enrichment, and Routing.
• Normalization addresses data complexity by parsing disparate log formats and mapping them to a common schema (like OCSF or ECS), enabling consistent queries.
• Enrichment adds critical context to events (e.g., threat intelligence, GeoIP, user info) from external sources, dramatically increasing their analytical value and reducing false positives.
• Routing provides architectural flexibility and cost optimization by sending events to different destinations (SIEM, data lake, cold storage) based on their value, priority, and retention requirements.

Preview of the Next Lesson

So far in this module, we've designed foundational architectural patterns for security products. However, we've largely ignored a critical real-world constraint for any SaaS product: multi-tenancy. How do you build these systems to serve thousands of customers securely and efficiently on shared infrastructure?

In our next and final lesson of this module, we will address that question directly: Analyze multi-tenant isolation trade-offs in security products across data, configuration, and performance dimensions.