Designing an Anomaly Detection System for Identity Audit Logs

Hello! Welcome back.

In our last lesson, we architected a high-throughput pipeline to produce a clean, structured stream of identity audit logs. This gives us a reliable source of truth for every critical action in our identity system. Now, we'll build on that foundation to proactively find threats within that data.

Today's learning outcome is to design an anomaly detection system that consumes identity audit logs to identify suspicious behavior like impossible travel. We will move beyond simple log review and architect a system that can automatically hunt for and flag potentially malicious activity in real-time. This is a classic distributed systems problem that intersects with data science, making it a perfect topic to explore from first principles.

The Architectural Blueprint for Anomaly Detection

At its core, an anomaly detection system identifies events that deviate from an established norm. In the context of identity security, these deviations often signal an account compromise. Our system will operate on the stream of audit logs we designed previously, analyzing them in real-time.

There are two primary approaches to this, which are often used together:

1. Rule-Based Detection: This involves defining explicit, deterministic rules for known-bad patterns. These are precise but can only catch what you already know to look for.
2. Machine Learning-Based Detection: This uses statistical methods, often unsupervised, to learn a model of "normal" behavior and flag outliers. This is powerful for discovering novel or subtle attack patterns.

A robust system combines both. Let's look at the overall architecture.

The architecture we will design mirrors this pattern, using Kafka as our input stream and a dedicated processing service to run our detection logic.

As the article illustrates, a modern anomaly detection pipeline is a multi-stage stream processing application. We'll focus on the core detection stages: the Rule Engine for explicit patterns and the Model Scorer for ML-based detection.

Part 1: Rule-Based Detection - The "Impossible Travel" Use Case

Let's start by architecting a solution for the specific "impossible travel" scenario. This is a classic example of a stateful, rule-based check.

The core idea is simple: if two consecutive successful logins for the same user occur from locations that are physically impossible to travel between in the elapsed time, it's a major red flag.

Based on this logic, we can design the rule engine component.

Architectural Design for Impossible Travel

1. Input: The engine consumes the enriched login_success events from our audit log topic in Kafka. Each event must contain user_id, event_timestamp, and enriched geo_location (latitude and longitude from the source IP).
2. State Management: The system must remember the previous login for every user. A low-latency key-value store is ideal for this. The user_id serves as the key. For each user, we store a small object containing the timestamp and location of their last successful login. Redis is an excellent choice for this due to its speed.
3. Processing Logic (per event):
   • On receiving a login event for user_A, the engine queries Redis for user_A's last login data.
   • If no prior login exists: This is the user's first observed login. The engine stores the current event's timestamp and location in Redis under the key user_A and takes no further action.
   • If a prior login exists:
     • Calculate the time difference: \Delta t = \text{current_timestamp} - \text{last_timestamp}.
     • Calculate the geographic distance ($\Delta d$) between the current and last locations using the Haversine formula. The math.geo_distance function in the article is an implementation of this.
     • Calculate the required speed: $v=\Delta d/\Delta t$.
     • Apply the rule: If $v$ exceeds a plausible speed (e.g., > 1000 km/h, which is faster than commercial air travel), generate an ImpossibleTravel alert.
     • Finally, update Redis with the current event's timestamp and location.

This stateful design, where we process each event in the context of past events for the same user, is fundamental to behavioral analysis.

Part 2: Machine Learning-Based Detection - Finding the "Unknown Unknowns"

Rules are great for known patterns, but adversaries are creative. What about a user logging in at 3 AM for the first time, or from a new device, or with an unusual frequency? These aren't explicitly "impossible," but they are suspicious. This is where machine learning comes in.

We will design an unsupervised anomaly detection system. Instead of being taught what is "bad," the model learns what is "normal" for each user and flags significant deviations.

Architecting the ML Detection Component

Following the structure from the video, our ML detection system will have several key stages.

1. Feature Engineering

Our stream processor (e.g., Flink) will consume the audit logs and compute a rich set of behavioral features for each user in real-time. This involves stateful operations over time windows.

Examples of features we could engineer from identity audit logs include:

• Activity Patterns:
  • Login frequency (count of logins in last 1h, 24h, 7d).
  • Time-of-day patterns (is the login happening outside the user's typical 9-5 window?).
  • Session duration statistics (mean, std dev).
• Network & Environment Features:
  • IP address reputation and novelty (is this a new or risky IP for this user?).
  • Number of distinct countries/cities logged in from in the last 30 days.
  • User-agent/device fingerprint novelty.
• Real-time Behavioral Features:
  • Number of failed logins preceding a success.
  • Time since last password reset or MFA change.

2. Model Selection and Architecture

Since we lack a large, labeled dataset of "compromised" vs. "safe" events, unsupervised models are the best fit. They learn a baseline of normal behavior and score events based on their deviation from that baseline.

• Isolation Forest: This algorithm builds a forest of random trees. The intuition is that anomalous points are "few and different" and will therefore be easier to isolate, meaning they will have a shorter average path from the root of the trees. It's fast and works well with high-dimensional data.
• Autoencoder: This is a type of neural network trained to reconstruct its own input. It first compresses the input into a low-dimensional representation (encoding) and then tries to rebuild the original from that compression. If the model is trained on "normal" data, it will have a high reconstruction error when it sees an anomalous event, which it hasn't learned to represent effectively.

3. Score Fusion and Alerting

Our system now produces multiple risk signals: a deterministic flag from the "impossible travel" rule, a score from the Isolation Forest, and a reconstruction error from the Autoencoder. The Score Fusion component, as described in the Designing a Fraud Detection System article, combines these into a single, unified risk score.

A simple weighted sum is a common starting point:

$$\text{FinalScore}=(w_1\times \text{RuleSignal})+(w_2\times {\text{MLScore}}_1)+(w_3\times {\text{MLScore}}_2)$$

This final score is then compared against thresholds to make a decision:

• Low Score: No action.
• Medium Score: Flag for analyst review (e.g., create a low-priority ticket).
• High Score: Trigger an immediate alert and potentially an automated response, like forcing MFA on the user's next login.

Conclusion

In this lesson, we designed a sophisticated, two-pronged anomaly detection system that builds directly upon the audit logging pipeline from our previous session. We combined the precision of rule-based engines for known threats like "impossible travel" with the discovery power of unsupervised machine learning to find novel attacks.

Key Takeaways:

• Hybrid Approach: A robust anomaly detection system leverages both deterministic rules (for precision and compliance) and ML models (for discovering unknown patterns).
• Stateful Stream Processing: Behavioral detection is inherently stateful. Systems must maintain per-entity state (e.g., last login location per user) using tools like Redis or Flink's managed state.
• Feature Engineering is Key: The quality of an ML detection system is heavily dependent on the quality of the features engineered from the raw data stream.
• Unsupervised Models for Security: In the absence of large labeled datasets, unsupervised models like Isolation Forests and Autoencoders are powerful tools for learning a "normal" baseline and detecting deviations.
• Score Fusion for Decisioning: Combining signals from multiple detectors into a single risk score allows for nuanced, threshold-based alerting and response.

Preview of the Next Lesson:

So far, we have focused on architectures for detecting suspicious activity after it has occurred. In our next lesson, we will shift our focus to preventing unauthorized access from the start. We will begin our module on Zero Trust by tackling the outcome: Design the end-to-end connection flow for a ZTNA service, contrasting the data and control planes with a traditional VPN architecture. This will be a fascinating look at how modern security is moving away from perimeter-based trust and towards a model of "never trust, always verify."