Runtime Threat Detection Engine Design

Hello! Welcome back to our course on architecting security products.

In our last lesson, we designed the sensory system of our RASP: a runtime context capture mechanism. We architected a system that uses hooks and thread-local storage to collect and correlate data, giving us a complete picture of a request's journey—from its source as an HTTP parameter to its sink in a database query.

We now have the rich data. The question is, what do we do with it? This brings us to today's topic: designing the brain of our RASP. We will architect the engine that analyzes the captured context and makes a crucial decision: is this action benign, or is it an attack?

Our learning outcome for this lesson is to: Design a vulnerability detection engine that analyzes captured runtime context for threats like SQL injection and command injection.

---

1. From Context to Verdict: The Detection Engine's Role

Let's quickly recall the RequestContext object we designed in the previous lesson. For a given request, it holds:

• Source: The incoming HTTP request details (URI, parameters, headers).
• Sink: The details of a sensitive operation, like a SQL query execution.
• Execution Flow: The stack trace connecting the source to the sink.

The detection engine is a component that receives this RequestContext and applies a set of rules or analytical techniques to produce a verdict: ALLOW or BLOCK.

The fundamental advantage of RASP is that its detection engine operates on high-fidelity data. Unlike a WAF, which analyzes raw HTTP traffic, a RASP analyzes the application's actual behavior. This allows for much more sophisticated and accurate detection logic.

Let's explore two primary architectural approaches for this engine: signature-based analysis and the more powerful syntax-based analysis.

---

2. Designing a SQL Injection (SQLi) Detection Engine

SQL injection remains one of the most critical web application vulnerabilities. A RASP is uniquely positioned to stop it with high precision. Let's design an engine to do just that.

Approach 1: Simple Signature Matching (The WAF-on-the-Inside)

The most straightforward approach is to treat the captured SQL query as a string and run regular expressions against it to find suspicious patterns.

• Input: The full SQL query string from RequestContext.
• Logic: Match against a list of regex patterns like /'\s+OR\s+'\d+'\s*=\s*'\d+/i or keywords like UNION SELECT, SLEEP(), etc.
• Verdict: If a pattern matches, BLOCK.

This is essentially how a traditional WAF works. While simple, it's brittle. Attackers are proficient at creating obfuscated payloads that bypass regex, and this approach can lead to false positives if the application legitimately handles strings that look like SQL. As software engineers building a robust system, we can do better by understanding the structure of the language itself.

Approach 2: Advanced Syntax & Specification-Based Analysis

The key insight for a powerful RASP is this: a legitimate user input should only ever be treated as a value within a SQL statement, not as part of the query's logic. An attack occurs when user input breaks out of its intended value context and modifies the query's structure.

To detect this, our engine must be able to parse and understand SQL grammar. This is achieved by building and analyzing an Abstract Syntax Tree (AST) of the query.

Let's design an AST-based SQLi detection engine. The process involves comparing the structure of the query as intended by the developer with the structure of the query containing user input.

Based on this principle, here is the architecture for our detection module:

1. Reconstruction: The engine receives the SQL template (e.g., SELECT * FROM users WHERE id = ?) and the user-provided parameters (e.g., 1 OR 1=1) from the RequestContext. It reconstructs the final query string.
2. Parsing: The engine uses a dedicated SQL parser library (e.g., JSqlParser for Java, or a library based on an ANTLR grammar) to transform the reconstructed query string into an AST.
3. Structural Comparison: The engine compares the generated AST against a "golden" AST. This golden AST represents the intended, non-malicious structure of the query.
   • For the query SELECT * FROM users WHERE id = 1, the WHERE clause is a simple binary expression (id = 1).
   • For the injected query SELECT * FROM users WHERE id = 1 OR 1=1, the WHERE clause becomes a more complex logical expression ((id = 1) OR (1 = 1)). The AST's structure is fundamentally different.
4. Verdict: If the AST of the runtime query shows a structural change in logic compared to the golden template, the engine flags it as a SQL injection and issues a BLOCK verdict.

This AST-based approach is incredibly powerful because it is immune to simple obfuscation techniques (like changing case, adding comments, or using different whitespace) that can fool regex-based engines. The parser normalizes all of these away.

Adding Specification and Schema Checks

We can further enhance this design by incorporating application-specific knowledge, as described in the "Griffin" RASP tool.

Drawing from Griffin's design, we can add two more checks to our engine:

• Schema Validation: Before even parsing the full SQL, the engine can perform a quick check. Does the user-supplied value for a parameter match the data type and size of the corresponding column in the database schema? For example, if user input is 'abc' but the target column id is an INT, the request can be blocked immediately. A RASP can have access to schema information, whereas a WAF cannot.
• Specification Learning: In a "learning mode," the RASP can automatically build a profile of all the legitimate query structures used by the application. In "protection mode," any query that generates an AST not found in the learned profile is blocked. This automates the creation of the "golden" templates.

---

3. Designing a Command Injection (CmdI) Detection Engine

The same architectural principles apply to detecting OS command injection. An attack occurs when user input is interpreted as a new command or command modifier, rather than a simple argument to the intended command.

An AST-based engine is the most robust way to detect this.

Let's design our CmdI detection engine based on this model:

1. Context Capture: Our hook on Runtime.exec(command) provides the full command string to the engine.
2. Preprocessing & Normalization: The engine first normalizes the command string, handling different encodings or obfuscation techniques to reveal the true underlying command.
3. Parsing: The engine uses a shell command parser to generate an AST for the command string.
4. Semantic Analysis: The engine traverses the AST and applies semantic rules. It looks for structural anomalies introduced by user input. For example:
   • Command Chaining: Did user input introduce a new command via operators like ;, &&, or ||? The AST would show two distinct command sub-trees instead of one.
   • Command Substitution: Did user input introduce `...` or $(...)? The AST would contain a substitution node where a simple value was expected.
5. Verdict: If the analysis reveals that user input altered the command's structure, the engine issues a BLOCK verdict. Otherwise, it's ALLOW.

For example, consider an application that runs ping -c 4 [user_input].

• If user_input is example.com, the AST represents a single ping command with two arguments.
• If user_input is example.com; ls -la, the AST will represent two separate commands. The engine detects this structural change and blocks the request.

---

Conclusion

In this lesson, we designed the analytical core of a RASP: the vulnerability detection engine. We've seen how moving beyond simple string matching to structural analysis provides far more accurate and resilient protection.

Key Takeaways:

• Syntax-based detection using Abstract Syntax Trees (ASTs) is the core architectural pattern for a modern RASP engine. It allows the RASP to understand the grammar and structure of a language like SQL or shell script.
• This approach is fundamentally more robust than signature-based methods because it is resilient to common obfuscation techniques.
• For SQLi detection, the engine compares the AST of a runtime query against a "golden" template or learned specification. Any structural deviation indicates an attack.
• For CmdI detection, the engine parses the command string into an AST and identifies when user input creates new commands or uses dangerous constructs like command substitution.
• A RASP can layer multiple checks, combining AST analysis with schema validation and other context-aware rules to increase detection accuracy.

Preview of the Next Lesson

We have now designed a RASP with sophisticated senses (context capture) and a powerful brain (detection engine). However, all this processing takes time. If our RASP slows down the application it's protecting, it won't be adopted. Performance is a critical non-functional requirement.

In our next lesson, we will address this challenge head-on. The learning outcome is: Design a RASP agent for minimal performance overhead using selective instrumentation, sampling, and caching strategies. We'll explore the engineering trade-offs required to build a RASP that is not only effective but also efficient enough for high-performance production environments.