Building a Vulnerability Matching Engine

Hello! Welcome back to our course on architecting security products.

In our previous lessons, we've built the foundational pieces of our vulnerability scanner. We designed a system to discover services and a secure, TUF-based pipeline to deliver a trusted vulnerability knowledge base to our distributed scanners. Now, we have a scanner that knows what services are running on the network and possesses an up-to-date, signed database of all known vulnerabilities.

The critical task now is to connect these two pieces of information. This lesson is dedicated to architecting the "brain" of the scanner, which addresses the learning outcome: Architect a vulnerability matching engine that correlates discovered services and versions against the knowledge base.

We will design the core logic that takes a discovered service, like "Apache httpd 2.4.53", and uses our knowledge base to determine if it's vulnerable. This involves tackling the messy reality of unstructured product names, complex versioning schemes, and inconsistent data standards.

1. The Architectural Blueprint

At a high level, the matching process takes raw discovery data and correlates it against a structured vulnerability database.

The diagram highlights the central role of CPE (Common Platform Enumeration). CPE is the standardized naming scheme used by the NVD to identify software and hardware. A typical CPE 2.3 string looks like this:

cpe:2.3:part:vendor:product:version:update:edition:language:sw_edition:target_sw:target_hw:other

• part is a for application, o for OS, h for hardware.
• vendor, product, and version are the key fields we'll focus on.

Our scanner discovers a service like "OpenVPN 2.4.11-I062.win10". The NVD knows about vulnerabilities for cpe:2.3:a:openvpn:openvpn:2.4.11:*:*:*:*:*:*:*. The core challenge for our matching engine is to bridge this gap.

We will architect our engine as a two-stage pipeline:

1. Stage 1: CPE Identification Engine. This is an information retrieval system. It takes unstructured, discovered text (e.g., a banner, a software name) and finds the most likely canonical CPE string from our local copy of the CPE dictionary.
2. Stage 2: Vulnerability Correlation Engine. This is a structured data processing system. It takes the identified CPE string and queries our local vulnerability database to find all associated CVEs, correctly handling complex version range logic.

Let's dive into the design of each stage.

2. Stage 1: The CPE Identification Engine

This engine's task is to solve a fuzzy search problem: map a string like "Mozilla Firefox 19.0 beta1" to a canonical CPE. There are several architectural approaches, from simple heuristics to more complex information retrieval techniques.

Approach A: Sanitization and Fuzzy Matching

A pragmatic, engineering-driven approach is to clean the input string as much as possible and then use a fuzzy string matching algorithm to find the best fit in the CPE dictionary.

Based on the paper, we can design the following pipeline:

1. Sanitization: Create a multi-step process to normalize the discovered software name and version.
   • Name Standardization: Remove common corporate suffixes ("Inc.", "LLC"), architecture strings ("x86_64"), and other noise. Convert to lowercase. For example, "OpenVPN Technologies, Inc." becomes "openvpn".
   • Version Simplification: Strip build metadata and platform identifiers. For example, "2.4.11-I602-Win10" is simplified to "2.4.11".
2. Candidate Retrieval: Query the local CPE dictionary (which contains fields like title, vendor, product) to find potential matches for the sanitized name. The paper suggests a clever prioritized query system to find high-confidence candidates first.
3. Fuzzy Matching: For each candidate, calculate a similarity score between the sanitized input name and the CPE's title/product field. An algorithm like Levenshtein distance (which RapidFuzz uses) is excellent for this. It measures the "edit distance" between two strings.
4. Selection: Select the CPE with the highest similarity score, provided it exceeds a certain confidence threshold (e.g., > 70%).

This approach is effective and relatively straightforward to implement. However, it can be sensitive to word order and may require many custom sanitization rules.

Approach B: Vector-Based Similarity Search

For a more mathematically robust solution, we can treat this as a classic information retrieval problem. Instead of matching strings, we convert the text to numerical vectors and find the one with the highest similarity.

Let's break down this powerful technique:

1. Vectorization: Represent both the query ("Apache Airflow Server 2.4.0") and each product title in the CPE dictionary ("Apache Software Foundation Airflow 2.4.0") as vectors. A common way to do this is with Term Frequency (TF).
   • First, create a "bag of words" containing all unique words from both strings.
   • Then, create a vector for each string where each element represents the count of a word from the bag.
2. Similarity Calculation: Use Cosine Similarity to measure the angle between the two vectors. $$\text{similarity}=\cos (\theta )=\frac{A\cdot B}{\Vert A\Vert \Vert B\Vert }$$ The result is a score between 0 (not similar at all) and 1 (identical). This is more robust than simple string matching because it's insensitive to word order and handles missing words gracefully.
3. Optimization: Calculating this for every entry in the CPE dictionary at runtime would be slow. As the article notes, you can pre-calculate the vector representations for every product in your local CPE dictionary during the database build process. You can also filter the search space to only include products that contain at least one word from the query.

This vector-based approach is more complex but provides a more flexible and accurate matching engine, reducing the need for extensive, brittle sanitization rules.

3. Stage 2: The Vulnerability Correlation Engine

Once Stage 1 provides a candidate CPE (e.g., cpe:2.3:a:jquery:jquery:2.3.1:*:*:*:*:*:*:*), Stage 2 must determine if it's vulnerable. This requires querying our local, TUF-secured vulnerability database.

The Data Model

First, let's define the schema for this database. A well-structured data model is essential for efficient querying.

Our local database will mirror this structure, with tables for CVEs, affected CPE configurations, and relationships between them, all populated from the NVD data feeds we securely download.

The Correlation Logic

Matching a CPE against this database isn't a simple lookup. The NVD specifies affected products in two primary ways.

Let's architect the logic based on this reading:

1. Full CPE Field Comparison: The first step is to ensure all non-version fields of our candidate CPE match the vulnerable configuration in the database. A wildcard (*) in the vulnerable configuration matches any value in our candidate CPE. For example, a vulnerable CPE with target_sw=* matches our candidate CPE regardless of its target_sw value.

2. Version Matching: This is the most complex part.

   • Case 1: Exact Version List. The CVE entry lists specific vulnerable CPEs. This is a simple equality check. Our candidate CPE's version must exactly match one in the list.
   • Case 2: Version Range. The CVE entry provides a generic CPE with versionStartIncluding and/or versionEndExcluding attributes. This requires a robust version comparison algorithm.

   Designing a Version Comparison Algorithm:
   As the search_vulns article highlights, version "numbers" can be incredibly complex (e.g., 13.1-48.47, 14su1). A simple lexicographical sort will fail. A robust algorithm, as described in the article, would work as follows:

   1. Segment: Split both versions into segments based on delimiters like ., -, or _.
   2. Sub-segment: Further split segments by changes in character class (from digit to letter or vice-versa). 14su1 becomes ['14', 'su', '1'].
   3. Compare: Iterate through the segments of both versions and compare them. Numeric parts are compared as integers, and string parts are compared lexicographically.
   4. Normalize: Build in logic to handle equivalences like p and patch, or u and update.

   Implementing this version comparison function is a core task in building the matching engine. It's a perfect example of a problem that requires a first-principles approach to handle real-world data.

3. Handling Data Quality: The NVD is not perfect. Our engine must be architected to handle its quirks. The search_vulns article points out issues like:

   • CVEs that seem to affect all versions of a product because no upper bound is specified.
   • CVEs that list only one specific version as vulnerable when earlier ones are also affected.

   Our engine should not just blindly trust the data. It should identify these ambiguous cases and can be configured to either flag them for manual review or apply heuristics (e.g., "if only version X is listed, assume all versions <= X are vulnerable"). This configurability makes the engine much more powerful for a security analyst.

Conclusion

In this lesson, we have architected the complete vulnerability matching engine, the analytical core of our scanner. We designed a system that can bridge the gap between messy, real-world discovery data and the structured world of vulnerability databases.

Key Takeaways:

• Two-Stage Architecture: Vulnerability matching is best designed as a two-stage process: a CPE Identification engine (a search/IR problem) followed by a Vulnerability Correlation engine (a structured data comparison problem).
• CPE Identification Strategies: This can be approached with a pragmatic Sanitization + Fuzzy Matching pipeline or a more robust Vector Similarity Search using techniques like TF and Cosine Similarity.
• Robust Version Comparison is Crucial: The heart of the correlation engine is a sophisticated algorithm that can parse and compare complex, multi-segment version strings to correctly evaluate version ranges.
• Design for Imperfect Data: A production-grade engine must be aware of and able to handle the inconsistencies and ambiguities present in real-world vulnerability data from sources like the NVD.

Preview of the Next Lesson:

So far, our scanner operates from the outside, looking at network ports and service banners. To get a truly accurate picture of a system's vulnerabilities, we need to look from the inside. In our next lesson, "Design authenticated scanning capabilities, including secure credential storage and privilege management for deep host inspection," we will architect the mechanisms that allow our scanner to securely log in to target systems. This will enable it to perform much deeper analysis, such as retrieving a complete list of installed software and checking patch levels directly.