Designing a Vulnerability Knowledge Base

Hello! Welcome back to our course on architecting security products.

In the last lesson, we designed a service detection engine that gives us a detailed inventory of what's running on a network: a list of hosts, ports, services, and their versions. This inventory is critical, but it's only one half of the equation. To find vulnerabilities, we need to compare this inventory against a comprehensive database of known security flaws.

This lesson is dedicated to building that database. We will directly address the learning outcome: Design the data model and ingestion pipeline for a vulnerability knowledge base from sources like the NVD. We will architect the system that fetches, stores, and maintains the vast universe of vulnerability data, setting the stage for our scanner's "matching" logic.

1. Why Build a Local Vulnerability Knowledge Base?

A vulnerability scanner needs to query vulnerability data constantly. A common first thought might be to just call a public API, like the one provided by the National Vulnerability Database (NVD), for every service the scanner finds. However, for a production-grade system, building a local copy of this data is a superior architectural choice.

Let's explore the reasoning behind this. The following article, while describing a specific implementation, provides an excellent summary of the strategic advantages.

As the article highlights, a local knowledge base gives us:

• Performance: Local database queries are orders of magnitude faster than remote API calls, dramatically speeding up scans.
• Reliability & Offline Capability: The scanner is not dependent on the availability of an external API or an internet connection. This is essential for scanning internal, air-gapped networks.
• No Rate Limiting: Public APIs impose rate limits to ensure fair usage. A high-volume scanner would quickly hit these limits, crippling its operation.
• Customization: A local database allows us to enrich the data with internal context, such as business-specific risk scores, asset ownership, or remediation tracking information, which is impossible with a public API.

With these goals in mind, let's design the two core components: the ingestion pipeline and the data model.

2. The Ingestion Pipeline: Sourcing Data from the NVD

Our knowledge base needs a source of truth. The most authoritative and widely used source for enriched vulnerability data is the National Vulnerability Database (NVD), maintained by NIST. The NVD builds upon the basic list of Common Vulnerabilities and Exposures (CVEs) by adding crucial analysis and metadata, such as severity scores (CVSS), weakness types (CWE), and affected software configurations (CPE).

The NVD provides a comprehensive set of APIs for accessing this data. Our ingestion pipeline will be a service designed to interact with these APIs to populate and synchronize our local database.

Based on this documentation, we can design our ingestion pipeline with two main operational modes:

A. Initial Population (The "Full Sync")

The first time our service runs, it needs to fetch the entire history of CVEs from the NVD. Since the NVD contains hundreds of thousands of records, we cannot fetch them in a single request. The pipeline must implement pagination:

1. Make an initial request to the CVE API (e.g., https://services.nvd.nist.gov/rest/json/cves/2.0).
2. The response will contain the totalResults and the first "page" of vulnerabilities (e.g., 2,000 records).
3. The pipeline calculates the number of remaining pages needed.
4. It then iterates, making subsequent requests and incrementing the startIndex parameter (startIndex=2000, startIndex=4000, etc.) until all vulnerabilities have been fetched.

B. Ongoing Synchronization (The "Delta Sync")

After the initial sync, we need an efficient way to keep our database up-to-date. Re-fetching the entire dataset daily would be incredibly inefficient. Instead, our pipeline will run on a schedule (e.g., every few hours) and perform a delta sync:

1. The pipeline records the timestamp of its last successful run.
2. On the next run, it calls the CVE API using the lastModStartDate and lastModEndDate parameters. For example, it might request all CVEs modified in the last 24 hours.
3. This returns a much smaller, manageable set of new and updated CVEs.
4. The pipeline then processes these changes, inserting new vulnerabilities and updating existing ones in our local database (an "UPSERT" operation).

This two-mode approach ensures both completeness and efficiency.

3. The Data Model: Structuring Vulnerability Knowledge

Now that we know how to get the data, we need to decide how to store it. The NVD API returns data in a rich, nested JSON format. Our task as system architects is to translate this into a structured, queryable data model.

Let's first examine the structure of the data we will receive.

The configurations object is the key to our scanner's functionality. It connects a CVE to a specific set of products using Common Platform Enumeration (CPE) names. A CPE is a standardized string that uniquely identifies a software product, like cpe:2.3:a:apache:http_server:2.4.53:*:*:*:*:*:*:*.

The following diagram illustrates how our scanner will eventually use this data. Our service detection engine (from the previous lesson) produces an inventory. Our knowledge base contains the CPE-to-CVE mappings. The matching engine will connect the two.

Data Model Implementation: Relational vs. Document

Given your software engineering background, you know there are two primary approaches we could take for our database schema:

1. Relational Model (e.g., PostgreSQL)

This approach normalizes the data into distinct, related tables. This is often preferred for its data integrity, structured querying capabilities, and efficient storage. A simplified relational schema might look like this:

• vulnerabilities
  • id (Primary Key)
  • cve_id (e.g., "CVE-2021-44228", Unique, Indexed)
  • description (Text)
  • published_date (Timestamp)
  • last_modified_date (Timestamp)
• cvss_scores
  • id (PK)
  • vulnerability_id (Foreign Key to vulnerabilities.id)
  • version (e.g., "3.1", "2.0")
  • vector_string (Text)
  • base_score (Decimal)
  • severity (Text)
• affected_products
  • id (PK)
  • vulnerability_id (FK to vulnerabilities.id)
  • is_vulnerable (Boolean)
  • cpe_uri (Text, Indexed)
  • version_start_including (Text, nullable)
  • version_end_excluding (Text, nullable)
  • ... (and other version range fields)
• weakness_mappings (Many-to-Many join table)
  • vulnerability_id (FK)
  • cwe_id (e.g., "CWE-79")

2. Document Model (e.g., MongoDB, or PostgreSQL with JSONB)

This approach embraces the original JSON structure. We could have a single collection where each document is the complete JSON object for a single CVE.

• Pros:
  • Extremely simple ingestion pipeline (fetch JSON, store JSON).
  • Flexible schema; no migration needed if the NVD adds new fields.
• Cons:
  • Querying for specific nested fields (like a particular CPE in the configurations array) can be less performant and more complex than a SQL JOIN.
  • Data is duplicated (e.g., the same CPE string might appear in thousands of documents), leading to larger storage requirements.

For a system where structured, high-performance lookups on product and version information are paramount, the relational model is generally the stronger architectural choice. It allows us to build targeted indexes (especially on cve_id and cpe_uri) that are essential for the scanner's matching engine to perform efficiently.

4. High-Level Architecture of the System

Let's tie these components together. The vulnerability knowledge base system consists of the database itself and the scheduled ingestion pipeline that keeps it alive.

A high-level diagram of the ingestion pipeline. A scheduled job triggers a service that fetches data from the NVD API. The service parses this data and loads it into a local PostgreSQL database, which serves as the vulnerability knowledge base.

This architecture creates a robust, self-contained, and high-performance foundation for our vulnerability scanner.

Conclusion

In this lesson, we have designed the complete system for building and maintaining a local vulnerability knowledge base. We established the strategic importance of a local database and architected both the ingestion pipeline and the underlying data model needed to support it.

Key Takeaways:

• Local is Better: A local vulnerability knowledge base provides superior performance, reliability, and customizability compared to relying on public APIs at scan time.
• NVD is the Source: The National Vulnerability Database is the authoritative source for enriched CVE data, including CVSS scores and, most importantly, affected product configurations (CPEs).
• Ingestion is a Two-Step Process: The pipeline must support a one-time "full sync" using pagination and a recurring "delta sync" using date-range filters to stay current efficiently.
• Data Modeling is Key: A normalized relational schema is often the best choice for this use case, as it provides the query performance and data integrity needed for a fast and accurate matching engine. The link between CVE and CPE is the most critical relationship to model correctly.

Preview of the Next Lesson:

We now have the two essential inputs for vulnerability scanning:

1. A list of discovered services and versions (from our service detection engine).
2. A comprehensive database of vulnerabilities and the products they affect.

The next lesson, "Architect a vulnerability matching engine that correlates discovered services and versions against the knowledge base," is where we bring it all together. We will design the logic that takes a discovered service like "Apache httpd 2.4.53" and queries our new knowledge base to definitively determine if it is vulnerable.