Scan Orchestration Engine Architecture

Hello! Welcome to a new module in your course on designing security products.

In our previous module, we completed a deep dive into architecting a Zero Trust Network Access (ZTNA) solution. We explored core distributed systems patterns like the separation of control and data planes, agent-based architectures, and policy engines. We concluded by focusing on high availability and observability, the foundations of any reliable service.

Now, we shift our focus to a different but equally critical category of security product: Vulnerability Scanners. You'll find that many of the architectural principles we've mastered—such as managing distributed agents and orchestrating tasks—are directly applicable here, though adapted for a new purpose.

This lesson kicks off Module 8, "Architecting Vulnerability Scanners." Our goal today is to address the learning outcome: Design the high-level architecture of a scan orchestration engine for target management, scheduling, and distributed worker management. We will lay the groundwork for a scalable system that can manage what to scan, when to scan it, and how to distribute the work across a fleet of scanners.

1. The Components of a Vulnerability Management System

At its core, a vulnerability management system is a distributed task execution platform. It needs a central "brain" to manage work and distributed "hands" to perform it. To visualize this, let's look at the architecture of a real-world, open-source vulnerability scanner, Greenbone.

From this diagram, we can identify the key actors in our system design:

• Orchestration Engine (Control Plane): This is the central gvmd (Greenbone Vulnerability Manager Daemon). It's the brain of the operation, responsible for managing users, defining targets, scheduling scans, and processing results.
• Distributed Workers (Data Plane): These are the OpenVAS and Notus scanners. They are the hands that execute the actual scan tasks assigned by the orchestrator. They are "workers" in a distributed system.
• Targets: These are the Target Systems that the workers scan.
• Interfaces: The Greenbone Security Assistant (the web UI) and GMP Clients (APIs) allow users to interact with the orchestration engine.

Our primary focus in this lesson is to design the architecture of the Orchestration Engine, which is responsible for the "management" and "scheduling" parts of the learning outcome.

2. The Core Abstraction: The Scan Job

To build a generic and extensible engine, we need a standard way to represent a unit of work. We'll call this a "scan job" or "activity." This is more than just a transient entry in a message queue; it's a persistent, stateful object that represents a scan from its creation to its completion.

As the article explains, defining a polymorphic work unit is key. The central scheduler doesn't need to understand the specifics of how to run a network scan versus a code scan; it only needs to manage the lifecycle of the activity.

Let's apply this model to our vulnerability scanner:

• type_discriminator: This would specify the kind of scan, e.g., NETWORK_SCAN, WEB_APP_SCAN, SAST_SCAN.
• payload: A JSON object containing all the data a worker needs to execute the scan. For a NETWORK_SCAN, this might be:

  {
    "targets": ["10.0.0.0/24", "192.168.1.5"],
    "port_list": "top-1000",
    "scan_profile": "full_tcp_connect",
    "credentials_id": "cred-123" // for authenticated scans
  }
  

• routing_context: Determines which pool of workers should execute this job, e.g., internal_scanners, external_pci_scanners. This is crucial for directing scans to the right environment.
• scheduling_metadata: Contains priority, activation_time (for scheduled scans), and retry_policy.
• status: Tracks the job's progress through the state machine: Ready -> Pending -> Executing -> Completed/Failed.

3. Managing Distributed Workers: The Lease Model

With a clear definition of a "scan job," our next challenge is to safely and reliably assign these jobs to our distributed scanner workers. We must prevent two workers from running the same job and gracefully handle a worker crashing mid-scan.

A simple "fire-and-forget" approach using a message queue is insufficient. A robust solution is a pull-based, optimistic concurrency model using leases.

Let's summarize the key architectural implications of the lease model:

1. Worker-Initiated Communication: Workers (scanners) poll the central orchestration engine for work. This is a "pull" model, which is highly advantageous because it doesn't require the orchestrator to have inbound network access to workers. This allows workers to be deployed behind firewalls and NAT gateways with no special network configuration.
2. Atomic Operations: The orchestrator uses an atomic database transaction (e.g., UPDATE ... WHERE ... RETURNING ...) to select a Ready job and transition it to Pending while assigning a lease token and expiration time. This prevents race conditions where two workers might claim the same job.
3. Built-in Fault Tolerance: If a worker crashes, goes offline, or is terminated by an auto-scaler, it will stop renewing its lease. When the lease expires, the orchestrator automatically transitions the job's status to Restarting or back to Ready, making it available for another worker to pick up. This provides excellent resilience without complex external health-checking systems.
4. State Management: The lease status is managed centrally by the orchestrator, providing a single source of truth for which worker is doing what at any given time.

4. Scheduling: Deciding What to Run and When

Now that we have a mechanism to dispatch jobs, we need the logic to decide which jobs to dispatch from the pool of Ready jobs. A simple First-Come, First-Served (FCFS) queue is often inadequate for a production system. We need more sophisticated scheduling to handle priorities, resource constraints, and fairness.

Let's synthesize these concepts into a hybrid scheduling strategy for our scan engine. This is where we combine target management, worker management, and scheduling logic.

• Target and Worker Tagging: Both targets and workers can be assigned key-value tags.

  • Targets: env:prod, pci-scope:true, data-sovereignty:eu
  • Workers: location:aws-us-east-1, network:internal, capability:authenticated-scan

• Multilevel Queues via Tag-Based Routing: The scheduler uses tags to enforce routing. The routing_context on a job (e.g., eu-prod-scanners) is matched against workers with corresponding tags. This creates logical queues, just like the "Fast," "Medium," and "Heavy" worker pools in the case study. This ensures, for example, that scans tagged for data sovereignty in the EU are only ever run on workers located in the EU.

• Priority-Aware Job Selection: Within a given queue, the scheduler selects jobs based on priority. The database query to fetch the next job would look something like this:

  SELECT *
  FROM scan_jobs
  WHERE status = 'Ready'
    AND routing_context = 'internal_scanners' -- Worker is polling this queue
    AND activation_time <= NOW()
  ORDER BY priority DESC, created_at ASC
  LIMIT 1;
  

• Capacity and Fair-Share Enforcement: Before leasing a job to a worker, the scheduler checks against tenant-level quotas (e.g., max_concurrent_scans_per_team). If a team is at its limit, its lower-priority jobs are deferred, ensuring fairness and preventing any single tenant from monopolizing the scanning infrastructure.

5. High-Level Architecture Synthesis

Let's bring all these pieces together into a coherent architectural diagram for our scan orchestration engine.

A high-level diagram of the scan orchestration engine. The Control Plane contains the API, Target/Worker/Job Managers, and the Scheduler, all interacting with a central database. The Data Plane consists of distributed Scan Workers that poll for jobs via the Worker Manager and execute them.

The workflow for a scheduled scan would be:

1. A user defines a Target (e.g., a CIDR range) and a Scan Configuration (e.g., "Daily Light Scan") via the UI/API.
2. The Job Manager creates a scan_job record in the database with a future activation_time.
3. When the time comes, the Scheduler finds this job (its status is Ready).
4. Meanwhile, a Scan Worker in the appropriate pool (location:internal) polls the Worker Manager for a job.
5. The Scheduler, having selected the job based on priority and fairness, approves it. The Worker Manager performs an atomic database update to lease the job to this specific worker, setting the status to Pending.
6. The worker receives the job payload, transitions its local state to Executing, and begins the scan. It periodically sends a "heartbeat" to the Worker Manager to renew its lease.
7. Upon completion, the worker uploads the scan results and reports the final status (Completed or Failed) to the Job Manager, which terminates the lease.
8. If the worker crashes, its lease expires. The Scheduler sees the expired lease, marks the job for restart, and it becomes Ready to be picked up by another worker.

Conclusion

In this lesson, we designed the complete high-level architecture for a robust and scalable scan orchestration engine. We went beyond simple queuing and designed a system capable of managing a complex, distributed security application.

Key Takeaways:

• A scan orchestration engine acts as a control plane, managing targets, jobs, and a fleet of distributed workers (the data plane).
• Modeling scan jobs as persistent, stateful "activities" with a defined lifecycle is a flexible and powerful abstraction.
• A pull-based model with leases is a superior pattern for distributed worker management, providing fault tolerance and network-friendliness.
• Effective scheduling combines multiple algorithms: tag-based routing for workload isolation, priority for urgency, and capacity limits for fairness.

Preview of the Next Lesson:

We have successfully designed the "brain" of our vulnerability scanner. Now, it's time to start designing the "hands." In our next lesson, we will begin our deep dive into the worker itself by tackling the most fundamental task in network scanning: discovering what's on the network. We will fulfill the outcome: Compare SYN, connect, and UDP port scanning techniques, analyzing their network mechanisms and stealth trade-offs.