Skip to content
BranchesTags

Repository files navigation

NVIDIA AI Blueprint: Vulnerability Analysis for Container Security

This repository powers the build experience for the NVIDIA AI Blueprint for Vulnerability Analysis for Container Security. It combines NVIDIA NeMo Agent Toolkit (NAT), NVIDIA NIM microservices (NIM), and reusable deep-research infrastructure from the NVIDIA AI-Q Blueprint.

Container scanners identify known vulnerabilities, but a finding alone does not show whether the affected code is present in the active container environment, reachable through its execution paths, or exploitable under its runtime controls. Establishing that context—and documenting a defensible decision—typically requires manual research across advisories, packages, binaries, configuration, and container behavior.

The blueprint automates this evidence collection and contextual analysis so security teams can prioritize vulnerabilities that pose real risk while retaining cited reports for analyst review.

Given one or more CVE or GitHub Security Advisory (GHSA) identifiers and a container image, the workflow:

  1. researches each vulnerability from public sources;
  2. inspects the container's extracted root filesystem and runtime configuration;
  3. determines whether the vulnerable code is present, reachable, and exploitable; and
  4. produces cited Markdown reports and a structured Vulnerability Exploitability eXchange (VEX) classification.

Note

Generated reports are designed to support analyst review. For security decisions or policy gates, validate the cited evidence and classification in the context of your environment.

Table of Contents

Software components

The blueprint uses:

  • NVIDIA NeMo Agent Toolkit for workflow registration, CLI execution, evaluation, serving, and asynchronous jobs.
  • NVIDIA NIM-compatible inference. The default workflow uses nvidia/nemotron-3-ultra-550b-a55b for the primary agents and nvidia/nemotron-3.5-lightning-30b-a3b for focused CVE research subagents.
  • The NVIDIA AI-Q Blueprint aiq-agent package for the CVE Researcher's DeepAgents runtime, research middleware, model roles, and tool controls.
  • LangGraph for the staged pipeline graph.
  • Tavily and public vulnerability sources such as the CVE List, NVD, GitHub Advisories, OSV, and vendor advisories.
  • crane for digest resolution and container rootfs extraction.
  • Docker Compose, FastAPI, and Dask for the supported local service deployment.

Target audience

This blueprint is intended for:

  • Security analysts and vulnerability-management teams evaluating scanner findings in container context.
  • Platform and DevSecOps engineers integrating container vulnerability analysis into CI/CD, exception, or policy workflows.
  • AI practitioners in cybersecurity applying NeMo Agent Toolkit and NVIDIA NIM to develop AI-assisted vulnerability research and exploitability analysis.

Use case description

Determining whether a documented vulnerability is exploitable in a specific container is labor-intensive. A scanner finding alone does not establish that the affected package is installed in the active environment, that the vulnerable code is reachable from the container entrypoint, or that required dependencies and runtime conditions exist.

The workflow reduces that manual burden by collecting cited vulnerability intelligence, inspecting the container itself, verifying packages and binaries, tracing reachability, and publishing a reviewable VEX decision. It is designed to preserve uncertainty: when the analysis cannot establish a usable result, it reports unknown with uncertain justification instead of silently treating the finding as safe.

How it works

  1. Materialize the image. The caller provides image_name, image_tag, and one or more CVE/GHSA identifiers. If filesystem_path is absent, the orchestrator resolves the immutable image digest and extracts a flattened rootfs with crane. A caller may instead provide an existing rootfs.
  2. Collect evidence in parallel. The CVE Researcher runs once per vulnerability while the Container Analyzer runs once per image. For CVE inputs, middleware checks the exact CVE List record before research; missing, rejected, unpublished, or unavailable lookups are preserved as prominent warnings while best-effort research and downstream analysis continue. The Container Analyzer isolates public web research from filesystem inspection, keeping proprietary or pre-release container information out of network-enabled research.
  3. Analyze exploitability. For each vulnerability with a completed research report, deterministic pre-triage checks installed packages, versions, distribution backports, binaries, and runtime environments. Clear-cut cases are resolved without deep-analysis LLM calls; cases requiring reachability or data-flow evidence proceed to tool-assisted analysis.
  4. Assign VEX. The VEX Categorizer applies a precedence-ordered decision process and records a structured reasoning trace.
  5. Publish evidence and decisions. The workflow saves a vulnerability research report for each identifier, one container report per image, and exploitability and VEX reports for each vulnerability–container pair; each VEX decision is available as Markdown and JSON. The returned PipelineResult includes their paths or service URLs and summarizes each vulnerability's VEX status, justification, and errors, along with the overall run outcome.

Per-vulnerability work fans out concurrently. A failure for one identifier does not discard usable sibling results; the aggregate result is partial when at least one vulnerability succeeds and at least one fails.

Key components

The architecture is a three-stage multi-agent workflow. Stage 1 has two parallel branches: the CVE Researcher and Container Analyzer. Their reports feed the Exploitability Analyzer in Stage 2, and the resulting exploitability report drives the VEX Categorizer in Stage 3.

Vulnerability analysis multi-agent system architecture

  • Workflow inputs: The two inputs shown on the left side of the diagram arrive together in a JSON request, as defined by PipelineInput.

    • CVE/GHSA findings: Supply one or more vulnerability identifiers, typically from a container scanner. Each identifier starts its own research branch and produces a separate exploitability assessment and VEX decision. Optional package and version hints can help disambiguate the affected component.
    • Container image: Supply the container repository and tag to analyze. The image is inspected once, and its container report and rootfs evidence are shared across every submitted vulnerability. A caller may instead provide an already extracted rootfs and image configuration.

  • NAT/LangGraph pipeline orchestrator: The registered PipelineWorkflow owns input validation, image preprocessing, and conditional graph execution.

    • Preprocessing: The orchestrator combines image_name and image_tag into the registry reference. When filesystem_path is absent, the image materializer resolves the immutable digest, uses crane to export a flattened OCI root filesystem, captures the image configuration, and caches the result by digest. A caller-provided rootfs bypasses extraction. The resulting rootfs is the authoritative source for installed packages, binaries, libraries, and configuration.
    • Graph execution: Stage 1 runs one CVE Researcher per identifier in parallel with one shared Container Analyzer per image. Successful reports advance to the Exploitability Analyzer and then the VEX Categorizer.
    • Operational controls: The orchestrator applies stage timeouts, retries, concurrency and rate limits, semantic report caching, and fail-safe unknown/uncertain results for unresolved vulnerabilities.

  • Stage 1A — CVE Researcher (AI-Q DeepAgents): The CVE Researcher plugin receives web-only tools and has no access to the container rootfs. Exact CVE inputs are validated against the public CVE List before model execution. Two context-isolated research subagents collect complementary evidence:

    • The advisory researcher uses vendor advisories, CVE/GHSA records, NVD, OSV, and patches to identify affected packages, vulnerable version ranges, fixed versions, and vulnerable functions or APIs.
    • The exploit-intelligence researcher investigates public exploits and proofs of concept, CISA KEV membership, FIRST EPSS signals, observed exploitation, and attack prerequisites, recording sources and UTC as-of dates.
    • The coordinating agent validates and publishes one cited Markdown vulnerability report for each identifier under <output-root>/vulnerability_reports/ (see Output Structure for the complete layout).

  • Stage 1B — Container Analyzer: The container_analyzer runs once per image and deliberately separates its two tool environments. Phase 1 may use web_search and web_fetch to research images from recognized public registries; it can be skipped for proprietary or pre-release images. Phase 2 receives the image configuration and only path-restricted read, list, grep, and glob tools for the extracted rootfs. It identifies the container type, packages, entrypoints, runtime environments, and security controls, then publishes one Markdown container security report under <output-root>/container_reports/.

  • Stage 2 — Exploitability Analyzer: The exploitability_analyzer evaluates each vulnerability against the shared container report and extracted rootfs to determine whether the affected code is present, reachable, and exploitable in that image. Deterministic pre-triage resolves clear package, version, and environment cases; ambiguous cases proceed to an LLM-driven investigation that invokes filesystem, binary, package, and security-control tools to examine reachability, attacker-controlled data flow, dependencies, and mitigations. The resulting exploitability reports are published to <output-root>/exploitability_reports/ and advance to VEX categorization.

  • Stage 3 — VEX Categorizer: The vex_categorizer reads the exploitability and container reports for each vulnerability–image pair. Its constrained reason tool records evidence at each step of a precedence-ordered decision flow: association, code presence, execution path, trigger conditions, and protective controls. The categorizer returns:

    • VEX status (VEXStatus definition):
      • affected — The vulnerable code is present and exploitable in the analyzed image.
      • not_affected — A blocking condition, such as an unaffected version, absent or unreachable code, missing prerequisites, or an effective protection, prevents exploitation in the analyzed image.
    • Justification (decision-category definitions): false_positive, code_not_present, code_not_reachable, requires_configuration, requires_dependency, requires_environment, compiler_protected, runtime_protected, perimeter_protected, mitigating_control_protected, or vulnerable.

  • Reports and analyst-facing result: The workflow writes the cited vulnerability, container, and exploitability reports as Markdown, then publishes each VEX decision as both Markdown and a JSON sidecar under <output-root>/vex_results/. The returned PipelineResult includes the resolved image digest, shared container result, per-vulnerability package identity, stage report paths or service URLs, VEX status and justification, errors, and the aggregate complete, partial, or failed outcome. The report API exposes these artifacts for analyst review without requiring clients to construct filenames.

NIM microservices

The NAT nim provider calls OpenAI-compatible NVIDIA inference endpoints. By default, host-side runs use https://integrate.api.nvidia.com/v1; Compose routes the same protocol through its local NGINX cache. Set NV_BASE_URL to use a compatible gateway or self-hosted endpoint.

Models are role-specific and can be overridden without editing Python. See Customizing the LLM models. Any model or prompt change must use a new stage-cache namespace so results from a previous policy are not reused.

Prerequisites

The primary Docker Compose path requires:

  • Git with submodule support.
  • Docker with the Compose plugin v2.24.0 or newer (docker compose version).
  • curl.
  • API keys for NVIDIA NIM inference and Tavily web research. See Obtain API keys for instructions.
  • At least 20 GiB of free local space for the validated Morpheus quick start.

Python, uv, a native C++ compiler, and a host-installed crane are required only for the Optional Local Development Setup. The application image includes them for the recommended Compose workflow.

Hardware requirements

When using NVIDIA-hosted inference, the vulnerability-analysis application does not require a local NVIDIA GPU. CPU and memory needs depend on container size, request concurrency, and the number of vulnerabilities. The Compose deployment starts conservatively with one async job slot and two Dask worker processes, each configured with a best-effort limit of 40% of detected memory. See Sizing the local Dask cluster before increasing concurrency.

Extracted root filesystems dominate application disk use. Provision at least 20 GiB free for the validated Morpheus example and additional space for each distinct image digest. Docker image layers and the deployment's proxy caches use separate Docker-managed storage. See Caching for cache locations, sizing guidance, inspection, and cleanup.

Self-hosted NIMs require supported NVIDIA GPUs, drivers, and the NVIDIA Container Toolkit. Exact GPU capacity depends on the selected NIM; consult the NVIDIA NIM support matrix for verified deployment profiles and hardware. The provided docker-compose.nim.yml overlay reserves four GPUs by default. See Customizing the LLM models and deploy/README.md for configuration and deployment guidance.

Operating system requirements

The recommended workflow runs in Linux containers through Docker Compose. Linux is the primary deployment platform. Docker Desktop can be used for local development on macOS, but self-hosted NIMs require a supported Linux/NVIDIA GPU environment. Host-side development instructions are provided for macOS and Linux; validate other platforms before production use.

API definition

The service generates its OpenAPI specification from the active NeMo Agent Toolkit and FastAPI configuration, keeping the published schema aligned with the configured workflow and installed dependencies. After starting the service with Docker Compose, use these paths relative to its base URL:

  • /docs for the interactive Swagger UI
  • /openapi.json for the machine-readable OpenAPI specification

The source-of-truth data contracts are PipelineInput, which defines the container image and one or more vulnerability identifiers to analyze, and PipelineResult, which returns per-vulnerability evidence and a complete, partial, or failed outcome. The custom VulnAnalysisFastApiWorker extends NAT's workflow API with preflight and Dask-aware readiness checks, report discovery and download, enriched asynchronous job progress, optional bearer-token authentication, and terminal webhooks.

See Optional manual HTTP server (nat serve) for the endpoint reference, request examples, and response semantics.

Getting started

The supported first-run experience is the supervised Docker Compose API. Run commands from the repository root.

Install system requirements

Install Git, Docker with Compose v2.24.0 or newer, and curl, then verify the installed versions:

git --version
docker compose version
curl --version

Obtain API keys

Two credentials are required for an end-to-end run:

  • NVIDIA API key — Generate a key from the NVIDIA API Catalog or an NVIDIA organization with access to the configured NIM. It is read from NVIDIA_API_KEY.
  • Tavily API key — Create a key from Tavily. It is read from TAVILY_API_KEY.

Use the skip_web_research input option when the Container Analyzer should not research a proprietary or pre-release image on the public web. The CVE Researcher will still use Tavily for public vulnerability research, so TAVILY_API_KEY still needs to be configured. For local runs, store both keys in the .env file described in Set up the environment file. For production deployments, supply them through your orchestrator's secret-management features or a dedicated secret manager so they remain outside source control.

If a registry requires authentication, log in with the credential appropriate for that registry. Self-hosted NIM images from NGC also require NGC access and Docker authentication.

Set up the workflow repository

Clone the repository and initialize the AI-Q submodule:

git clone https://github.com/NVIDIA-AI-Blueprints/vulnerability-analysis.git
cd vulnerability-analysis
git submodule update --init --recursive

The repository root contains configs and data symlinks to the packaged configuration and examples. All following commands assume the repository root as the working directory.

Set up the environment file

Copy the reviewed template:

cp .env.example .env

Note

The .env file is ignored by Git. Keep real API keys and client secrets out of commits.

Set at least:

NVIDIA_API_KEY=your-nvidia-api-key
NV_BASE_URL=https://integrate.api.nvidia.com/v1
TAVILY_API_KEY=your-tavily-api-key

Compose loads .env through deploy/compose.sh; the vuln-analysis CLI loads it automatically. Source it before invoking nat directly on the host.

Quick Start: Docker Compose API

This is the primary first-run workflow. It builds the application, starts the supervised API and its Jupyter/NGINX companions, checks readiness, and submits the reviewed three-vulnerability Morpheus fixture asynchronously.

Record the invoking host identity so bind-mounted reports and caches remain writable, then start the stack:

bash deploy/configure-host-identity.sh
bash deploy/compose.sh up --build -d --wait --wait-timeout 300
bash scripts/curl-with-body.sh http://localhost:26466/ready

The checked-in pipeline_input.json contains exactly the three reviewed cases from morpheus_2506_eval_test.csv. Preflight checks the input, local prerequisites, credentials, and configured models without inference or search work:

bash scripts/curl-with-body.sh -X POST http://localhost:26466/preflight \
  -H 'Content-Type: application/json' \
  --data @data/examples/pipeline_input.json

Submit the workflow through the asynchronous API:

bash scripts/curl-with-body.sh -X POST http://localhost:26466/v1/workflow/async \
  -H 'Content-Type: application/json' \
  --data @data/examples/pipeline_input.json

Copy the returned job_id and poll every 5–10 seconds until status is success, failure, or interrupted:

JOB_ID='paste-job-id-here'
bash scripts/curl-with-body.sh "http://localhost:26466/v1/workflow/async/job/$JOB_ID"

To follow workflow progress while the job runs, open a second terminal and stream the API logs:

bash deploy/compose.sh logs -f vuln-analysis-api

Press Ctrl+C to stop following the logs; the workflow continues running.

On terminal success, inspect the returned output.outcome, follow its report URLs, or list reports with GET /reports. Stop the stack when finished:

bash deploy/compose.sh down

See deploy/README.md for service operations and deploy/1_Deploy_CVE.ipynb for the executable deployment walkthrough.

Optional Local Development Setup

Use this path to develop, test, run NAT directly on the host, or execute the standalone notebook. Compose-only users can skip it.

Local development requires Python 3.13, uv 0.9.9 or newer, a C++ build toolchain for the transitive annoy dependency, and crane for automatic image extraction.

Before installing anything, check whether uv is already available and which version is active (the version check runs only if uv is found):

command -v uv && uv --version

If the reported version is 0.9.9 or newer, continue to the environment setup below. If no version is reported, choose your preferred installation method and use its New install command. If an older version is reported, use the Upgrade existing installation command for the method originally used.

  • Standalone installer

    New install:

    curl -LsSf https://astral.sh/uv/install.sh | sh

    Upgrade an existing installation with uv self update.

  • Homebrew: Install with brew install uv, or upgrade with brew upgrade uv.

  • pipx: Install with pipx install uv, or upgrade with pipx upgrade uv.

After installing or upgrading, rerun command -v uv && uv --version to verify the active installation and version.

Install crane with brew install crane on macOS or go install github.com/google/go-containerregistry/cmd/crane@latest on a host with Go. Install a native compiler (xcode-select --install on macOS or build-essential on Ubuntu/Debian), then create the locked environment:

c++ --version
git submodule update --init --recursive
uv venv --python 3.13 .venv
source .venv/bin/activate
uv sync --locked

For development tools, use uv sync --locked --group dev.

On a restricted CI host where the normal uv cache is not writable, redirect only the cache:

export UV_CACHE_DIR="${TMPDIR:-/tmp}/vulnerability-analysis-uv-cache"
mkdir -p "$UV_CACHE_DIR"

Do not use sudo uv; root-owned files in the checkout can break later host and container commands.

For direct NAT commands, load .env into the shell:

set -a
source .env
set +a
test -n "$NVIDIA_API_KEY"
test -n "$TAVILY_API_KEY"

Preparing Container Inputs

Every request supplies image_name, image_tag, and at least one validated CVE or GHSA identifier. The pipeline can extract the image automatically or inspect a rootfs that you already materialized.

Option A — Automatic (recommended)

Omit filesystem_path:

{
  "vulns": [
    {"vuln_id": "GHSA-53q9-r3pm-6pq6"},
    {"vuln_id": "GHSA-ph84-rcj2-fxxm"},
    {"vuln_id": "CVE-2024-9287"}
  ],
  "image_name": "nvcr.io/nvidia/morpheus/morpheus",
  "image_tag": "25.06-runtime",
  "output_dir": "outputs",
  "cache": true
}

The first run resolves the immutable digest, extracts the rootfs under ${VULN_ANALYSIS_IMAGE_CACHE:-.cache/vuln_analysis/images}, and reuses it later. Registry authentication comes from the standard Docker credential configuration.

Automatic extraction supports these optional controls:

Variable Default Purpose
VULN_ANALYSIS_MAX_CONCURRENT_EXTRACTIONS 1 Cross-process extraction slots per host
VULN_ANALYSIS_LOCAL_SCRATCH unset Node-local staging for registries or destination caches on slow network storage
VULN_ANALYSIS_CRANE_EXPORT_ATTEMPTS 4 Bounded export attempts for retryable failures

Use absolute cache and scratch paths when workers do not share a working directory.

Option B — Pre-extract with the standalone helper

Use the helper to warm the cache, debug extraction, or provide an existing rootfs:

uv run python scripts/extract_image.py nvcr.io/nvidia/morpheus/morpheus:25.06-runtime /tmp/img-cache

The command prints the resolved digest and rootfs path. Set filesystem_path to that rootfs, keep image_name and image_tag, and optionally set image_config_path to the adjacent config.json. Providing image_digest also enables semantic caching for container-dependent stages.

Running the Workflow

The same registered cve_pipeline runs through notebooks, nat run, the HTTP service, and nat eval.

From the quick start user guide notebook

Choose a notebook for an executable walkthrough of the workflow. The deployment and customization notebooks form a sequence, while the local quick start provides an independent path without Compose.

Notebook Best for What it covers
deploy/1_Deploy_CVE.ipynb First-time users and platform operators who want the supported Compose deployment. Configure the environment, build and start the services, submit synchronous and asynchronous requests, inspect reports and VEX results, and explore the HTTP API.
deploy/2_Customize_CVE.ipynb Developers and security practitioners adapting the workflow to their own images, vulnerabilities, models, or inference capacity. Build single- and multi-vulnerability inputs, work with private images or a pre-extracted rootfs, run through the API or local NAT, tune models and concurrency, generate standalone-agent inputs, and optionally run the evaluation fixture.
quick_start/quick_start_guide.ipynb Analysts and developers who prefer a local command-line workflow or want a focused end-to-end example. Validate local prerequisites, build a typed container input, run the complete pipeline, inspect stage results and VEX artifacts, and generate connected inputs for reproducing individual agents.

For the supported containerized walkthrough, start Compose as described in Quick Start: Docker Compose API, open http://localhost:8000, and run deploy/1_Deploy_CVE.ipynb from top to bottom. Continue with deploy/2_Customize_CVE.ipynb when you are ready to adapt the requests, models, or runtime settings.

For local development, complete the Optional Local Development Setup, activate the repository .venv, and open quick_start/quick_start_guide.ipynb. It uses the same stable input, artifact capture, and standalone-input generation as the command-line workflow documented below.

After a kernel restart, rerun the selected notebook from the beginning instead of reusing files left in /tmp.

From the command line

The following commands validate the included single-vulnerability input, then run the complete workflow. run-pipeline saves the normalized input, full NAT log, and validated result under .tmp/quick_start_v3/:

uv run vuln-analysis preflight --input-file data/examples/quick_start_input.json
uv run vuln-analysis run-pipeline \
  --pipeline-input data/examples/quick_start_input.json \
  --artifacts-dir .tmp/quick_start_v3

To run the same input with NAT directly, without saving the normalized input, run log, and validated result together:

uv run nat run --config_file=configs/config.yml \
  --input_file=data/examples/quick_start_input.json

PipelineResult.outcome is authoritative:

  • complete — every requested vulnerability produced a usable result; success is true.
  • partial — at least one vulnerability succeeded and at least one failed; success is false.
  • failed — no vulnerability produced a usable result; success is false.

The wrapper and server convert semantic failure into a non-success process or HTTP contract while preserving the structured result for diagnosis.

Preflight

Preflight check validates the complete input, credential variables, provider URLs, local paths, crane availability, NVIDIA/Tavily credentials, and configured model availability. It performs no inference or search work.

uv run vuln-analysis preflight --input-file data/examples/quick_start_input.json
uv run vuln-analysis preflight --input-file data/examples/quick_start_input.json --json

Exit code 0 means ready, 1 means a prerequisite failed, and 2 means the input could not be read or validated. Run it immediately before uncached work.

Individual Agents

Run an agent by itself to reproduce or debug one stage, inspect its report before continuing, or compare VEX policies without rerunning unrelated work. Each standalone configuration invokes exactly one agent; it does not run that agent's upstream or downstream stages.

Complete the Optional Local Development Setup and load .env before using these commands. Every agent uses the configured NIM endpoint. The CVE Researcher also uses Tavily for public vulnerability research; the Container Analyzer uses Tavily only when public-image web research is enabled.

Agent Use it to Input required
CVE Researcher Research or reproduce one vulnerability independently of any container One cve_id or ghsa_id; optional package_hint and version_hint
Container Analyzer Inspect an image independently of any vulnerability image_name, image_tag, and an extracted rootfs in filesystem_path; optional image_config_path and skip_web_research
Exploitability Analyzer Reassess one vulnerability against one container without repeating research or image analysis vulnerability_report_path, container_report_path, and the extracted rootfs in filesystem_path
VEX Categorizer Reclassify an existing exploitability result or compare decision policies exploitability_report_path and container_report_path

The checked-in cve_researcher_input.json is self-contained and runnable. The checked-in container_analyzer_input.json, exploitability_analyzer_input.json, and vex_categorizer_input.json illustrate their schemas but contain placeholder paths.

For runnable inputs connected to an existing pipeline run, use generate-standalone-inputs. If you already completed the command-line example above, run only the final command:

uv run vuln-analysis preflight --input-file data/examples/quick_start_input.json
uv run vuln-analysis run-pipeline \
  --pipeline-input data/examples/quick_start_input.json \
  --artifacts-dir .tmp/quick_start_v3
uv run vuln-analysis generate-standalone-inputs \
  --pipeline-input .tmp/quick_start_v3/pipeline_input.json \
  --pipeline-result .tmp/quick_start_v3/pipeline_result.json \
  --output-dir .tmp/quick_start_v3/standalone_inputs \
  --workspace-root .

For a multi-vulnerability result, add --vuln-id with an identifier present in pipeline_result.json. The generator always writes CVE Researcher and Container Analyzer inputs. It adds an Exploitability Analyzer input when successful vulnerability and container reports are available, and a VEX Categorizer input when a successful exploitability report is also available.

The agents appear below in pipeline order, but generated inputs already reference the pipeline's saved artifacts. Run only the agent you need; one standalone result does not update another agent's input.

1. Run the CVE Researcher

Use the self-contained example directly, or substitute the generated cve_researcher_input.json to reproduce the vulnerability selected from a pipeline result:

uv run nat run --config_file=configs/config-cve-researcher.yml \
  --input_file=data/examples/cve_researcher_input.json

2. Run the Container Analyzer

The Container Analyzer does not require a vulnerability report. To run it without prior pipeline artifacts, first extract the image, then replace filesystem_path and image_config_path in container_analyzer_input.json with the helper's rootfs_path and config_path. Otherwise, use the generated input:

uv run nat run --config_file=configs/config-container-analyzer.yml \
  --input_file=.tmp/quick_start_v3/standalone_inputs/container_analyzer_input.json

3. Run the Exploitability Analyzer

Use a generated input, or author one that points to valid vulnerability and container reports plus the extracted rootfs:

uv run nat run --config_file=configs/config-exploitability-analyzer.yml \
  --input_file=.tmp/quick_start_v3/standalone_inputs/exploitability_analyzer_input.json

4. Run a VEX Categorizer

Run the standard decision policy:

uv run nat run --config_file=configs/config-vex-categorizer.yml \
  --input_file=.tmp/quick_start_v3/standalone_inputs/vex_categorizer_input.json

To compare the relaxed policy against the same reports, run the custom categorizer with the same input:

uv run nat run --config_file=configs/config-custom-vex-categorizer.yml \
  --input_file=.tmp/quick_start_v3/standalone_inputs/vex_categorizer_input.json

The final two commands intentionally share one input. They invoke separate tools: vex_categorizer applies the standard decision policy, while custom_vex_categorizer applies the relaxed policy.

Each command prints a typed JSON result to the terminal. Common fields include success, error, retryable, report_content, tool_call_count, and llm_call_count. The CVE Researcher adds identifier and validation metadata; the Container Analyzer adds phase1_skipped; the Exploitability Analyzer adds verdict, confidence, and final-validation metadata; and the VEX Categorizer adds status, justification, and publishable reasoning. report_content contains the generated Markdown.

Standalone configurations do not write the pipeline's report files or VEX JSON sidecar. Use the complete pipeline when you need those published artifacts.

Optional manual HTTP server (nat serve)

The Compose quick start already hosts the service. Use this host-side command only after completing Optional Local Development Setup, and do not run it while Compose owns port 26466:

uv run nat serve --config_file=configs/config.yml --host 0.0.0.0 --port 26466 \
  --max_running_async_jobs=1 --dask_worker_memory_limit=0.4

nat serve starts a local Dask cluster for asynchronous processing. Monitor its scheduler, workers, and tasks at http://localhost:8787/status. The Compose deployment publishes the same dashboard through WORKFLOW_HOST_DASK_PORT, which defaults to 8787.

The shipped service supports:

Method and path Purpose
POST /preflight Validate input, local prerequisites, credentials, and configured models
POST /generate Run synchronously
POST /v1/workflow/async Submit a background job and return job_id
GET /v1/workflow/async/job/{job_id} Poll status, progress, and the terminal result
GET /health Process liveness
GET /ready API, Dask scheduler, and worker readiness
GET /reports Discover report metadata and URLs
GET /reports/{group}/{report_type}/{filename} Read or download one report
GET /docs Generated OpenAPI documentation

The request body for workflow endpoints is the same PipelineInput JSON accepted by nat run.

Synchronous request (/generate)

bash scripts/curl-with-body.sh -X POST http://localhost:26466/generate \
  -H 'Content-Type: application/json' \
  --data @data/examples/quick_start_input.json

Complete and partial results return HTTP 200 with X-Workflow-Outcome; failed analysis returns HTTP 422 with the full structured result. Uncaught server exceptions use a workflow_error envelope.

Asynchronous jobs (/v1/workflow/async)

Use async submission for automated and multi-CVE workloads. Run /preflight first, then submit the same PipelineInput body to POST /v1/workflow/async. The response returns a job_id and current status, allowing the client to release the submission connection while Dask queues and runs the analysis.

Poll GET /v1/workflow/async/job/{job_id} until status is success, failure, or interrupted; submitted and running are nonterminal. When available, progress reports image-extraction activity and per-stage item counts for the CVE Researcher, Container Analyzer, Exploitability Analyzer, and VEX Categorizer. At a terminal state:

  • success includes a structured output; inspect output.outcome for complete or partial.
  • For failure, inspect error. If the workflow produced a structured result with no usable vulnerability result, that result is preserved in output with outcome: failed; a failure before the workflow produces a PipelineResult might not have output.
  • interrupted means the job ended before producing a terminal analysis result. Inspect error, confirm /ready and /preflight, then decide whether to resubmit.

Do not plan around a fixed runtime. An uncached request can require image export and rootfs extraction, public vulnerability research, container inspection, package and binary analysis, and multiple model calls for every vulnerability. Image size, vulnerability count, model and provider latency, tool behavior, rate limits, and configured concurrency can all change the duration substantially.

Caching can reduce repeat work substantially. The digest-keyed image cache avoids extracting the same immutable image again, while the semantic stage cache reuses successful reports when the inputs, policy namespace, and upstream report content still match. Image, vulnerability, or upstream-evidence changes can produce a cache miss. Model, prompt, tool, or decision-policy changes must use a new stage-cache namespace so the affected work runs again instead of reusing an incompatible report.

Async submission does not make a single analysis inherently faster. It makes long or variable-duration work easier to operate: clients avoid holding one HTTP request open, can poll durable status and progress, can receive an optional terminal webhook, and can let Dask queue or run multiple jobs up to the configured worker capacity.

Sizing the local Dask cluster

With process workers and one thread per worker, NAT creates max_running_async_jobs + 1 worker processes. dask_worker_memory_limit applies per worker and is best-effort, not a container memory cap. Leave memory for FastAPI, the scheduler, Docker, and the host. Set VULN_ANALYSIS_MAX_RUNNING_ASYNC_JOBS and VULN_ANALYSIS_DASK_WORKER_MEMORY_LIMIT in .env, recreate the API service, and validate representative jobs before increasing concurrency.

Pushing job completion to a consumer (webhooks)

Optional terminal webhooks are disabled by default. Enable them with VULN_ANALYSIS_WEBHOOK_ENABLED, set VULN_ANALYSIS_WEBHOOK_URL, and configure optional OAuth2 client-credentials and HMAC signing through the remaining VULN_ANALYSIS_WEBHOOK_* variables in .env.example. Delivery uses a stable event ID, a durable SQLite ledger, and bounded retries. Client-supplied callback_context is echoed verbatim for correlation. See docs/webhook-schema.md for the canonical envelope.

Serving as an authenticated service (OAuth2 / JWT bearer)

Inbound authentication is disabled by default. Set VULN_ANALYSIS_AUTH_ENABLED=true plus issuer, audience, optional scopes, JWKS URI, and allowed algorithms to validate OAuth2/OIDC bearer tokens. /health, /ready, and documentation paths remain public by default; workflow, preflight, and report endpoints are protected. See .env.example for every VULN_ANALYSIS_AUTH_* setting.

Report download endpoints

Discover reports before fetching them:

bash scripts/curl-with-body.sh http://localhost:26466/reports
bash scripts/curl-with-body.sh --get http://localhost:26466/reports \
  --data-urlencode 'vulnerability=GHSA-53q9-r3pm-6pq6' \
  --data-urlencode 'container=morpheus_25.06-runtime' \
  --data-urlencode 'report_type=vex_results'

Known aliases resolve to the same reports, so no canonical filename needs to be guessed. Vulnerability entries include submitted and resolved CVE/GHSA aliases; VEX entries include structured status fields and a json_url; clients should follow returned URLs instead of constructing filenames.

Filters can be combined and use intersection semantics. Image-wide Container Analyzer reports are excluded from vulnerability-filtered results because they do not carry a vulnerability identity; use the container filter to find them. Append ?download=true to a returned report URL to force download. VULN_ANALYSIS_REPORTS_ROOT can expose reports produced by an earlier nat run without rerunning analysis.

Output Structure

The default output_dir layout is:

outputs/
├── vulnerability_reports/       # CVE Researcher Markdown
├── container_reports/           # Container Analyzer Markdown
├── exploitability_reports/      # Exploitability Analyzer Markdown
├── vex_results/                 # VEX Markdown and JSON sidecars
└── failed_reports/              # Rejected intermediate reports, when present

A submitted GHSA can resolve to a canonical CVE, so downstream filenames may use another identifier. Use the report_url fields or GET /reports?vulnerability=<submitted-id> instead of constructing paths.

Customizing the Workflow

For an executable walkthrough, run deploy/2_Customize_CVE.ipynb after deploy/1_Deploy_CVE.ipynb. It creates single- and multi-vulnerability requests, shows private-image and pre-extracted-rootfs inputs, runs custom inputs locally or through HTTP, builds a rate-limited config, and generates standalone-agent inputs.

Customizing the container input

Copy an example so the checked-in input remains unchanged:

mkdir -p .tmp
cp data/examples/quick_start_input.json .tmp/custom_pipeline_input.json

Edit .tmp/custom_pipeline_input.json for the analysis you want:

  • Put one or more entries in vulns; each requires vuln_id and may include package_hint and version_hint.
  • Set image_name to the repository without a tag and put the tag in image_tag.
  • Omit filesystem_path for automatic extraction. For an existing rootfs, follow Option B and set filesystem_path; optionally add image_config_path and image_digest.
  • Set skip_web_research: true for a private image. This skips only the Container Analyzer's public-image research; the CVE Researcher still uses Tavily for vulnerability research.
  • Set output_dir to group the reports. Set cache: false to bypass all stage caches, or override individual stages with cache_cve_researcher, cache_container_analyzer, cache_exploitability_analyzer, or cache_vex_categorizer.

Validate the edited request with uv run vuln-analysis preflight --input-file .tmp/custom_pipeline_input.json, then submit it through the command-line or HTTP path.

Customizing the LLM models

Set DEFAULT_MODEL_NAME in .env to change the fallback model, or set CVE_RESEARCHER_MODEL, CVE_SUBAGENT_MODEL, CONTAINER_ANALYZER_MODEL, EXPLOITABILITY_ANALYZER_MODEL, and VEX_CATEGORIZER_MODEL for individual roles. Set NV_BASE_URL when using another OpenAI-compatible NVIDIA endpoint. Run preflight to confirm that the selected endpoint exposes each configured model.

Environment changes take effect when the process starts. Reload .env before host-side nat commands, restart a notebook kernel, or recreate the Compose API with bash deploy/compose.sh up -d vuln-analysis-api. To limit load on an inference endpoint, copy configs/config.yml and edit workflow.max_concurrency and workflow.llm_max_rate, as demonstrated in the customization notebook. Whenever a model, prompt, tool set, or decision policy changes, assign a new workflow.cache_namespace; use workflow.cache_stage_namespaces.<stage> when only one stage changed.

Customizing agent behavior and VEX policy

Use configs/config.yml for the standard VEX decision policy or configs/config-relaxed-vex.yml for the checked-in relaxed policy:

uv run nat run --config_file=configs/config-relaxed-vex.yml \
  --input_file=.tmp/custom_pipeline_input.json

For another VEX policy, copy the relaxed config and its file-backed prompt, update functions.vex_categorizer.prompt, and give workflow.cache_stage_namespaces.vex_categorizer a new value. In a copied full-workflow config, other agents' tool lists and iteration limits are under functions.

Customizing the output and integrations

Set output_dir in each request to choose where that run writes reports. The report API serves the workflow's configured output directory by default; set VULN_ANALYSIS_REPORTS_ROOT before starting the service to expose a different existing report tree. Keep request output directories inside that served root when clients need report URLs and downloads.

For event-driven consumers, add an opaque callback_context object to the request and configure terminal webhooks; the same object is echoed in the webhook. Enable OAuth2/JWT bearer validation before exposing the service outside a trusted local environment.

Evaluating the Workflow

Evaluation is the regression-testing layer for this agentic workflow. It runs the same cve_pipeline against reviewed labels and measures whether changes to models, prompts, tools, or orchestration preserve expected behavior. The evaluation pipeline uses NeMo Agent Toolkit evaluation, a custom CSV dataset parser, and accuracy evaluators defined in this package.

The checked-in morpheus_2506_eval_test.csv contains only three reviewed cases—two code_not_reachable and one vulnerable—for nvcr.io/nvidia/morpheus/morpheus:25.06-runtime. It is a smoke/regression fixture, not a statistically representative benchmark. Teams should replace or extend it with a representative, independently reviewed dataset before drawing performance conclusions.

Running Evaluation (Overview)

Evaluation is intended for local development and research. Complete Optional Local Development Setup, then run these commands from the repository root:

uv run vuln-analysis preflight --input-file data/examples/pipeline_input.json
uv run nat eval --config_file=configs/config-eval.yml --deadline-seconds 3600

Preflight checks credentials, model access, crane, and input paths before the more costly evaluation. It does not pull the image, so authenticate to the container registry separately when required. nat eval runs the workflow and evaluators locally using configs/config-eval.yml; no Compose stack or HTTP service is needed. The default dataset omits rootfs paths, so uncached images are pulled and extracted automatically. For an interactive example, see the opt-in evaluation cell in the customization notebook.

Evaluation artifacts are written to .tmp/evaluators/ by default. They include the workflow output, one JSON file per configured evaluator, and eval_summary.json. Increase --deadline-seconds if a representative uncached run needs more than the configured 3,600 seconds.

Independent rows continue when one row fails, but the command exits nonzero if the deadline expires, evaluation raises, evaluator output is missing or malformed, or any evaluation case fails. A usable incomplete run records status: "partial", so CI cannot mistake incomplete evaluation for success.

Inputting Evaluation Datasets

Evaluation datasets are CSV files under data/eval_datasets/. Each row represents one vulnerability/container pair and requires vuln_id, image_name, image_tag, and ground_truth_label. Optional filesystem_path and image_config_path columns select a pre-extracted rootfs; leaving them empty uses automatic image extraction.

Select another dataset without editing the checked-in config:

uv run nat eval --config_file=configs/config-eval.yml \
  --override eval.general.dataset.file_path data/eval_datasets/morpheus_2506_eval_test.csv

The parser named by eval.general.dataset.function converts each row into a single-vulnerability PipelineInput and passes the ground-truth justification to the evaluators. To use another file shape, register a compatible NAT custom dataset parser and update that function path.

Accuracy Evaluators

The accuracy evaluator reports exact VEX-justification accuracy, binary vulnerable/not-vulnerable accuracy, precision, recall, F1, and a confusion matrix. Because unknown/uncertain is a valid fail-safe rather than a VEX conclusion, config-eval.yml evaluates three explicit policies:

  • accuracy_exclude removes uncertain predictions from scored cases.
  • accuracy_wrong counts every uncertain prediction as incorrect.
  • accuracy_vulnerable maps uncertain predictions to the conservative vulnerable class.

Each evaluator writes <evaluator-name>_output.json with aggregate metrics and per-case reasoning. Keep all three policies visible when comparing changes; reporting only the most favorable treatment can hide regressions in workflow completion.

Configuration

Environment Variables

Variable Required Purpose
NVIDIA_API_KEY Yes Credential for NIM-compatible inference
NV_BASE_URL No Inference endpoint; direct default is https://integrate.api.nvidia.com/v1
TAVILY_API_KEY Yes Credential for web research
TAVILY_API_BASE_URL No Compatible Tavily endpoint/proxy
DEFAULT_MODEL_NAME and per-agent model variables No Model selection
VULN_ANALYSIS_IMAGE_CACHE No Extracted rootfs cache root
VULN_ANALYSIS_STAGE_CACHE No Semantic stage-report cache root
CACHE_ENABLED No Default cache behavior when input omits cache
VULN_ANALYSIS_MAX_RUNNING_ASYNC_JOBS No Compose async-job capacity
VULN_ANALYSIS_DASK_WORKER_MEMORY_LIMIT No Per-worker best-effort Dask memory limit
VULN_ANALYSIS_AUTH_* No Inbound JWT validation
VULN_ANALYSIS_WEBHOOK_* No Terminal webhook delivery

See .env.example for extraction, identity, auth, and webhook controls.

Concurrency & Rate Limiting

workflow.max_concurrency bounds simultaneous agent tasks per fan-out stage; null or 0 is unlimited. workflow.llm_max_rate is a shared no-burst request-per-second limit; null disables it. A function-level llm_max_rate overrides the workflow setting. Evaluation also has eval.general.max_concurrency for independent dataset rows.

Caching

The application maintains:

  • a digest-keyed image cache under ${VULN_ANALYSIS_IMAGE_CACHE:-.cache/vuln_analysis/images}; and
  • successful semantic stage reports under ${VULN_ANALYSIS_STAGE_CACHE:-.cache/vuln_analysis/stages}.

The stage cache is independent of output_dir: a cache hit is materialized into the requested output group. Cache keys include semantic inputs, effective policy namespaces, and upstream report hashes while excluding presentation and runtime controls. A caller-provided rootfs needs image_digest to cache container-dependent stages safely.

Compose also caches repeated NIM and Tavily traffic through its nginx-cache proxy. These caches live in Docker volumes, separate from the application image and stage caches, and vuln-analysis cache prune does not remove them. See deploy/README.md for inspection and full-reset instructions.

For the validated Morpheus run, keep at least 20 GiB of free local space.

Inspect managed caches:

uv run vuln-analysis cache info
uv run vuln-analysis cache info --json

Pruning is always a dry run unless --yes is provided. Stop active workflows before execution:

uv run vuln-analysis cache prune --images --image-digest sha256:<64-hex-digest>
uv run vuln-analysis cache prune --images --image-digest sha256:<64-hex-digest> --yes
uv run vuln-analysis cache prune --stages
uv run vuln-analysis cache prune --stages --yes
uv run vuln-analysis cache prune --all

Image pruning removes recognized digest directories; stage pruning removes recognized cached reports and metadata. Neither removes user-facing outputs, async job state, unknown files, active locks, or Docker volumes.

Cite

Please consider citing our paper when using this code in a project. You can use the following BibTeX:

@inproceedings{zemicheal2024llm,
  title={LLM agents for vulnerability identification and verification of CVEs},
  author={ZeMicheal, Tadesse and Chen, Hsin and Davis, Shawn and Allen, Rachel and Demoret, Michael and Song, Ashley},
  booktitle={Proceedings of the Conference on Applied Machine Learning in Information Security (CAMLIS 2024)},
  pages={161--173},
  year={2024},
  publisher={CEUR Workshop Proceedings},
  volume={3920},
  url={https://ceur-ws.org/Vol-3920/}
}

Note

The paper describes the earlier workflow architecture represented by the 2.0.0 release. The current implementation builds on the same vulnerability-analysis concepts but uses a different architecture. Refer to this README and the current code for technical and operational details.

License

The source code in this repository is available under the Apache License 2.0. NVIDIA services, models, and containers used with the blueprint may have additional governing terms.

About

Rapidly identify and mitigate container security vulnerabilities with generative AI.

Resources

Readme
Apache-2.0 license

Security policy

Security policy
Activity
Custom properties

Stars

241 stars

Watchers

12 watching

Forks

109 forks
Report repository

Releases

Packages

Contributors

Languages