threat_modeling.md

🕵️ AI Threat Modeling

🎯 Overview

Threat modeling for AI systems shifts from traditional "STRIDE" (Spoofing, Tampering, Repudiation, Information Disclosure, Denial of Service, Elevation of Privilege) to include Cognitive and Behavioral Threats.

---

🏗️ The AI Threat Model Framework

1. Identify Assets

• The Model: Weights, architecture, system prompts.
• The Data: Training sets, RAG documents, user conversation history.
• The Infrastructure: Vector DBs, API endpoints, orchestration logic.
• The Agency: Permissions to execute code, read files, or send emails.

2. Map Trust Boundaries

graph LR
    User((User))
    subgraph Trust_Zone_1[Untrusted]
        Input[Input API]
    end
    subgraph Trust_Zone_2[Internal]
        Orchestrator[Orchestrator]
        LLM[Model]
        VectorDB[(Vector DB)]
    end
    subgraph Trust_Zone_3[Privileged]
        Tools[Exec Tools]
    end
    
    User -- Prompts --> Input
    Input -- Sanitized? --> Orchestrator
    Orchestrator -- Retrieval --> VectorDB
    VectorDB -- Context --> Orchestrator
    Orchestrator -- Inference --> LLM
    LLM -- Action Request --> Tools
    
    style Trust_Zone_1 fill:#ffcccc
    style Trust_Zone_2 fill:#ccffcc
    style Trust_Zone_3 fill:#ccccff

Loading

---

🧨 AI-Specific Threat Categories

A. Semantic Manipulation

• Description: Crafting inputs that exploit the model's linguistic understanding.
• Example: Roleplay attacks ("You are now a Linux terminal...").

B. Supply Chain Poisoning

• Description: Malicious data or code injected via third-party libraries or models.
• Example: A poisoned HuggingFace adapter that leaks data when a specific keyword is used.

C. Resource Exhaustion (AI-DoS)

• Description: Overwhelming the system with computationally expensive queries.
• Example: Multi-modal inputs designed to maximize token consumption or inference time.

---

📊 Threat Scoring (DREAD for AI)

Category	High Risk (5)	Low Risk (1)
Damage	Unauthorized fund transfer	Wrong text formatting
Reproducibility	100% success with static payload	Stochastic (random) success
Exploitability	No technical skill required	Requires deep ML knowledge
Affected Users	All users	Single specific session
Discoverability	Publicly documented bypass	Deeply hidden logic bug

---

🛠️ Countermeasure Strategy

Tip

Defense in Depth: Never rely on a single system prompt. Implement external guardrails (e.g., LlamaGuard), input sanitizers, and output monitors.

1. Hardened System Prompts: Use XML-like tags to separate data from instructions.
2. Context Isolation: Ensure RAG data cannot override system directives.
3. Human-in-the-Loop (HITL): Require approval for high-privilege agent actions.