Prisma AIRS - AI Model Security for Azure DevOps

Scan AI/ML model artifacts with Prisma AIRS AI Model Security inside Azure DevOps, and fail the build when a model is BLOCKED (pickle code-execution gadgets, unsafe operators, unapproved formats, license issues, and so on).

This is the Azure DevOps companion to the GitHub Actions, GitLab, and Jenkins integrations. Everything here was validated end to end against a live AIRS tenant and a clean Ubuntu/Python 3.12 environment.

---

TL;DR - the two things that trip everyone up

1. model-security-client requires Python >=3.11,<3.14. Microsoft-hosted ubuntu-latest (22.04) defaults to Python 3.10, which cannot install the SDK at all. Always select 3.12 with UsePythonVersion@0 (already done in the YAML here). This is the #1 reason a "simple" ADO pipeline fails.

2. A self-hosted Ubuntu agent also needs python3-venv if anything tries to build a virtualenv (sudo apt-get install -y python3-venv). Stock Ubuntu ships Python without the venv module wired up, so python3 -m venv aborts with an opaque CalledProcessError. The scripts here avoid venvs in CI, and airs_artifact_scan.py --auto-install now prints this exact remediation instead of crashing.

A third, subtler one: this is AI Model Security, not AIRS Runtime Security. They are different products with different SDKs. See "Model Security vs Runtime Security".

---

What's in here

File	Purpose
azure-pipelines.yml	Pattern A - gate a repo/build: scan models under models/, fail on BLOCKED.
azure-pipelines-feed-scan.yml	Pattern B - scan an existing Azure Artifacts feed of models on a schedule.
pipelines/azure-pipelines-selfhosted-demo.yml	Self-hosted-agent demo variant - scans a benign then a malicious model in one run (used for the live end-to-end proof).
airs_artifact_scan.py	The scanner. Downloads feed packages (or reads a local dir) and scans each with the real model-security CLI. Exit 1 on BLOCKED.
getPYPIurl.sh	Exchanges your SCM service-account creds for the private PyPI index URL.
test-models/make_test_models.py	Generates a benign model and a known-malicious pickle to prove the gate works.
requirements.txt	The one dependency (model-security-client), intentionally unpinned to pull latest.

---

Prerequisites

• A Prisma AIRS AI Model Security license, an SCM service account (client id + secret), your TSG ID, and a Security Group. Because feed/local files are scanned from disk, use a Security Group whose Source Type = Local.
• An Azure DevOps project, and a pipeline agent. Microsoft-hosted ubuntu-latest works (the YAML pins Python 3.12). Self-hosted Linux works too - see the venv note above.

Get your Security Group UUID:

model-security list-security-groups | jq -r '.security_groups[] | select(.source_type=="LOCAL") | "\(.name)  \(.uuid)"'

---

Pipeline variables

Create these as pipeline variables, or (recommended) in a variable group named prisma-airs-model-security and uncomment the - group: line in the YAML:

Name	Kind	Notes
MODEL_SECURITY_CLIENT_ID	variable	SCM service-account client id
MODEL_SECURITY_CLIENT_SECRET	secret	SCM service-account client secret
TSG_ID	variable	Tenant Service Group id
SECURITY_GROUP_UUID	variable	LOCAL-source-type Security Group UUID
MODEL_SECURITY_API_ENDPOINT	variable (optional)	defaults to https://api.sase.paloaltonetworks.com/aims

---

Pattern A - gate a repo / build (azure-pipelines.yml)

Put your models under models/ (committed, or restored as a build artifact in an earlier step), point the pipeline at this repo, and set the variables above. On every run it installs the latest model-security-client, scans models/, publishes a JSON report artifact, and fails the build if any model is BLOCKED.

Pattern B - scan an Artifacts feed (azure-pipelines-feed-scan.yml)

For a customer who already publishes models as Universal Packages into an Azure Artifacts feed. Set FEED_NAME, grant the build service Feed Reader, and it scans every package in the feed daily. This is the corrected version of the original POV script.

Running the scanner directly

# Scan a local directory of models
python airs_artifact_scan.py --model-dir ./models --security-group-uuid <LOCAL-SG-UUID>

# Scan every Universal Package in a feed (latest version of each)
export AZURE_DEVOPS_EXT_PAT=<PAT with Packaging:read>
python airs_artifact_scan.py \
  --org https://dev.azure.com/MyOrg --project MyProject --feed MyModelFeed \
  --security-group-uuid <LOCAL-SG-UUID> --block-on-errors --report report.json

# Scan one package/version
python airs_artifact_scan.py --org ... --project ... --feed ... \
  --package my-model --version 2.1.0 --security-group-uuid <LOCAL-SG-UUID>

Exit codes: 0 all ALLOWED, 1 at least one BLOCKED (stop the build), 2 configuration/tooling/scan error.

---

Prove it works (60-second demo)

# 1. Generate a benign model and a malicious pickle
python test-models/make_test_models.py

# 2. Benign -> ALLOWED, exit 0
python airs_artifact_scan.py --model-dir test-models/out/benign-model -sg <LOCAL-SG-UUID>; echo "exit $?"

# 3. Malicious -> BLOCKED, exit 1, with the detected gadget
python airs_artifact_scan.py --model-dir test-models/out/malicious-model -sg <LOCAL-SG-UUID>; echo "exit $?"

Expected on the malicious model:

[BLOCKED] malicious-model  5/7 rules passed
          - PAIT-PKL-100: Load Time Code Execution Check (posix.system) in model.pkl
          - UNAPPROVED_FORMATS: Stored In Approved File Format in model.pkl
RESULT: BLOCKED - one or more models failed the Security Group. Build stopped.

---

Model Security vs Runtime Security - do not confuse them

These are two different products:

	AI Model Security (this repo)	AIRS Runtime Security
Package	model-security-client (private PyPI) + model-security CLI	pan-aisecurity (public PyPI)
What it inspects	the model binary (static analysis)	prompts/responses of a running app
Catches	pickle RCE, unsafe operators, bad formats, license	prompt injection, DLP, toxic content, URLs
Verdict field	eval_outcome: ALLOWED / BLOCKED	action: allow / block
Concept	Security Group + rules	Security Profile + custom topics

A naive integration can use pan-aisecurity to scan a text description of each model ("this is a .pkl, pickle is risky") and send it to the runtime content scanner. That returns ALLOW and detects nothing real - the actual pickle exploit sails straight through. Model scanning must inspect the artifact bytes, which is what the model-security CLI does.

---

Gotchas found while building this (field notes)

• Python 3.10 cannot install the SDK. >=3.11,<3.14 required. Pin 3.12.
• python3-venv missing on stock Ubuntu breaks any venv bootstrap.
• Relative --model-path can return a false BLOCKED in model-security 1.0.1 and 1.1.0 - the scan fails closed on a path it cannot resolve. Always pass absolute paths. airs_artifact_scan.py resolves them for you.
• --silent suppresses the result JSON, not just logs. Use --log-level critical when you need to parse stdout.
• Latest client is 1.1.0 (Linux). The install here is unpinned, so CI always gets latest. (On macOS/arm64 the newest index-visible build may be older; that affects local laptops only, not Linux CI agents.)

---

Troubleshooting

Symptom	Cause / fix
ERROR: Packages require a different Python. 3.10.x not in '<3.14,>=3.11'	Add UsePythonVersion@0 with versionSpec: '3.12' before installing.
subprocess.CalledProcessError ... '-m', 'venv'	sudo apt-get install -y python3-venv (or python3.10-venv).
Failed to obtain SCM access token	Bad MODEL_SECURITY_CLIENT_ID/SECRET or TSG_ID.
Failed to retrieve PyPI URL	Check MODEL_SECURITY_API_ENDPOINT; confirm the service account has Model Security access.
Benign model scans as BLOCKED	You passed a relative path to the CLI. Use absolute (the script handles this).
az artifacts universal download fails	Build service needs Feed Reader; map AZURE_DEVOPS_EXT_PAT: $(System.AccessToken).
Feed listing returns 0 packages	Confirm org/project/feed names and that the PAT has Packaging: read.

---

Reference implementation for field use. Validated end to end on 2026-06-03: against a live AIRS tenant, in a clean Ubuntu/Python 3.12 container, and in a real Azure DevOps pipeline on a self-hosted agent - benign model passed, a malicious pickle was BLOCKED (PAIT-PKL-100) and failed the build, and the JSON report was published as a build artifact. Pair with the runtime/red-teaming pipeline integrations for full MLSecOps lifecycle coverage: scan the artifact at build, red-team the running target at deploy.

---

Companion repositories

Use these together across the MLSecOps lifecycle - scan the artifact at build, red-team the running target at deploy:

• model-security-pipeline-integration - the GitHub Actions version of model scanning
• red-teaming-pipeline-integration - automated AI red teaming of deployed targets

Contact

Scott Thornton - AI Security Researcher

• Website: perfecxion.ai
• Email: scott@perfecxion.ai
• LinkedIn: linkedin.com/in/scthornton
• ORCID: 0009-0008-0491-0032
• GitHub: @scthornton

Security issues: please report via SECURITY.md