extra tools (#72)

* extra tools

* remove latex

* size fixes

* size fixes

* tweaks

* report fixes

* fixes

* add dedupe

* fix temporario bug

* fix temporario bug

* add azure

* add validation steps to report

* awesome webs

* fixes

* openai fixes

---------

Co-authored-by: QA Team <qa@bugcatcher.com>

Author: SimonGurney

.github/workflows/build_pr.yml

@@ -1,4 +1,4 @@
-name: Build and publish release on new tag
+name: Build and publish PR image
 on:
   pull_request:
     types: [opened, synchronize, reopened]
@@ -20,19 +20,11 @@ jobs:
       - name: Set up Docker Buildx
         id: buildx
         uses: docker/setup-buildx-action@v3
-      - name: Build and push SAIST-Lite
-        run: |
-          docker buildx build . \
-            --push \
-            --progress=plain \
-            --tag docker.io/punksecurity/saist:lite-pr-${{ github.event.number }} \
-            --platform linux/amd64 \
-            --target saist
-      - name: Build and push SAIST
-        run: |
-          docker buildx build . \
-            --push \
-            --progress=plain \
-            --tag docker.io/punksecurity/saist:pr-${{ github.event.number }} \
-            --platform linux/amd64 \
-            --target saist-tex
+      - name: Build and push SAIST
+        run: |
+          docker buildx build . \
+            --push \
+            --progress=plain \
+            --tag docker.io/punksecurity/saist:pr-${{ github.event.number }} \
+            --platform linux/amd64 \
+            --target saist

.github/workflows/build_release.yml

@@ -28,4 +28,4 @@ jobs:
           VERSION: ${{ steps.version.outputs.version }}
         with:
           push: true
-          targets: "full,lite"
+          targets: "release"

.github/workflows/tests.yml

@@ -0,0 +1,30 @@
+name: Tests
+
+on:
+  pull_request:
+    types: [opened, synchronize, reopened]
+
+jobs:
+  pytest:
+    runs-on: ubuntu-latest
+
+    steps:
+      - name: Check out code
+        uses: actions/checkout@v4
+
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+          cache: pip
+          cache-dependency-path: |
+            requirements.txt
+            requirements-dev.txt
+
+      - name: Install dependencies
+        run: |
+          python -m pip install --upgrade pip
+          pip install -r requirements-dev.txt
+
+      - name: Run pytest
+        run: pytest -q

Dockerfile

@@ -19,43 +19,8 @@ COPY saist .
 # Exports
 ENV SAIST_COMMAND "docker run punksecurity/saist" 
 ENV SAIST_CSV_PATH "/app/results.csv" 
-ENV SAIST_TEX_FILENAME "report.tex"
 ENV SAIST_PDF_FILENAME "report.pdf"
 ENV SAIST_WEB_HOST "0.0.0.0"
 ENV PYTHONUNBUFFERED 1
 ENTRYPOINT [ "python3", "/app/main.py" ]
 CMD [ "-h" ]
-
-FROM alpine as tex-dl
-WORKDIR /tmp
-RUN mkdir -p /opt/texlive/bin
-
-RUN wget https://ftp.math.utah.edu/pub/texlive-utah/bin/aarch64-alpine322.tar.xz  && \
-    tar xvf aarch64-alpine322.tar.xz && ls -ltra && \
-    mv aarch64-alpine322 /opt/texlive/bin/aarch64-linuxmusl
-
-RUN wget https://ftp.math.utah.edu/pub/texlive-utah/bin/x86_64-alpine322.tar.xz  && \
-    tar xvf x86_64-alpine322.tar.xz && \
-    mv x86_64-alpine322 /opt/texlive/bin/x86_64-linuxmusl
-
-FROM saist AS saist-tex
-
-ARG TL_MIRROR="https://texlive.info/CTAN/systems/texlive/tlnet"
-
-COPY saist/latex/texlive.profile /tmp
-COPY --from=tex-dl /opt/texlive/bin /tmp/texlive/bin
-RUN apk add --no-cache perl curl fontconfig xz && \
-    mkdir -p "/tmp/texlive" && cd "/tmp/texlive" && \
-    wget "$TL_MIRROR/install-tl-unx.tar.gz" && \
-    tar xzvf ./install-tl-unx.tar.gz && \
-    "./install-tl-"*"/install-tl" --location "$TL_MIRROR" --custom-bin=/tmp/texlive/bin/$(uname -m)-linuxmusl -profile "/tmp/texlive.profile" && \
-    rm -vf "/opt/texlive/install-tl" && \
-    rm -vf "/opt/texlive/install-tl.log" && \
-    rm -vrf /tmp/*
-
-ENV PATH="${PATH}:/opt/texlive/bin/custom"
-
-ARG TL_PACKAGES="lineno titlesec upquote minted blindtext booktabs fontawesome latexmk parskip xcolor"
-
-RUN tlmgr update --self && \
-    tlmgr install ${TL_PACKAGES}

README.md

@@ -29,10 +29,12 @@ We support OLLAMA for local / offline code scanning.
 - **Diff scanning**: Git commits, branches, or PRs
 - **Multi-LLM support**: `OpenAI`, `Anthropic`, `Bedrock`, `DeepSeek`, `Gemini`, `Ollama`
 - **Filesystem, Git, GitHub PR scanning modes**
-- **Pattern-based file inclusion/exclusion** using `.saist.include` and `.saist.ignore`
-- **Interactive chat** with your findings
-- **Web server** UI to view results
-- **CSV export** of findings
+- **Pattern-based file inclusion/exclusion** using `.saist.include` and `.saist.ignore`
+- **Project-specific analysis skills** loaded from Markdown files to teach SAIST app routing, authorization, framework conventions, and other local security context
+- **LLM-generated analysis skills** for bootstrapping those files in a separate run
+- **Interactive chat** with your findings
+- **Web server** UI to view results
+- **CSV export** of findings
 - **PDF report**: Generate PDF reports of SAIST findings
 - **CI/CD pipeline friendly** (exit 1 on findings)
 
@@ -73,16 +75,18 @@ export SAIST_LLM_API_KEY=your-api-key
 
 | Task | Command |
 |:-----|:--------|
-| Get a DevSecOps poem | `saist/main.py --llm openai poem` |
-| Scan a local folder | `saist/main.py --llm deepseek filesystem /path/to/code` |
-| Scan a local folder with ollama from within docker| `docker run --network=host -v <folder_path>:/vulnerableapp -v $PWD/reporting:/app/reporting punksecurity/saist --llm ollama --llm-model gemma3:4b fileystem /vulnerableapp` |
+| Get a DevSecOps poem | `saist/main.py --llm openai poem` |
+| Scan a local folder | `saist/main.py --llm deepseek filesystem /path/to/code` |
+| Scan a local folder file-by-file | `saist/main.py --llm deepseek --deep filesystem /path/to/code` |
+| Scan a local folder with ollama from within docker| `docker run --network=host -v <folder_path>:/vulnerableapp -v $PWD/reporting:/app/reporting punksecurity/saist --llm ollama --llm-model gemma3:4b fileystem /vulnerableapp` |
 | Scan a local Git repo | `saist/main.py --llm openai git /path/to/repo` |
 | Scan a local Git repo (branch diff) | `saist/main.py --llm openai git /path/to/repo --ref-for-compare main --ref-to-compare feature-branch` |
 | Scan a GitHub PR (and update the PR) | `saist/main.py --llm anthropic github yourorg/yourrepo 1234 --github-token your-token` |
 | Launch web server to view findings | `saist/main.py --llm deepseek --web filesystem /path/to/code` |
 | Interactive shell after scanning | `saist/main.py --llm ollama --interactive filesystem /path/to/code` |
-| Export findings as CSV | `saist/main.py --llm openai --csv filesystem /path/to/code` |
-| Scan with docker and export findings as PDF report | `docker run -v <folder_path>:/vulnerableapp -v $PWD/reporting:/app/reporting punksecurity/saist --llm openai --pdf filesystem /vulnerableapp` |
+| Export findings as CSV | `saist/main.py --llm openai --csv filesystem /path/to/code` |
+| Generate analysis skills | `saist/main.py --llm openai --generate-skills filesystem /path/to/code` |
+| Scan with docker and export findings as PDF report | `docker run -v <folder_path>:/vulnerableapp -v $PWD/reporting:/app/reporting punksecurity/saist --llm openai --pdf filesystem /vulnerableapp` |
 | Scan with docker and export findings as PDF report with a project title | `docker run -v <folder_path>:/vulnerableapp -v $PWD/reporting:/app/reporting punksecurity/saist --llm openai --pdf --project-name "Project Name" filesystem /vulnerableapp` |
 | Scan with docker and retain cache for future runs | `docker run -v <folder_path>:/vulnerableapp -v $PWD/SAISTCache:/app/SAISTCache punksecurity/saist --llm openai filesystem /vulnerableapp` |
 | Change caching folder | `saist/main.py --llm openai --cache-folder /path/to/cache filesystem /path/to/code` |
@@ -106,7 +110,7 @@ saist respects **file include/exclude rules** via two optional files in the root
     - `build/` will ignore the entire build folder
     - `*.log` will ignore all log files
 
-You can also provide include/exclude patterns using the command-line arguments `--include` and `--exclude`.
+You can also provide include/exclude patterns using the command-line arguments `--include` and `--exclude`.
 - Patterns provided via command-line arguments are appended to any patterns loaded from the rule files.
 - Examples:
     - `--include '**/*.py' --include '**/*.ts'` includes all Python and TypeScript files
@@ -132,23 +136,51 @@ docs/
 This setup will:
 - Only scan `.py`, `.ts`, and specific `.js` files
 - Ignore anything under `tests/` and `docs/`
----
-
-
-## 📄 PDF report generation
+---
+
+## 🧠 Analysis Skills
+
+SAIST can load project-specific analysis skill files from `.saist/skills/*.md`. These files are added to the security review prompt so future scans understand application-specific details such as routing, authentication, authorization, framework conventions, data access, validation boundaries, dependencies, configuration, and security-sensitive workflows.
+
+Generate an initial set of skill files as a separate run:
+
+```bash
+saist/main.py --llm openai --generate-skills filesystem /path/to/code
+```
+
+Then review or edit the generated Markdown files and run SAIST normally. Skill files are loaded automatically on future scans:
+
+```bash
+saist/main.py --llm openai filesystem /path/to/code
+```
+
+Useful options:
+
+| Option | Description |
+|:------|:------------|
+| `--skills-path` | Folder containing skill Markdown files. Defaults to `.saist/skills` under the scanned project. |
+| `--generate-skills` | Ask the configured LLM to generate skill files and then exit. |
+| `--overwrite-skills` | Replace existing skill files during generation. Without this, existing files are preserved. |
+| `--disable-skills` | Do not load skill files during analysis. |
+| `--skills-max-bytes` | Limit total skill guidance added to analysis prompts. |
+| `--skills-sample-files` / `--skills-sample-bytes` | Control how much project context is sampled when generating skills. |
+
+When skills are loaded, SAIST salts its findings cache with the skill content so updated guidance gets a fresh analysis run.
+
+---
+
+
+## 📄 PDF report generation
 
 saist allows you to generate PDF reports summarizing your findings, making it easier to share insights with your team.
 
-To create a PDF report, simply use the `--pdf` flag when running the scan. By default, the report will be saved to
-`reporting/report.pdf`. You can customize the filename by using the `--pdf-filename` option followed by your desired
-filename.
-
-To add a project name onto the title page of the PDF report, use the `--project-name` option followed by your desired title.
-
-> It is recommended to use the provided Docker image for generating PDF reports, as it includes the necessary TeX suite,
-which can be quite large. This ensures that all dependencies are met and the report is generated properly.
-
-If not, you need to install latexmk to make it work.
+To create a PDF report, use the `--pdf` flag when running the scan. By default, the report will be saved to
+`reporting/report.pdf`. You can customize the filename by using the `--pdf-filename` option followed by your desired
+filename.
+
+To add a project name onto the title page of the PDF report, use the `--project-name` option followed by your desired title.
+
+PDF reports are generated with the built-in ReportLab renderer, so no external document-rendering toolchain is required.
 
 ### 🐋 Example (Docker)
 
@@ -169,13 +201,23 @@ docker run -v$PWD/code:/code -v$PWD/reporting:/app/reporting punksecurity/saist
 
 | Option | Description |
 |:------|:------------|
-| `--llm` | Select LLM (`anthropic`, `deepseek`, `gemini`, `ollama`, `openai`) |
-| `--llm-api-key` | API key for your LLM |
-| `--llm-model` | (Optional) Specific model (e.g., `gpt-4o`) |
-| `--interactive` | Chat with the LLM after scan |
-| `--web` | Launch a local web server |
-| `--disable-tools` | Disable tool use during file analysis to reduce LLM token usage |
-| `--disable-caching` | Disable finding caching during file analysis |
+| `--llm` | Select LLM (`anthropic`, `azure-foundry`, `bedrock`, `deepseek`, `gemini`, `ollama`, `openai`) |
+| `--llm-api-key` | API key for your LLM |
+| `--llm-model` | (Optional) Specific model (e.g., `gpt-4o`) |
+| `--thinking` | Pydantic AI thinking effort: `minimal`, `low`, `medium`, `high`, `xhigh`, or `disabled` |
+| `--openai-base-uri` | Base URI for OpenAI-compatible services. Can also be set with `SAIST_OPENAI_BASE_URI`. |
+| `--azure-openai-endpoint` | Azure AI Foundry or Azure OpenAI endpoint. Can also be set with `AZURE_OPENAI_ENDPOINT`; `/openai/v1/` endpoints use the Responses API without `api-version`. |
+| `--azure-openai-api-version` | Azure OpenAI API version for non-v1 endpoints. Can also be set with `OPENAI_API_VERSION`. |
+| `--interactive` | Chat with the LLM after scan |
+| `--web` | Launch a local web server |
+| `--disable-tools` | Disable tool use during file analysis to reduce LLM token usage |
+| `--deep` | For filesystem scans, analyze every file individually. Without this, filesystem scans send a file inventory and let the LLM inspect files with tools, then report file coverage. |
+| `--iterations` | Number of tool-driven filesystem scan passes to run when `--deep` is not set. Defaults to `1`; concurrency is capped by `--llm-rate-limit`. |
+| `--skills-path` | Folder containing SAIST analysis skill Markdown files |
+| `--generate-skills` | Generate SAIST analysis skill files and exit |
+| `--overwrite-skills` | Replace existing skill files during skill generation |
+| `--disable-skills` | Do not load skill files during analysis |
+| `--disable-caching` | Disable finding caching during file analysis |
 | `--skip-line-length-check` | Skip checking files for a maximum line length |
 | `--max-line-length` | Maximum allowed line length, files with lines longer than this value will be skipped |
 | `--i, --include` | Pattern to explicitly include |

docker-bake.hcl

@@ -11,16 +11,11 @@ target "base" {
 target "nightly" {
   inherits = ["base"]
   tags = ["docker.io/punksecurity/saist:nightly"]
-}
-
-target "lite" {
-  inherits = ["base"]
-  tags = ["docker.io/punksecurity/saist:lite-${VERSION}", "docker.io/punksecurity/saist:lite-latest"]
   target = "saist"
 }
 
-target "full" {
+target "release" {
   inherits = ["base"]
   tags = ["docker.io/punksecurity/saist:${VERSION}", "docker.io/punksecurity/saist:latest"]
-  target = "saist-tex"
+  target = "saist"
 }

pytest.ini

@@ -0,0 +1,3 @@
+[pytest]
+pythonpath = saist
+testpaths = tests

requirements-dev.txt

@@ -0,0 +1,2 @@
+-r requirements.txt
+pytest==8.3.5

requirements.txt

@@ -1,10 +1,11 @@
-pydantic==2.11.7
+pydantic==2.12.5
 requests==2.32.4
 GitPython==3.1.44
 python-dotenv==1.1.1
-pydantic-ai==0.3.7
+pydantic-ai-slim[anthropic,bedrock,google,openai]==1.93.0
 aiofiles==24.1.0
 rich==14.0.0
 flask==3.1.1
 ollama==0.5.1
 gitignore-parser==0.1.13
+reportlab==4.5.0