feat: Add MLflow Prompt Registry provider (squashed commit)

Add a new remote provider that integrates MLflow's Prompt Registry with
Llama Stack's prompts API, enabling centralized prompt management and
versioning using MLflow as the backend.

Features:
- Full implementation of Llama Stack Prompts protocol
- Support for prompt versioning and default version management
- Automatic variable extraction from Jinja2-style templates
- MLflow tag-based metadata for efficient prompt filtering
- Flexible authentication (config, environment variables, per-request)
- Bidirectional ID mapping (pmpt_<hex> ↔ llama_prompt_<hex>)
- Comprehensive error handling and validation

Implementation:
- Remote provider: src/llama_stack/providers/remote/prompts/mlflow/
- Inline reference provider: src/llama_stack/providers/inline/prompts/reference/
- MLflow 3.4+ required for Prompt Registry API support
- Deterministic ID mapping ensures consistency across conversions

Testing:
- 15 comprehensive unit tests (config validation, ID mapping)
- 18 end-to-end integration tests (full CRUD workflows)
- GitHub Actions workflow for automated CI testing with MLflow server
- Integration test fixtures with automatic server setup

Documentation:
- Complete provider configuration reference
- Setup and usage examples with code samples
- Authentication options and security best practices

Signed-off-by: William Caban <[email protected]>
Co-Authored-By: Claude <[email protected]>

Author: williamcaban

.github/workflows/integration-mlflow-tests.yml

@@ -0,0 +1,125 @@
+name: MLflow Prompts Integration Tests
+
+run-name: Run the integration test suite with MLflow Prompt Registry provider
+
+on:
+  push:
+    branches:
+      - main
+      - 'release-[0-9]+.[0-9]+.x'
+  pull_request:
+    branches:
+      - main
+      - 'release-[0-9]+.[0-9]+.x'
+    paths:
+      - 'src/llama_stack/providers/remote/prompts/mlflow/**'
+      - 'tests/integration/providers/remote/prompts/mlflow/**'
+      - 'tests/unit/providers/remote/prompts/mlflow/**'
+      - 'uv.lock'
+      - 'pyproject.toml'
+      - 'requirements.txt'
+      - '.github/workflows/integration-mlflow-tests.yml' # This workflow
+  schedule:
+    - cron: '0 0 * * *'  # Daily at 12 AM UTC
+
+concurrency:
+  group: ${{ github.workflow }}-${{ github.ref == 'refs/heads/main' && github.run_id || github.ref }}
+  cancel-in-progress: true
+
+jobs:
+  test-mlflow:
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        python-version: ${{ github.event.schedule == '0 0 * * *' && fromJSON('["3.12", "3.13"]') || fromJSON('["3.12"]') }}
+      fail-fast: false
+
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@1af3b93b6815bc44a9784bd300feb67ff0d1eeb3 # v6.0.0
+
+      - name: Install dependencies
+        uses: ./.github/actions/setup-runner
+        with:
+          python-version: ${{ matrix.python-version }}
+
+      - name: Setup MLflow Server
+        run: |
+          docker run --rm -d --pull always \
+            --name mlflow \
+            -p 5555:5555 \
+            ghcr.io/mlflow/mlflow:latest \
+            mlflow server \
+              --host 0.0.0.0 \
+              --port 5555 \
+              --backend-store-uri sqlite:///mlflow.db \
+              --default-artifact-root ./mlruns
+
+      - name: Wait for MLflow to be ready
+        run: |
+          echo "Waiting for MLflow to be ready..."
+          for i in {1..60}; do
+            if curl -s http://localhost:5555/health | grep -q '"status": "OK"'; then
+              echo "MLflow is ready!"
+              exit 0
+            fi
+            echo "Not ready yet... ($i/60)"
+            sleep 2
+          done
+          echo "MLflow failed to start"
+          docker logs mlflow
+          exit 1
+
+      - name: Verify MLflow API
+        run: |
+          echo "Testing MLflow API..."
+          curl -X GET http://localhost:5555/api/2.0/mlflow/experiments/list
+          echo ""
+          echo "MLflow API is responding!"
+
+      - name: Build Llama Stack
+        run: |
+          uv run --no-sync llama stack list-deps ci-tests | xargs -L1 uv pip install
+
+      - name: Install MLflow Python client
+        run: |
+          uv pip install 'mlflow>=3.4.0'
+
+      - name: Check Storage and Memory Available Before Tests
+        if: ${{ always() }}
+        run: |
+          free -h
+          df -h
+
+      - name: Run MLflow Integration Tests
+        env:
+          MLFLOW_TRACKING_URI: http://localhost:5555
+        run: |
+          uv run --no-sync \
+            pytest -sv \
+            tests/integration/providers/remote/prompts/mlflow/
+
+      - name: Check Storage and Memory Available After Tests
+        if: ${{ always() }}
+        run: |
+          free -h
+          df -h
+
+      - name: Write MLflow logs to file
+        if: ${{ always() }}
+        run: |
+          docker logs mlflow > mlflow.log 2>&1 || true
+
+      - name: Upload all logs to artifacts
+        if: ${{ always() }}
+        uses: actions/upload-artifact@330a01c490aca151604b8cf639adc76d48f6c5d4 # v5.0.0
+        with:
+          name: mlflow-logs-${{ github.run_id }}-${{ github.run_attempt }}-${{ matrix.python-version }}
+          path: |
+            *.log
+          retention-days: 1
+
+      - name: Stop MLflow container
+        if: ${{ always() }}
+        run: |
+          docker stop mlflow || true

docs/docs/providers/prompts/index.mdx

@@ -0,0 +1,92 @@
+---
+sidebar_label: Prompts
+title: Prompts
+---
+
+# Prompts
+
+## Overview
+
+This section contains documentation for all available providers for the **prompts** API.
+
+The Prompts API enables centralized management of prompt templates with versioning, variable handling, and team collaboration capabilities.
+
+## Available Providers
+
+### Inline Providers
+
+Inline providers run in the same process as the Llama Stack server and require no external dependencies:
+
+- **[inline::reference](inline_reference.mdx)** - Reference implementation using KVStore backend (SQLite, PostgreSQL, etc.)
+  - Zero external dependencies
+  - Supports local SQLite or PostgreSQL storage
+  - Full CRUD operations including deletion
+  - Ideal for local development and single-server deployments
+
+### Remote Providers
+
+Remote providers connect to external services for centralized prompt management:
+
+- **[remote::mlflow](remote_mlflow.mdx)** - MLflow Prompt Registry integration (requires MLflow 3.4+)
+  - Centralized prompt management across teams
+  - Built-in versioning and audit trail
+  - Supports authentication (per-request, config, or environment variables)
+  - Integrates with Databricks and enterprise MLflow deployments
+  - Ideal for team collaboration and production environments
+
+## Choosing a Provider
+
+### Use `inline::reference` when:
+- Developing locally or deploying to a single server
+- You want zero external dependencies
+- SQLite or PostgreSQL storage is sufficient
+- You need full CRUD operations (including deletion)
+- You prefer simple configuration
+
+### Use `remote::mlflow` when:
+- Working in a team environment with multiple users
+- You need centralized prompt management
+- Integration with existing MLflow infrastructure
+- You need authentication and multi-tenant support
+- Advanced versioning and audit trail capabilities are required
+
+## Quick Start Examples
+
+### Using inline::reference
+
+```yaml
+prompts:
+  - provider_id: local-prompts
+    provider_type: inline::reference
+    config:
+      run_config:
+        storage:
+          stores:
+            prompts:
+              type: sqlite
+              db_path: ./prompts.db
+```
+
+### Using remote::mlflow
+
+```yaml
+prompts:
+  - provider_id: mlflow-prompts
+    provider_type: remote::mlflow
+    config:
+      mlflow_tracking_uri: http://localhost:5555
+      experiment_name: llama-stack-prompts
+      auth_credential: ${env.MLFLOW_TRACKING_TOKEN}
+```
+
+## Common Features
+
+All prompt providers support:
+- Create and store prompts with version control
+- Retrieve prompts by ID and version
+- Update prompts (creates new versions)
+- List all prompts or versions of a specific prompt
+- Set default version for a prompt
+- Automatic variable extraction from `{{ variable }}` templates
+
+For detailed documentation on each provider, see the individual provider pages linked above.