|
| 1 | +--- |
| 2 | +title: Introduction |
| 3 | +description: Autonomous vLLM deployment optimizer — tunes your serving config overnight against your real workload. |
| 4 | +--- |
| 5 | + |
| 6 | +aevyra-forge tunes your vLLM deployment overnight. Give it a model, a GPU, |
| 7 | +and a workload trace. It runs an autonomous loop — propose a config, boot the |
| 8 | +server, benchmark against your real workload, keep or revert, repeat. By |
| 9 | +morning you have a deployment recipe that beats hand-tuned defaults, with a |
| 10 | +full audit trail of every experiment. |
| 11 | + |
| 12 | +```bash |
| 13 | +pip install aevyra-forge |
| 14 | +``` |
| 15 | + |
| 16 | +vLLM exposes roughly 40 serving args. The defaults are conservative — designed |
| 17 | +to work safely on any GPU, not to max out any specific one. On a T4 with a |
| 18 | +chat workload the defaults leave significant throughput on the table: |
| 19 | + |
| 20 | +``` |
| 21 | +Baseline (defaults): 2718 tok/s P99: 241 ms |
| 22 | +After Forge (8 exps): 3421 tok/s P99: 187 ms (+26%) |
| 23 | +``` |
| 24 | + |
| 25 | +Forge finds that gain by searching the joint space of batching, caching, and |
| 26 | +memory knobs — and keeping only the changes that improve the score on your |
| 27 | +actual workload. |
| 28 | + |
| 29 | +## Where Forge fits |
| 30 | + |
| 31 | +```mermaid |
| 32 | +flowchart LR |
| 33 | + W[Witness\ncaptures trace]:::w |
| 34 | + V[Verdict\ngrades it]:::v |
| 35 | + O[Origin\nfinds culprits]:::o |
| 36 | + R[Reflex\nfixes prompts]:::r |
| 37 | + F[Forge\ntunes deployment]:::f |
| 38 | +
|
| 39 | + W --> V --> O --> R |
| 40 | + W --> F |
| 41 | +
|
| 42 | + classDef w fill:#3FBFFF,color:#fff,stroke:none |
| 43 | + classDef v fill:#6E3FF3,color:#fff,stroke:none |
| 44 | + classDef o fill:#FF6B6B,color:#fff,stroke:none |
| 45 | + classDef r fill:#2ECC71,color:#fff,stroke:none |
| 46 | + classDef f fill:#FF6B35,color:#fff,stroke:none |
| 47 | +``` |
| 48 | + |
| 49 | +Forge operates on the infrastructure layer. Where Reflex rewrites prompts and |
| 50 | +Origin diagnoses agent failures, Forge maximises throughput and minimises |
| 51 | +latency for a model that's already doing the right thing. |
| 52 | + |
| 53 | +## The loop |
| 54 | + |
| 55 | +```mermaid |
| 56 | +flowchart LR |
| 57 | + WL[Workload\nJSONL trace]:::data |
| 58 | + HW[Hardware\nauto-detected]:::data |
| 59 | + PB[Playbook\nheuristics]:::forge |
| 60 | + AG[Agent\n1 LLM call]:::forge |
| 61 | + VL[vLLM\nserver]:::bench |
| 62 | + BN[bench\nreplay workload]:::bench |
| 63 | + SC([Score\nthroughput × accuracy]):::score |
| 64 | + RY[best_recipe.yaml]:::output |
| 65 | +
|
| 66 | + WL & HW --> AG |
| 67 | + PB --> AG |
| 68 | + AG -->|mutated recipe| VL |
| 69 | + VL --> BN |
| 70 | + BN -->|BenchResult| SC |
| 71 | + SC -->|keep / revert| AG |
| 72 | + SC --> RY |
| 73 | +
|
| 74 | + classDef data fill:#6E3FF3,color:#fff,stroke:none |
| 75 | + classDef forge fill:#9B6BFF,color:#fff,stroke:none |
| 76 | + classDef bench fill:#3FBFFF,color:#fff,stroke:none |
| 77 | + classDef score fill:#FF6B35,color:#fff,stroke:none |
| 78 | + classDef output fill:#2ECC71,color:#fff,stroke:none |
| 79 | +``` |
| 80 | + |
| 81 | +Each iteration: the agent reads the playbook and experiment history, proposes |
| 82 | +one targeted change, and Forge measures whether it actually helps. The audit |
| 83 | +trail captures every decision — config, result, rationale — so you can see |
| 84 | +exactly how the winning recipe was found. |
| 85 | + |
| 86 | +## Tuning layers |
| 87 | + |
| 88 | +| Layer | What it tunes | Status | |
| 89 | +|---|---|---| |
| 90 | +| **1. Config** | vLLM serving args: batching, caching, parallelism | ✅ v0.1 | |
| 91 | +| **2. Quantization** | INT4/FP8/INT8, KV cache precision | 🔧 v0.2 | |
| 92 | +| **3. Kernel synthesis** | Custom kernels via AutoKernel | 🚧 v0.3 | |
| 93 | + |
| 94 | +Layer 1 has the highest leverage per experiment because it requires no |
| 95 | +recompilation. Forge escalates to Layer 2 when Layer 1 converges. |
| 96 | + |
| 97 | +## Works with any GPU and LLM |
| 98 | + |
| 99 | +Auto-detects NVIDIA and AMD GPUs via `nvidia-smi` / `rocm-smi`. Works with |
| 100 | +any OpenAI-compatible LLM for the agent. |
| 101 | + |
| 102 | +```bash |
| 103 | +pip install aevyra-forge # Claude included by default |
| 104 | +pip install aevyra-forge[openai] # add OpenAI / OpenRouter / Together / Groq |
| 105 | +``` |
| 106 | + |
| 107 | +| Provider | Env var | |
| 108 | +|---|---| |
| 109 | +| **Anthropic** | `ANTHROPIC_API_KEY` | |
| 110 | +| **OpenAI** | `OPENAI_API_KEY` | |
| 111 | +| **OpenRouter** | `OPENROUTER_API_KEY` | |
| 112 | +| **Ollama** | — | |
| 113 | + |
| 114 | +Python 3.10+. Apache-2.0 licensed. |
| 115 | + |
| 116 | +<CardGroup cols={2}> |
| 117 | + <Card title="Quick start" icon="bolt" href="/forge/quickstart"> |
| 118 | + Run your first tuning session in 15 minutes |
| 119 | + </Card> |
| 120 | + <Card title="Tutorial" icon="book-open" href="/forge/tutorial-colab-quickstart"> |
| 121 | + Dry-run walkthrough with real log output |
| 122 | + </Card> |
| 123 | + <Card title="Concepts: Recipe" icon="file-code" href="/forge/concepts/recipe"> |
| 124 | + The artifact Forge proposes, mutates, and keeps or reverts |
| 125 | + </Card> |
| 126 | + <Card title="Concepts: Playbook" icon="book" href="/forge/concepts/playbook"> |
| 127 | + The agent's instruction manual |
| 128 | + </Card> |
| 129 | +</CardGroup> |
0 commit comments