Skip to content

Commit cc00e00

Browse files
salmanltIshavyas9
salmanlt
authored and
Ishavyas9
committed
New docs: Updated and restructured Agent Assurance docs created a new Video Agent doc
1 parent 1d46d5c commit cc00e00

27 files changed

Lines changed: 7673 additions & 6 deletions

‎docs/agent-assurance-ci-cd.md‎

Lines changed: 219 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,219 @@
1+
---
2+
id: agent-assurance-ci-cd
3+
title: Run Agent Assurance in CI/CD
4+
hide_title: false
5+
sidebar_label: CI/CD and Automation
6+
description: Run Agent Assurance headlessly in CI, consume NDJSON output, handle exit codes, isolate state, and build a safe agent testing gate.
7+
keywords:
8+
- rook ci cd
9+
- ai agent testing github actions
10+
- rook headless
11+
- agent testing pipeline
12+
url: https://www.testmuai.com/support/docs/agent-assurance-ci-cd/
13+
site_name: TestMu AI
14+
slug: agent-assurance-ci-cd/
15+
canonical: https://www.testmuai.com/support/docs/agent-assurance-ci-cd/
16+
---
17+
18+
import BrandName, { BRAND_URL } from '@site/src/component/BrandName';
19+
20+
<script type="application/ld+json"
21+
dangerouslySetInnerHTML={{ __html: JSON.stringify({
22+
"@context": "https://schema.org", "@type": "BreadcrumbList",
23+
"itemListElement": [
24+
{ "@type": "ListItem", "position": 1, "name": "Home", "item": BRAND_URL },
25+
{ "@type": "ListItem", "position": 2, "name": "Support", "item": `${BRAND_URL}/support/docs/` },
26+
{ "@type": "ListItem", "position": 3, "name": "CI/CD", "item": `${BRAND_URL}/support/docs/agent-assurance-ci-cd/` }
27+
]
28+
}) }}
29+
/>
30+
31+
# Run Agent Assurance in CI/CD
32+
33+
<head>
34+
<meta name="robots" content="noindex, nofollow" />
35+
</head>
36+
37+
Rook's headless commands use the same discovery, generation, profile, permission, execution, judging, and evidence paths as the interactive TUI. Use them to build a release gate after you have proved the workflow interactively against the same agent and profile.
38+
39+
## Prepare the Project Interactively
40+
41+
Before enabling a pipeline:
42+
43+
1. Run `/explore`, `/generate`, `/profile add`, and a one-scenario `/run` locally.
44+
2. Review the generated `.testmuai/rook/` agent records, scenarios, and profile.
45+
3. Replace all literal credentials with `${VAR}` references.
46+
4. Commit the project Rook files that define the suite and profile.
47+
5. Keep global credentials, environment values, permission grants, and sessions out of the repository.
48+
6. Seed test fixtures and verify the profile reset command.
49+
50+
> **Authentication in unattended environments:** The current pre-alpha release uses the interactive LambdaTest login flow and exposes no dedicated service-token login flag. Use a protected persistent runner with a pre-authenticated `ROOK_HOME`, or follow your organization's approved secret provisioning process. Never commit or upload a personal Rook credential store as a repository artifact.
51+
52+
## Isolate Global State
53+
54+
Set `ROOK_HOME` to a protected runner directory:
55+
56+
```bash
57+
export ROOK_HOME="$RUNNER_TEMP/rook-home"
58+
```
59+
60+
For a persistent self-hosted runner, choose a stable protected path so token renewal is retained. Ensure only the runner identity can read it.
61+
62+
Project evidence continues to be written under:
63+
64+
```text
65+
$GITHUB_WORKSPACE/.testmuai/rook/
66+
```
67+
68+
## Verify the Environment
69+
70+
```bash
71+
rook --version
72+
rook doctor
73+
rook auth status
74+
rook plan --json
75+
```
76+
77+
An unreachable controller does not mean a token is invalid. When `rook auth status` cannot reach the controller, it can return success with a warning. The next real operation still fails if connectivity is unavailable.
78+
79+
## Run a Deterministic Suite
80+
81+
Use explicit agent and scenario IDs in CI:
82+
83+
```bash
84+
rook agent list --json
85+
rook profile use staging --entity refund-desk
86+
rook run \
87+
--entity refund-desk \
88+
--only SC-001,SC-004,SC-014 \
89+
--no-narrative \
90+
--json
91+
```
92+
93+
Headless `rook run` currently does not expose class, category, tag, concurrency, free-form selection, or RCA flags. Resolve and review the intended IDs in the committed suite.
94+
95+
## Authorize Required Operations
96+
97+
An unattended command cannot answer a permission prompt. Pass exact, temporary allowances that were reviewed with the workflow:
98+
99+
```bash
100+
rook run \
101+
--entity refund-desk \
102+
--only SC-001,SC-004 \
103+
--allow 'run(https://refund-agent.staging.example.com/v1/chat)' \
104+
--json
105+
```
106+
107+
The allowance applies only to that process. Repeat `--allow` for each exact rule.
108+
109+
Avoid broad shell or MCP allowances. `--allow` adds authority; it does not remove a broader permission already stored in the selected `ROOK_HOME`.
110+
111+
## Consume JSON Output
112+
113+
`--json` emits newline-delimited JSON events. Process one object per line rather than parsing human prose:
114+
115+
```bash
116+
rook run --entity refund-desk --only SC-001 --json > rook-events.ndjson
117+
```
118+
119+
Use `--verbose` when diagnostic tool activity and cost events are needed:
120+
121+
```bash
122+
rook run --entity refund-desk --only SC-001 --verbose --json
123+
```
124+
125+
Do not assume every command returns one aggregate JSON object. Preserve the NDJSON stream as a job artifact for debugging.
126+
127+
## Handle Exit Codes
128+
129+
| Exit code | Meaning | Recommended pipeline action |
130+
|---|---|---|
131+
| `0` | No agent defect was recorded in the verdicts that were produced | Check run completion, coverage, and Unable-to-Verify output before continuing |
132+
| `1` | Rook could not test the agent or the requested state was not reached | Fail as infrastructure or harness error |
133+
| `2` | Agent failure or adversarial compromise was observed | Fail as an agent-quality finding |
134+
| `3` | Authentication is required or no longer valid | Stop and repair runner authentication |
135+
| `4` | Session budget or credits were exhausted | Stop, review scope, and adjust approved budget or credits |
136+
137+
An observed agent failure outranks an invocation error if both occur in one run.
138+
139+
> **Exit code `0` does not prove suite completion:** In the current pre-alpha release, `rook run` and `rook report` derive their exit status from recorded verdicts. An interrupted or partial run can therefore exit `0` when its completed scenarios contain no recorded defect. A release gate must also inspect the saved `run.yaml`: require a completed status, review the stop reason, and compare selected scenario counts against completed verdict counts. Fail closed when the requested suite did not finish.
140+
141+
## Example GitHub Actions Job
142+
143+
This example assumes a protected self-hosted runner already has an authenticated Rook home and can reach the staging agent and controller.
144+
145+
```yaml
146+
name: Rook agent assurance
147+
148+
on:
149+
pull_request:
150+
151+
jobs:
152+
rook:
153+
runs-on: self-hosted
154+
permissions:
155+
contents: read
156+
env:
157+
ROOK_HOME: /var/lib/rook-ci/home
158+
ROOK_AGENT_TOKEN: ${{ secrets.ROOK_AGENT_TOKEN }}
159+
steps:
160+
- uses: actions/checkout@v4
161+
162+
- uses: actions/setup-node@v4
163+
with:
164+
node-version: 20
165+
166+
- name: Install pinned Rook build
167+
env:
168+
GH_TOKEN: ${{ secrets.ROOK_REPOSITORY_TOKEN }}
169+
ROOK_VERSION: <reviewed-commit-sha>
170+
run: |
171+
curl -fsSL -H "Authorization: Bearer ${GH_TOKEN}" \
172+
https://raw.githubusercontent.com/LambdatestIncPrivate/rook/stage/scripts/install.sh | bash
173+
174+
- name: Verify Rook environment
175+
run: |
176+
rook --version
177+
rook doctor
178+
rook auth status
179+
180+
- name: Run release-gate scenarios
181+
run: |
182+
rook profile use staging --entity refund-desk
183+
rook run \
184+
--entity refund-desk \
185+
--only SC-001,SC-004,SC-014 \
186+
--no-narrative \
187+
--allow 'run(https://refund-agent.staging.example.com/v1/chat)' \
188+
--json | tee rook-events.ndjson
189+
190+
- name: Print report
191+
if: always()
192+
run: rook report --entity refund-desk
193+
194+
- name: Upload evidence
195+
if: always()
196+
uses: actions/upload-artifact@v4
197+
with:
198+
name: rook-evidence
199+
path: .testmuai/rook/agents/refund-desk/runs/
200+
```
201+
202+
**Pin Rook by commit SHA** and review updates before changing it. Scope the repository token to read access for the private release repository.
203+
204+
## Separate Generation From the Gate
205+
206+
Scenario generation uses models and can change the suite. A stable release gate should run reviewed, committed scenario IDs. Move generation into a separate scheduled or manually approved workflow:
207+
208+
```bash
209+
rook explore . --force --all --json
210+
rook generate --entity refund-desk --total 30 --json
211+
```
212+
213+
Review the resulting scenario diff before it changes the required gate.
214+
215+
## Preserve Evidence on Failure
216+
217+
Upload the NDJSON stream and the run directory with `if: always()`. A failed invocation still records the request, and a budget or controller stop preserves completed scenarios.
218+
219+
**Before granting broad access to CI logs or artifacts, review them for secrets and personal data.**

0 commit comments

Comments
 (0)