Skip to content

Commit c65cfda

Browse files
surishubhamsurishubham
authored
Merge pull request #2939 from surishubham/testmuCom
Merge pull request #2938 from sakshamarora-arch/feat/kane-cli-testmd-…
2 parents 5c16ade + f435d53 commit c65cfda

2 files changed

Lines changed: 388 additions & 0 deletions

File tree

‎docs/kane-cli-testmd.md‎

Lines changed: 383 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,383 @@
1+
---
2+
id: kane-cli-testmd
3+
title: Test.md
4+
sidebar_label: Test.md
5+
description: "Write browser tests as Markdown files that replay from cache after the first run: no LLM cost, faster execution. Commit tests to git, share recordings, and run in CI."
6+
keywords:
7+
- kane cli testmd
8+
- kane cli replayable tests
9+
- kane cli test files
10+
- kaneai
11+
- testmu ai
12+
- markdown tests
13+
- browser test automation
14+
url: https://www.testmuai.com/support/docs/kane-cli-testmd/
15+
site_name: TestMu AI
16+
slug: kane-cli-testmd/
17+
displayed_sidebar: KaneCLISidebar
18+
canonical: https://www.testmuai.com/support/docs/kane-cli-testmd/
19+
---
20+
21+
import Tabs from '@theme/Tabs';
22+
import TabItem from '@theme/TabItem';
23+
import BrandName, { BRAND_URL } from '@site/src/component/BrandName';
24+
25+
<script type="application/ld+json"
26+
dangerouslySetInnerHTML={{ __html: JSON.stringify({
27+
"@context": "https://schema.org",
28+
"@type": "BreadcrumbList",
29+
"itemListElement": [{
30+
"@type": "ListItem",
31+
"position": 1,
32+
"name": "Home",
33+
"item": "https://www.testmuai.com"
34+
},{
35+
"@type": "ListItem",
36+
"position": 2,
37+
"name": "Support",
38+
"item": "https://www.testmuai.com/support/docs/"
39+
},{
40+
"@type": "ListItem",
41+
"position": 3,
42+
"name": "Kane CLI",
43+
"item": "https://www.testmuai.com/support/docs/kane-cli-introduction/"
44+
}]
45+
})
46+
}}
47+
></script>
48+
49+
`testmd` lets you write browser tests as Markdown files (`_test.md`) and commit them to your repo. On the first run, the AI agent authors each step and saves a recording. On every subsequent run, each step **replays from cache** with no LLM cost and much faster execution. Commit the test file and its recordings to git so teammates and CI can re-run the same tests without re-authoring.
50+
51+
## Quick Start
52+
53+
**Step 1:** Create a test file. The filename must end in `_test.md`:
54+
55+
```markdown
56+
---
57+
mode: testing
58+
---
59+
60+
# Amazon search
61+
62+
## Open Amazon
63+
Open https://www.amazon.com.
64+
65+
## Search for headphones
66+
Type "wireless headphones" into the search box and submit.
67+
Verify at least one product result is visible.
68+
```
69+
70+
**Step 2:** Run the test:
71+
72+
```bash
73+
kane-cli testmd run amazon_test.md --agent
74+
```
75+
76+
On the first run, the agent authors each step and caches the recording. On every later run, the steps replay from cache instantly.
77+
78+
:::tip
79+
Always combine `--agent` with `--headless` in CI/CD environments to avoid display server errors.
80+
:::
81+
82+
---
83+
84+
## When to Use testmd vs run
85+
86+
`kane-cli run` is one-shot. It runs an objective, uploads results, and exits. It is ideal for quick, one-off verifications like checking if a page loads correctly or extracting a value from a live site.
87+
88+
`kane-cli testmd run` is for tests you want to persist. Use it when you are building a login flow smoke test, a regression suite, or any test you plan to re-run across builds. The test file lives in your repo, recordings are cached and committed alongside it, and every subsequent run replays from cache without consuming LLM credits. Teammates and CI pick up the same recordings and replay them identically.
89+
90+
If you run an ad-hoc objective with `kane-cli run` and later decide you want to keep it, use the `--name` flag to save it as a `_test.md` file (see [Recording a Test from a Live Session](#recording-a-test-from-a-live-session) below).
91+
92+
---
93+
94+
## File Format
95+
96+
A `_test.md` file has four parts in order:
97+
98+
### YAML Frontmatter
99+
100+
Configuration between `---` markers at the top of the file:
101+
102+
```yaml
103+
---
104+
mode: testing
105+
max_steps: 30
106+
headless: true
107+
---
108+
```
109+
110+
**Supported keys:**
111+
112+
| Key | Scope | Description |
113+
|-----|-------|-------------|
114+
| `mode` | root | `testing` (default) or `action`. Testing mode pushes through auth walls for negative-test assertions. Action mode halts on auth walls. |
115+
| `max_steps` | root + step | Max agent reasoning steps per step. Default: `30` |
116+
| `timeout` | root + step | Hard kill per step, in seconds |
117+
| `headless` | root | Run Chrome with no visible window |
118+
| `variables` | root + step | Variables with `{{name}}` syntax, same format as [Variables & Context](/support/docs/kane-cli-variables-and-context/) |
119+
| `code_export` | root + step | Generate Playwright code after the run |
120+
| `code_language` | root + step | `python` or `javascript` for code export |
121+
| `global_context` / `local_context` | root + step | Inline Markdown or file path for agent context |
122+
123+
### Title and Steps
124+
125+
After the frontmatter, add a `# H1` title for the test. This is purely decorative. Kane CLI ignores everything before the first `## ` heading.
126+
127+
Each `## H2` heading marks a test step. The heading text is a label for your reference; the agent reads the **step body** underneath it. Write the body as either plain English prose describing what the agent should do, or a single `@import <path>` line to pull in a reusable helper file. Do not mix prose and `@import` in the same step.
128+
129+
### Per-Step Config Overrides
130+
131+
You can override frontmatter settings for individual steps by adding a `yaml` fenced block immediately under the step heading:
132+
133+
````markdown
134+
## Submit the form
135+
```yaml
136+
timeout: 90
137+
optional: true
138+
```
139+
Click submit and verify the confirmation banner.
140+
````
141+
142+
Setting `optional: true` tells Kane CLI that a failure on this step should not fail the overall test. The run continues to the next step.
143+
144+
---
145+
146+
## Replay and Cascade Rule
147+
148+
This is the most important concept in `testmd`.
149+
150+
### Replay
151+
152+
After the first run, each step replays from its cached recording with no AI agent, no LLM cost, and much faster execution. A step replays only if **all** of these hold:
153+
154+
- A recording for that step exists
155+
- The step's prose is unchanged since the recording
156+
- The step's `yaml` block is unchanged
157+
- No earlier step invalidated it
158+
159+
### Cascade
160+
161+
Editing step N **re-authors step N and every step after it**. Each step starts where the previous step left off (URL, login state, open tabs). When step 3 changes, step 4 cannot safely replay against state that no longer exists.
162+
163+
:::warning
164+
A one-line tweak at the top of a 20-step test re-authors all 20 steps on the next run. To minimize re-authoring, edit only the steps you need to change.
165+
:::
166+
167+
**Useful commands:**
168+
169+
| Action | How |
170+
|--------|-----|
171+
| Re-record one step | Edit only that step (steps after it cascade automatically) |
172+
| Force full re-authoring | Use `--author` flag for one run |
173+
| Wipe cache entirely | Run `rm -rf output-<stem>/` |
174+
175+
---
176+
177+
## Reusing Flows with `@import`
178+
179+
Extract repeating flows (login, setup, cookie banner dismissal) into helper files:
180+
181+
```markdown
182+
## Sign in
183+
@import ./helpers/login.md
184+
```
185+
186+
**Rules:**
187+
188+
- Helper filename **must not** end in `_test.md`. Only `_test.md` files are valid entry points
189+
- Path resolves relative to the **importing file**, not the shell's working directory
190+
- The step body must be exactly `@import <path>` with no mixed prose or extra lines
191+
- The step's `yaml` block may contain **only** `optional`
192+
- `optional: true` on `@import` is allowed only at the root file level, not on nested imports
193+
- Variables and context propagate into helpers automatically
194+
195+
:::note
196+
Editing a helper re-authors that step in **every test that imports it**, plus everything after the import in those tests. The same cascade rule applies.
197+
:::
198+
199+
---
200+
201+
## Recording a Test from a Live Session
202+
203+
Run an ad-hoc objective with the `--name` flag to save it as a replayable test file:
204+
205+
```bash
206+
kane-cli run "Search for noise-cancelling headphones on amazon.com" --name amazon-search
207+
```
208+
209+
On exit, Kane CLI writes the test file to `.testmuai/tests/amazon-search_test.md`. Move that file into your repo and re-run it with `testmd run`.
210+
211+
:::note
212+
Without `--name`, ad-hoc runs are ephemeral and nothing is written to disk.
213+
:::
214+
215+
---
216+
217+
## Commands
218+
219+
| Command | Purpose |
220+
|---------|---------|
221+
| `kane-cli testmd run <path> --agent` | Run a test file |
222+
| `kane-cli testmd list` | List `*_test.md` files under the current directory |
223+
| `kane-cli testmd status <path>` | Show Test Manager identity and local sync state |
224+
| `kane-cli testmd export <path>` | Regenerate code export from existing recordings (no browser launch) |
225+
| `kane-cli testmd delete <path>` | Delete the test and its `output-<stem>/` cache locally. Does NOT delete from Test Manager |
226+
227+
---
228+
229+
## Flags for `testmd run`
230+
231+
All [`kane-cli run` flags](/support/docs/kane-cli-cli-reference/) apply (`--agent`, `--headless`, `--max-steps`, `--timeout`, `--variables`, etc.), plus these additional flags:
232+
233+
| Flag | Default | Description |
234+
|------|---------|-------------|
235+
| `--name <name>` | none | Persist the run under this name. Must match `[a-zA-Z0-9_-]+` |
236+
| `--on-lock-conflict <mode>` | none | Behavior when another user holds the test's edit lock: `readonly` (replay-only, no upload), `fail` (exit 2), or `wait` (block until released) |
237+
| `--retry` | off | On replay failure, restart with a shrinking replay window |
238+
| `--retry-count <n>` | `3` | Max retry restarts before falling back to full re-author |
239+
| `--author` | off | Force authoring every step, skipping replay entirely |
240+
| `--code-language <lang>` | `python` | Code export language: `python` or `javascript` |
241+
242+
:::note
243+
Flag values win over frontmatter for all settings **except** `variables`. The file owns variables. You can add new keys via flags but cannot override file-defined ones.
244+
:::
245+
246+
---
247+
248+
## Output Directory
249+
250+
After a run, Kane CLI creates an output directory next to the test file:
251+
252+
```
253+
amazon_test.md
254+
output-amazon/
255+
Result.md # Human-readable run report
256+
.internal/ # Cached recordings, do not edit
257+
playwright-python-code/ # Only if code_export is enabled
258+
```
259+
260+
`output-<stem>/` is **commit-safe**. Commit it to git so teammates and CI replay the same recordings.
261+
262+
For tests using `@import`, helper recordings land next to the helper file in `helper-output-<helper>-<root>-<step>/` directories. These are also commit-safe.
263+
264+
### Result.md
265+
266+
After each run, `Result.md` is generated with:
267+
268+
| Section | Contents |
269+
|---------|----------|
270+
| Frontmatter | `status`, `started`, `duration_s`, `session_id` |
271+
| Step results | One entry per step: `✓ passed`, `✗ failed`, or `⏭ skipped` (suffixed `(optional)` for soft-failing optional steps) |
272+
| Import failure paths | For `@import` steps that failed, the path to the failing sub-step inside the helper |
273+
274+
:::tip
275+
To check whether a test passed or where it failed, read `Result.md` instead of re-running the test.
276+
:::
277+
278+
---
279+
280+
## CI/CD Usage
281+
282+
```bash
283+
kane-cli testmd run ./tests/checkout_test.md \
284+
--agent \
285+
--headless \
286+
--on-lock-conflict wait \
287+
--retry
288+
```
289+
290+
| Flag | Purpose |
291+
|------|---------|
292+
| `--agent` | Structured NDJSON to stdout |
293+
| `--headless` | No browser window |
294+
| `--on-lock-conflict wait` | Block instead of failing if a teammate is editing the same test |
295+
| `--retry` | Automatically recover from transient replay failures |
296+
297+
### Exit Codes
298+
299+
| Code | Meaning |
300+
|------|---------|
301+
| 0 | ✅ Passed |
302+
| 1 | ❌ Failed (test ran but did not pass) |
303+
| 2 | ⚠️ Error (auth, setup, parse error, or `--on-lock-conflict fail`) |
304+
| 3 | ⏱️ Timeout, cancelled, or `--on-lock-conflict wait` timed out |
305+
306+
---
307+
308+
## Common Parse Errors
309+
310+
Parse errors abort **before** any browser launch with exit code `2`:
311+
312+
| Error Message | Fix |
313+
|---------------|-----|
314+
| `frontmatter is missing closing '---'` | Add the trailing `---` |
315+
| `invalid YAML in frontmatter` | Validate the YAML block |
316+
| `step body must be exactly one of prose / @import` | Split into two steps |
317+
| `step config on @import may only contain 'optional'` | Remove other keys from the yaml block |
318+
| `cannot @import a test file` | Imports may only reference helpers (not files ending in `_test.md`) |
319+
| `cyclic reference` | Restructure helpers to break the circular dependency |
320+
| `chrome config is global-only` | Move Chrome key to root frontmatter |
321+
| `'<key>' is run-level and cannot be set per-step` | Move `mode` / `on_lock_conflict` to root frontmatter |
322+
| `unknown config key` | Remove or fix the key |
323+
| `auth/identity keys are CLI-only` | Pass `username` / `access_key` as CLI flags, not in frontmatter |
324+
325+
---
326+
327+
## Example: Full Test with Imports
328+
329+
**`tests/checkout_test.md`:**
330+
331+
```markdown
332+
---
333+
mode: testing
334+
headless: true
335+
variables:
336+
username:
337+
value: "testuser@example.com"
338+
password:
339+
value: "s3cret!"
340+
secret: true
341+
---
342+
343+
# Checkout flow
344+
345+
## Login
346+
@import ./helpers/login.md
347+
348+
## Add item to cart
349+
Go to the products page, search for "wireless headphones", and click "Add to Cart" on the first result.
350+
351+
## Verify cart
352+
Go to the cart page. Assert the cart contains 1 item. Store the total price as 'cart_total'.
353+
354+
## Complete checkout
355+
Click "Proceed to Checkout", fill in shipping details, and assert the order confirmation page loads.
356+
```
357+
358+
**`tests/helpers/login.md`:**
359+
360+
```markdown
361+
Go to https://app.example.com/login.
362+
Enter {{username}} in the email field.
363+
Enter {{password}} in the password field.
364+
Click "Sign In".
365+
Assert the dashboard loads.
366+
```
367+
368+
**Run:**
369+
370+
```bash
371+
kane-cli testmd run tests/checkout_test.md --agent
372+
```
373+
374+
---
375+
376+
## Next Steps
377+
378+
- [Writing Objectives](/support/docs/kane-cli-writing-objectives/): Learn how to write effective natural language objectives
379+
- [Variables & Context](/support/docs/kane-cli-variables-and-context/): Parameterize tests with variables, secrets, and context files
380+
- [Agent Mode](/support/docs/kane-cli-agent-mode/): Parse structured NDJSON output from Kane CLI
381+
- [CI/CD Integration](/support/docs/kane-cli-cicd/): Add Kane CLI to your pipeline
382+
- [CLI Reference](/support/docs/kane-cli-cli-reference/): Full flag and command reference
383+
- [Troubleshooting](/support/docs/kane-cli-troubleshooting/): Debug common issues

‎sidebars.js‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5256,6 +5256,11 @@ module.exports = {
52565256
label: "Test Manager Integration",
52575257
id: "kane-cli-tms-integration",
52585258
},
5259+
{
5260+
type: "doc",
5261+
label: "Test.md",
5262+
id: "kane-cli-testmd",
5263+
},
52595264
],
52605265
},
52615266
{

0 commit comments

Comments
 (0)