feat(generate): ship draft suite generation, approval gating, and CI review flow

Author: hidai25

README.md

@@ -65,9 +65,9 @@ The first two layers alone catch most regressions — fully offline, zero cost.
 ### The workflow
 
 ```bash
-evalview capture --agent http://localhost:8000/invoke   # 1. Record real interactions
-evalview snapshot                                        # 2. Save as baseline
-evalview check                                           # 3. Catch regressions
+evalview generate --agent http://localhost:8000         # 1. Draft a regression suite
+evalview snapshot tests/generated --approve-generated   # 2. Approve + baseline
+evalview check tests/generated                          # 3. Catch regressions
 evalview monitor                                         # 4. Watch continuously (+ Slack alerts)
 # ✅ All clean — or ❌ REGRESSION: score 85 → 71
 ```
@@ -76,7 +76,9 @@ evalview monitor                                         # 4. Watch continuously
 
 Choose the shortest path for your use case:
 
-- New project: `evalview capture --agent ...` → `evalview snapshot` → `evalview check`
+- New project, no traffic yet: `evalview generate --agent ...` → `evalview snapshot --approve-generated` → `evalview check`
+- Existing traffic or staging logs: `evalview generate --from-log traffic.jsonl`
+- Production-shaped tests from real usage: `evalview capture --agent ...` → `evalview snapshot` → `evalview check`
 - Existing tests, no baselines yet: `evalview snapshot`
 - CI gate for regressions: [Golden Traces](docs/GOLDEN_TRACES.md) and [CI/CD Integration](docs/CI_CD.md)
 - Framework-specific setup: [Framework Support](docs/FRAMEWORK_SUPPORT.md)
@@ -245,7 +247,23 @@ evalview check --semantic-diff
 pip install evalview
 ```
 
-### Step 1 — Capture real interactions as tests
+### Step 1 — Generate or capture tests
+
+If you have no test suite yet, start with generation:
+
+```bash
+evalview generate --agent http://localhost:8000
+# Writes draft YAML tests to tests/generated/
+# Also writes tests/generated/generated.report.json for CI review
+```
+
+If you already have logs from staging or production:
+
+```bash
+evalview generate --from-log traffic.jsonl
+```
+
+If you want tests based on real user flows instead of planned probes:
 
 ```bash
 evalview capture --agent http://localhost:8000/invoke
@@ -254,9 +272,19 @@ evalview capture --agent http://localhost:8000/invoke
 # Tests are saved to tests/test-cases/ automatically
 ```
 
-> **Why capture first?** Tests from real usage catch real regressions. Auto-generated tests from guessed queries score poorly and give you false confidence.
+> **When to use which?**
+> `generate` is the fastest path from zero to a draft suite.
+> `capture` is the highest-signal path when you already have real usage to replay.
+
+### Step 2 — Review and save as your baseline
 
-### Step 2 — Save as your baseline
+Generated tests are draft-only until you approve them:
+
+```bash
+evalview snapshot tests/generated --approve-generated
+```
+
+Captured or hand-written tests snapshot normally:
 
 ```bash
 export OPENAI_API_KEY='your-key'   # for LLM-as-judge scoring
@@ -269,6 +297,14 @@ evalview snapshot
 evalview check   # run this after every change
 ```
 
+### Review generated suites in CI
+
+```bash
+evalview ci comment --results tests/generated/generated.report.json --dry-run
+```
+
+That review comment summarizes discovered tools, generated behavior paths, coverage gaps, and the approval workflow before baselining.
+
 ### No agent yet? Try the demo
 
 ```bash

docs/CI_CD.md

@@ -6,6 +6,34 @@
 
 EvalView is CLI-first. You can run it locally or add to CI.
 
+## Review Generated Suites in PRs
+
+If you use `evalview generate`, every run writes a machine-readable suite report:
+
+```bash
+tests/generated/generated.report.json
+```
+
+Turn that into a PR comment:
+
+```bash
+evalview ci comment --results tests/generated/generated.report.json
+```
+
+The generated-suite comment includes:
+- discovered tools
+- draft behavior paths
+- coverage gaps
+- approval instructions for `snapshot --approve-generated`
+
+Recommended flow:
+
+```bash
+evalview generate --agent http://localhost:8000
+evalview ci comment --results tests/generated/generated.report.json
+evalview snapshot tests/generated --approve-generated
+```
+
 ---
 
 ## GitHub Action (Recommended)

docs/CLI_REFERENCE.md

@@ -110,6 +110,7 @@ Options:
   -t, --test TEXT     Snapshot only this specific test
   -n, --notes TEXT    Notes about this snapshot
   --variant TEXT      Save as named variant (max 5 per test)
+  --approve-generated Approve generated draft tests before snapshotting
 ```
 
 ### Examples
@@ -118,6 +119,7 @@ Options:
 evalview snapshot                           # Snapshot all passing tests
 evalview snapshot --test "my-test"          # Snapshot one test
 evalview snapshot --variant v2             # Save alternate acceptable behavior
+evalview snapshot tests/generated --approve-generated
 ```
 
 ---
@@ -150,6 +152,43 @@ evalview check --dry-run                    # Preview plan, no API calls
 evalview check --budget 0.50               # Cap spend at $0.50
 ```
 
+## `evalview generate`
+
+Generate a draft regression suite from a live agent or existing traffic logs.
+
+```bash
+evalview generate [OPTIONS]
+
+Options:
+  --agent URL                  Agent endpoint URL
+  --adapter TEXT               Adapter type (default: config or http)
+  --budget N                   Maximum probe runs / imported entries
+  --out DIR                    Output directory (default: tests/generated)
+  --seed FILE                  Newline-delimited seed prompts
+  --from-log PATH              Generate from a log file instead of live probing
+  --log-format FORMAT          auto|jsonl|openai|evalview
+  --include-tools TEXT         Comma-separated tool names to focus on
+  --exclude-tools TEXT         Comma-separated tool names to avoid
+  --allow-live-side-effects    Allow side-effecting prompts
+  --timeout FLOAT              Probe timeout in seconds
+  --dry-run                    Preview without writing files
+```
+
+### Examples
+
+```bash
+evalview generate --agent http://localhost:8000
+evalview generate --from-log traffic.jsonl
+evalview generate --agent http://localhost:8000 --include-tools search,calendar
+evalview generate --dry-run
+```
+
+Generated suites are draft-only until approved:
+
+```bash
+evalview snapshot tests/generated --approve-generated
+```
+
 ---
 
 ## `evalview expand`

docs/README.md

@@ -15,6 +15,7 @@ If you're new:
 | I want to… | Read this first | Then |
 |------------|-----------------|------|
 | Get EvalView running quickly | [Getting Started](GETTING_STARTED.md) | [CLI Reference](CLI_REFERENCE.md) |
+| Go from zero tests to a draft suite | [Test Generation](TEST_GENERATION.md) | [CI/CD Integration](CI_CD.md) |
 | Understand regression detection | [Golden Traces](GOLDEN_TRACES.md) | [Evaluation Metrics](EVALUATION_METRICS.md) |
 | Test a specific framework | [Framework Support](FRAMEWORK_SUPPORT.md) | the matching quick start below |
 | Set up CI/CD | [CI/CD Integration](CI_CD.md) | [Golden Traces](GOLDEN_TRACES.md) |
@@ -41,7 +42,7 @@ If you're new:
 | [Suite Types](SUITE_TYPES.md) | Separate capability tests from regression tests |
 | [Behavior Coverage](BEHAVIOR_COVERAGE.md) | Track gaps in the behaviors you test |
 | [Cost Tracking](COST_TRACKING.md) | Understand token and dollar usage |
-| [Test Generation](TEST_GENERATION.md) | Expand a seed test into broader coverage |
+| [Test Generation](TEST_GENERATION.md) | Generate a draft suite from an agent or logs |
 | [Trace Specification](TRACE_SPEC.md) | Execution trace format used across adapters |
 
 ## Frameworks