Skip to content

Commit fe446a8

Browse files
borgheiborghei
committed
feat(pm): tier 3 — output eval harness for 12 artifact-generating PM skills
Final layer of the depth investment. Deterministic scoring of any markdown artifact against per-skill rubrics. No LLM in the loop, stdlib only. FRAMEWORK (evals/) - engine.py — generic check evaluator with 13 check types: regex, regex_not, section_present, section_word_count, section_sentence_count, keyword_any, keyword_all, keyword_none, has_table, has_list, url_count, length_in_range, line_count_range. - run.py — runner with --all, --skill, --artifact, --format, --threshold, --output flags. - README.md — framework documentation + how to add a new rubric. RUBRICS (12 skills, 190+ criteria total) Each rubric anchors every criterion to a specific entry in the skill's references/red-flags.md or SKILL.md Success Criteria: - create-prd — 17 criteria scoring against 8-section PRD model - prfaq — 19 criteria scoring against Amazon Working Backwards - ai-feature-prd — 21 criteria scoring against eval/guardrails/model-fallback - brainstorm-okrs — 19 criteria scoring against SMART + Wodtke confidence - status-update — 19 criteria scoring against SBNR + R/Y/G + Asks - post-mortem — 19 criteria scoring against blameless + 5 Whys + owners - north-star-metric — 19 criteria scoring against NSM + input tree + counter - product-vision — 19 criteria scoring against Pichler / Raskin structure - pricing-prd — 20 criteria scoring against Westendorp + grandfathering - roadmap-comms — 19 criteria scoring against 3-variant audience fit - release-notes — 19 criteria scoring against value framing + categories - feedback-triage — 19 criteria scoring against Kano + RICE + ack templates SMOKE TEST (all 12 worked examples self-scored) - 6 skills @ 100/100 (ai-feature-prd, customer-feedback-triage, north-star-metric, post-mortem, product-vision, status-update-generator) - 6 skills @ 89-95/100 (brainstorm-okrs 95, pricing-prd 95, prfaq 94, roadmap-communication 94, release-notes 90, create-prd 89) - 12/12 pass at threshold 70 PM README + CHANGELOG updated. This commit completes the 3-tier PM depth investment: - Tier 1 (505cb8e): worked examples (54), data adapters (3), MCP tools (15) - Tier 2 (a8063e1): red-flag libraries (54), runnable pipelines (5) - Tier 3 (this): output eval harness (12 rubrics) + framework PM domain: 54 skills, 78 sub-files of worked examples, 54 red-flag libraries, 12 deterministic scorers, 5 chain pipelines, 3 live data adapters, 15 MCP tools. The skills moved from "documentation" to "production toolkit."
1 parent a8063e1 commit fe446a8

17 files changed

Lines changed: 2103 additions & 0 deletions

File tree

‎CHANGELOG.md‎

Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,45 @@ All notable changes to the Claude Skills Library will be documented in this file
55
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
66
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
77

8+
## [4.7.0] - 2026-05-22 (Tier 3 of PM depth)
9+
10+
### Added
11+
12+
**Output evaluation harness for artifact-generating PM skills (`evals/`).** Deterministic scoring (no LLM) for any markdown artifact, scored 0-100 against the skill's red-flags + success criteria.
13+
14+
Framework:
15+
- `evals/engine.py` — generic check evaluator. 13 supported check types: regex, regex_not, section_present, section_word_count, section_sentence_count, keyword_any, keyword_all, keyword_none, has_table, has_list, url_count, length_in_range, line_count_range.
16+
- `evals/run.py` — runner. Supports `--all`, `--skill <name>`, `--artifact <path>`, `--format markdown|json`, `--threshold <int>`, `--output <file>`.
17+
- `evals/README.md` — framework documentation.
18+
19+
12 rubrics (190+ criteria total, each anchored to a specific red-flag or success-criterion item):
20+
- create-prd · prfaq · ai-feature-prd · brainstorm-okrs · status-update-generator
21+
- post-mortem · north-star-metric · product-vision · pricing-prd
22+
- roadmap-communication · release-notes · customer-feedback-triage
23+
24+
Smoke-test (all 12 worked examples scored against their own rubric):
25+
26+
| Skill | Score |
27+
|---|---|
28+
| ai-feature-prd | 100 |
29+
| customer-feedback-triage | 100 |
30+
| north-star-metric | 100 |
31+
| post-mortem | 100 |
32+
| product-vision | 100 |
33+
| status-update-generator | 100 |
34+
| brainstorm-okrs | 95 |
35+
| pricing-prd | 95 |
36+
| prfaq | 94 |
37+
| roadmap-communication | 94 |
38+
| release-notes | 90 |
39+
| create-prd | 89 |
40+
41+
12/12 pass at threshold 70. High scores reflect that the worked examples were authored as exemplars; the rubrics are calibrated to catch realistic failures (drafts with missing sections, blame language, output-as-KR, watermelon status, etc.).
42+
43+
### Changed
44+
45+
- PM README updated with new "Output evaluation harness" section.
46+
847
## [4.6.0] - 2026-05-22 (Tier 2 of PM depth)
948

1049
### Added

‎evals/README.md‎

Lines changed: 139 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,139 @@
1+
# PM Skill Output Evaluation Harness
2+
3+
Deterministic scoring for artifacts generated by PM skills. Catches output-quality regressions without an LLM in the loop.
4+
5+
**Why:** PM skills produce artifacts (PRDs, post-mortems, OKRs, status updates, etc.). Without a scoring mechanism, "is this output good?" is a vibes call. The harness encodes the *red flags* and *success criteria* from each skill as concrete checks against the artifact.
6+
7+
## How it works
8+
9+
```
10+
evals/
11+
├── README.md # this file
12+
├── engine.py # generic check evaluator (stdlib only)
13+
├── run.py # runner: iterates rubrics, scores examples
14+
└── <skill>/
15+
└── rubric.json # per-skill rubric — list of weighted checks
16+
```
17+
18+
For each skill, a `rubric.json` lists weighted criteria. Each criterion has a `check` block with a `type` and parameters. The engine knows how to evaluate each type. The runner finds artifacts (typically in the skill's `examples/` folder) and scores them.
19+
20+
## Supported check types
21+
22+
| Type | Purpose | Parameters |
23+
|---|---|---|
24+
| `regex` | Pattern must match | `pattern`, `flags` (default `im`) |
25+
| `regex_not` | Pattern must NOT match | `pattern`, `flags` |
26+
| `section_present` | A markdown heading exists | `heading`, `level` (default 2) |
27+
| `section_word_count` | Section length in range | `heading`, `min`, `max` |
28+
| `section_sentence_count` | Section sentence count in range | `heading`, `min`, `max` |
29+
| `keyword_any` | At least one keyword present | `keywords` (list) |
30+
| `keyword_none` | All listed keywords absent | `keywords` (list) |
31+
| `keyword_all` | All listed keywords present | `keywords` (list) |
32+
| `has_table` | Contains a markdown table | `min_rows` (optional) |
33+
| `has_list` | Contains a bullet or numbered list | `min_items` (optional) |
34+
| `url_count` | URL count in range | `min`, `max` |
35+
| `length_in_range` | Total length (chars) in range | `min`, `max` |
36+
| `line_count_range` | Line count in range | `min`, `max` |
37+
38+
A criterion passes (full weight) or fails (zero). For partial credit, use multiple smaller criteria.
39+
40+
## Rubric format
41+
42+
```json
43+
{
44+
"skill": "create-prd",
45+
"version": "1.0.0",
46+
"description": "Scores an 8-section PRD against red-flags + success criteria.",
47+
"max_score": 100,
48+
"criteria": [
49+
{
50+
"id": "summary_present",
51+
"name": "Section 1 (Summary) present",
52+
"weight": 8,
53+
"check": {"type": "section_present", "heading": "Summary"}
54+
},
55+
{
56+
"id": "summary_concise",
57+
"name": "Summary is 2-4 sentences (10-second exec test)",
58+
"weight": 5,
59+
"check": {"type": "section_sentence_count", "heading": "Summary", "min": 2, "max": 4}
60+
},
61+
{
62+
"id": "no_jargon",
63+
"name": "No consultant-speak",
64+
"weight": 4,
65+
"check": {"type": "keyword_none", "keywords": ["synergy", "leverage", "deep-dive", "circle back", "low-hanging fruit"]}
66+
}
67+
]
68+
}
69+
```
70+
71+
## Running
72+
73+
```bash
74+
# Score one skill's worked example
75+
python evals/run.py --skill create-prd
76+
77+
# Score all skills
78+
python evals/run.py --all
79+
80+
# Score a specific artifact file (any markdown)
81+
python evals/run.py --skill create-prd --artifact path/to/my-prd.md
82+
83+
# JSON output for CI
84+
python evals/run.py --all --format json
85+
```
86+
87+
## Output
88+
89+
Markdown summary by default:
90+
91+
```
92+
# Evaluation Report
93+
94+
## create-prd / examples/shared-dashboards-prd.md
95+
Score: 87 / 100 (PASS — threshold 70)
96+
97+
### Passed (8 of 10 criteria)
98+
- ✅ Section 1 (Summary) present (8)
99+
- ✅ Summary is 2-4 sentences (5)
100+
- ...
101+
102+
### Failed (2 of 10)
103+
- ❌ Has explicit Assumptions table (8) — section "Assumptions" not found
104+
- ❌ KR1 has baseline -> target -> deadline format (6)
105+
```
106+
107+
## Adding a new rubric
108+
109+
1. Create `evals/<skill>/rubric.json`
110+
2. Anchor each criterion to a specific entry in `<skill>/references/red-flags.md` or the SKILL.md Success Criteria
111+
3. Total weights should sum to 100 (the runner normalizes if they don't)
112+
4. Run `python evals/run.py --skill <skill>` to test
113+
114+
## Limitations
115+
116+
- **No semantic understanding.** Checks are structural / lexical. The engine can verify "does the artifact have a Section 7 Solution?" but not "is the solution actually a solution?"
117+
- **No autograding of judgment.** A PR/FAQ scoring 95/100 may still describe a bad product. The harness catches *form* failures, not *substance* failures.
118+
- **Calibration matters.** Initial weights are author-curated guesses. Adjust after running against 5-10 real artifacts.
119+
120+
## Covered skills
121+
122+
Current rubrics:
123+
124+
| Skill | Rubric | Anchored to |
125+
|---|---|---|
126+
| create-prd | rubric.json | SKILL.md 8 sections + red-flags |
127+
| prfaq | rubric.json | Amazon Working Backwards |
128+
| ai-feature-prd | rubric.json | eval-spec + guardrails + model-selection sections |
129+
| brainstorm-okrs | rubric.json | SMART + Wodtke confidence |
130+
| status-update-generator | rubric.json | SBNR + R/Y/G + Asks |
131+
| post-mortem | rubric.json | Blameless + 5 Whys + action items with owners |
132+
| north-star-metric | rubric.json | NSM + input tree + counter-metrics |
133+
| product-vision | rubric.json | Pichler Vision Board / 5-10-year horizon |
134+
| pricing-prd | rubric.json | Westendorp + grandfathering + rollback |
135+
| roadmap-communication | rubric.json | Three-variant audience fit |
136+
| release-notes | rubric.json | Value framing + categorization |
137+
| customer-feedback-triage | rubric.json | Kano + scoring + ack template |
138+
139+
12 rubrics total.

‎evals/ai-feature-prd/rubric.json‎

Lines changed: 133 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,133 @@
1+
{
2+
"skill": "ai-feature-prd",
3+
"version": "1.0.0",
4+
"description": "Scores an 11-section AI Feature PRD against AI-specific success criteria: model selection w/ fallback, numeric eval thresholds, guardrails, deployment ramp w/ gates, HIL, EU AI Act tier — anchored to references/red-flags.md.",
5+
"criteria": [
6+
{
7+
"id": "summary_present",
8+
"name": "Section 1 (Summary) names the model class explicitly",
9+
"weight": 5,
10+
"check": {"type": "keyword_any", "keywords": ["Claude", "GPT", "Sonnet", "Haiku", "Llama", "Gemini", "BERT", "fine-tuned", "RAG", "LLM"]}
11+
},
12+
{
13+
"id": "objective_present",
14+
"name": "Section 4 (Objective) with Key Results",
15+
"weight": 5,
16+
"check": {"type": "regex", "pattern": "KR\\d|Key Result", "flags": "im"}
17+
},
18+
{
19+
"id": "ai_system_design_section",
20+
"name": "Section 9 (AI System Design) present",
21+
"weight": 6,
22+
"check": {"type": "regex", "pattern": "^##\\s+(?:9\\.?\\s+|Section\\s+9:?\\s+)?AI System Design", "flags": "im"}
23+
},
24+
{
25+
"id": "model_selection_with_fallback",
26+
"name": "Model selection includes a primary AND a fallback",
27+
"weight": 7,
28+
"check": {"type": "keyword_all", "keywords": ["primary", "fallback"]}
29+
},
30+
{
31+
"id": "switch_trigger",
32+
"name": "Model switch trigger named (outage, regression, cost cap)",
33+
"weight": 5,
34+
"check": {"type": "keyword_any", "keywords": ["switch trigger", "fail over", "outage", "regression", "cost cap"]}
35+
},
36+
{
37+
"id": "architecture_pattern",
38+
"name": "Architecture pattern declared (prompt / RAG / fine-tune / agent) with rejected alternatives",
39+
"weight": 5,
40+
"check": {"type": "keyword_any", "keywords": ["Prompt + RAG", "RAG", "fine-tune", "agentic", "rejected", "alternatives"]}
41+
},
42+
{
43+
"id": "eval_safety_section",
44+
"name": "Section 10 (Eval & Safety Plan) present",
45+
"weight": 7,
46+
"check": {"type": "regex", "pattern": "^##\\s+(?:10\\.?\\s+|Section\\s+10:?\\s+)?Eval", "flags": "im"}
47+
},
48+
{
49+
"id": "eval_numeric_thresholds",
50+
"name": "Eval section has numeric thresholds (%, ms, $)",
51+
"weight": 7,
52+
"check": {"type": "regex", "pattern": "(acceptance.*\\d+%|hallucinat.*\\d+%|>= ?\\d+%|<= ?\\d+%|p95.*\\d+\\s?ms|\\$0\\.\\d+)", "flags": "im"}
53+
},
54+
{
55+
"id": "hallucination_target",
56+
"name": "Hallucination rate target named",
57+
"weight": 5,
58+
"check": {"type": "regex", "pattern": "hallucinat[a-z]+\\s*(rate)?\\s*(<=|<|under|less than|of)?\\s*\\d", "flags": "im"}
59+
},
60+
{
61+
"id": "refusal_policy",
62+
"name": "Refusal policy explicit (enumerated categories, not one sentence)",
63+
"weight": 5,
64+
"check": {"type": "keyword_any", "keywords": ["Refusal policy", "refuses to", "never:", "redirect"]}
65+
},
66+
{
67+
"id": "golden_set_committed",
68+
"name": "Golden set / eval set is concrete (size + composition, not 'TBD')",
69+
"weight": 6,
70+
"check": {"type": "keyword_any", "keywords": ["golden set", "gold set", "200-item", "evals/", "adversarial", "common", "edge cases"]}
71+
},
72+
{
73+
"id": "no_tbd_eval",
74+
"name": "No 'golden set TBD' or 'eval TBD' deferral",
75+
"weight": 4,
76+
"check": {"type": "regex_not", "pattern": "(golden\\s*set|gold\\s*set|eval[a-z]*)\\s*:?\\s*TBD", "flags": "im"}
77+
},
78+
{
79+
"id": "guardrails_layers",
80+
"name": "Guardrails section enumerates layers (input / output / HIL / etc.)",
81+
"weight": 5,
82+
"check": {"type": "keyword_all", "keywords": ["input", "output", "guardrail"]}
83+
},
84+
{
85+
"id": "human_in_the_loop",
86+
"name": "Human-in-the-loop checkpoints declared",
87+
"weight": 6,
88+
"check": {"type": "keyword_any", "keywords": ["human-in-the-loop", "human in the loop", "HIL", "hard gate", "soft gate", "sampling review"]}
89+
},
90+
{
91+
"id": "failure_modes_table",
92+
"name": "Failure modes enumerated with detection + response",
93+
"weight": 5,
94+
"check": {"type": "keyword_all", "keywords": ["failure", "detection"]}
95+
},
96+
{
97+
"id": "eu_ai_act_tier",
98+
"name": "EU AI Act risk tier declared (minimal / limited / high / unacceptable)",
99+
"weight": 6,
100+
"check": {"type": "keyword_any", "keywords": ["EU AI Act", "Limited Risk", "High Risk", "High-risk", "Minimal Risk", "risk tier"]}
101+
},
102+
{
103+
"id": "operations_cost_section",
104+
"name": "Section 11 (Operations & Cost) present",
105+
"weight": 5,
106+
"check": {"type": "regex", "pattern": "^##\\s+(?:11\\.?\\s+|Section\\s+11:?\\s+)?(Operations|Cost)", "flags": "im"}
107+
},
108+
{
109+
"id": "cost_model_with_scale",
110+
"name": "Cost model includes 10x / scale scenario or per-tenant alerts",
111+
"weight": 5,
112+
"check": {"type": "keyword_any", "keywords": ["10x", "per-tenant", "80% of budget", "auto-throttle", "throttle at", "cost-per", "$/Mtok"]}
113+
},
114+
{
115+
"id": "deployment_ramp_gates",
116+
"name": "Deployment ramp shadow -> internal -> canary -> GA with gate metrics",
117+
"weight": 6,
118+
"check": {"type": "keyword_all", "keywords": ["shadow", "canary"]}
119+
},
120+
{
121+
"id": "no_100_percent_target",
122+
"name": "No naive 100% acceptance target (red flag #4)",
123+
"weight": 3,
124+
"check": {"type": "regex_not", "pattern": "acceptance.{0,30}(100%|>=\\s*99%|99\\.9%)", "flags": "im"}
125+
},
126+
{
127+
"id": "no_buzzwords",
128+
"name": "No marketing buzzwords",
129+
"weight": 2,
130+
"check": {"type": "keyword_none", "keywords": ["revolutionary", "next-generation", "world-class", "seamless"]}
131+
}
132+
]
133+
}

0 commit comments

Comments
 (0)