diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 51a4440..ca6288e 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -32,3 +32,13 @@ request features. Before filing a new issue, please check if one already exists. If you discover a security vulnerability, please follow the instructions in [SECURITY.md](SECURITY.md). **Do not** report security vulnerabilities through public GitHub issues. + +## Creating Rules and Metrics + +Detection rules and metrics are the primary extensibility surface of AI Engineer Coach. Each one is +a self-contained markdown file with YAML frontmatter and a small DSL — no code changes required to +ship a new one. Built-in rules live in [`src/core/rules/`](src/core/rules/) and metrics in +[`src/core/metrics/`](src/core/metrics/). + +See [docs/AUTHORING_RULES.md](docs/AUTHORING_RULES.md) for the full authoring guide: file format, +annotated rule and metric examples, the local testing workflow, and links to the DSL reference. diff --git a/docs/AUTHORING_RULES.md b/docs/AUTHORING_RULES.md new file mode 100644 index 0000000..619dc3b --- /dev/null +++ b/docs/AUTHORING_RULES.md @@ -0,0 +1,145 @@ +# Authoring Rules and Metrics + +Detection rules and metrics are the primary extensibility surface of AI Engineer Coach. Every rule +and metric is a self-contained markdown file with YAML frontmatter and a small DSL — no code changes +required to ship a new one. + +This guide covers contributing a built-in rule or metric to this repository. For the in-extension +authoring flow (live-test, threshold sliders, AI-assisted drafting), see the +[Rule Editor guide](https://microsoft.github.io/AI-Engineering-Coach/improve/rule-editor/). + +## Where rules and metrics live + +| Layer | Location | Trust | Use when | +|---|---|---|---| +| Built-in | [`src/core/rules/`](../src/core/rules/), [`src/core/metrics/`](../src/core/metrics/) | Trusted | Contributing a rule for everyone via this repo | +| Personal | `~/.ai-engineer-coach/rules/`, `~/.ai-engineer-coach/metrics/` | Prompted on first load | Private rules shared across all your workspaces | +| Project | `/.ai-engineer-coach/rules/`, `/.ai-engineer-coach/metrics/` | Prompted on first load | Workspace-specific rules checked into a repo | + +Personal and project rules follow the same file format as built-in rules but are loaded at runtime +through the trust gate in [`src/core/rule-trust.ts`](../src/core/rule-trust.ts). + +For the full DSL reference — field schema, function catalog, and metric primitives — open the +**DSL Reference** modal inside the extension (Rule Editor → DSL Reference) or the +[Rule Playground](https://microsoft.github.io/AI-Engineering-Coach/improve/rule-playground/). + +## Anatomy of a rule + +A rule file lives at `src/core/rules/.md`. Below is +[`lazy-prompting.md`](../src/core/rules/lazy-prompting.md) annotated section by section: + +````markdown +--- +id: lazy-prompting # Stable identifier, must match the filename +name: Lazy Prompting # Human-readable name shown in the UI +group: prompt-quality # Practice category: prompt-quality | session-hygiene | + # code-review | tool-mastery | context-management +severity: medium # low | medium | high — drives score weighting +scope: requests # requests (per-message) or sessions (per-session) +version: 1 # Bump when changing detection logic +tags: [prompt, quality, short] # Free-form tags surfaced in the rule filter +thresholds: # Tunable knobs. Referenced as thresholds. in DSL. + minChars: 30 # Sliders for these appear in the Rule Editor. + maxRatio: 0.3 + minSample: 10 +--- + +# Description +Short, user-facing summary of what this rule detects. + +# When Triggered +One-line finding text. Supports template variables: {{count}}, {{total}}, {{pct}}, +and anything you emit via extra. in the detection logic. + +# How to Improve +Concrete, actionable recommendation shown alongside the finding. + +# Examples +Template for the per-occurrence example list. {{message}} and any field on the +matched row (e.g. {{extra.charCount}}) are available. + +# Detection Logic +```detect +scan: requests # What to iterate over +match: messageLength < thresholds.minChars # Predicate; uses field schema + AND messageLength > 0 # + thresholds.* + DSL helpers +aggregate: ratio # count | ratio | sum | someWhere | ... +check: ratio > thresholds.maxRatio # The trigger condition + AND count > thresholds.minSample +examples: "{{messageText | truncate:80}}" ({{messageLength}} chars) +``` + +# Tests +```test +{messageText: "fix bug", messageLength: 7} -> triggered +{messageText: "Refactor the authentication middleware to use JWT tokens", messageLength: 60} -> clean +{messageText: "", messageLength: 0} -> clean +``` +```` + +The `# Tests` block is optional but strongly encouraged — each line is a synthetic row evaluated +against the rule. Tests run as part of the standard `npm test` suite. + +## Anatomy of a metric + +Metrics are simpler than rules — they emit a single named value that rules and dashboards can +consume. A metric file lives at `src/core/metrics/.metric.md`. Below is +[`weekend-requests.metric.md`](../src/core/metrics/weekend-requests.metric.md): + +```markdown +--- +id: weekend-requests # Stable identifier; reference as metrics.weekendRequests +name: Weekend Requests +scope: requests # requests or sessions +version: 1 +tags: [wellbeing, time] +--- + +# Filter +dayOfWeek(timestamp) == 0 OR dayOfWeek(timestamp) == 6 # Predicate over the scope + +# Metric +ratio # Aggregation: count | ratio | sum | avg + +# Examples +{{messageText | truncate:60}} ({{dayOfWeek(timestamp)}}) +``` + +Once defined, a metric is automatically available in the Rule Playground metric list and can be +referenced from any rule's detection logic. + +## Local testing workflow + +Before opening a pull request: + +1. **Install and build** + ```bash + npm install + npm run build + ``` +2. **Run the rule and metric tests** — these load every markdown file under `src/core/rules/` and + `src/core/metrics/`, validate the frontmatter, and execute any `# Tests` blocks: + ```bash + npm test + ``` +3. **Live-test against your own data.** Launch the extension in the VS Code Extension Development + Host (`F5`), open the **Rule Editor**, find your new rule, and click **Test Rule**. Use the + threshold sliders to sweep values without re-editing the markdown. +4. **Prototype DSL expressions** in the **Rule Playground** before committing them — it shares the + same runtime as the rule engine, so anything that evaluates there will behave identically in a + rule. +5. **Lint** + ```bash + npm run lint + ``` + +## References + +- Built-in rules: [`src/core/rules/`](../src/core/rules/) — 45 worked examples across all categories +- Built-in metrics: [`src/core/metrics/`](../src/core/metrics/) +- A minimal rule with tests: [`src/core/rules/lazy-prompting.md`](../src/core/rules/lazy-prompting.md) +- A rule using a custom aggregator and `extra.*` fields: [`src/core/rules/yolo-mode.md`](../src/core/rules/yolo-mode.md) +- A minimal metric: [`src/core/metrics/weekend-requests.metric.md`](../src/core/metrics/weekend-requests.metric.md) +- Anti-Patterns overview: [content/improve/anti-patterns.md](content/improve/anti-patterns.md) +- Rule Editor guide: [content/improve/rule-editor.md](content/improve/rule-editor.md) +- Rule Playground guide: [content/improve/rule-playground.md](content/improve/rule-playground.md)