Skip to content

Add a test-author skill — operationalize the test-contract gate for matrix YAML + server_behavior #841

Description

@jonathaneoliver

Motivation

The test-contract gate now lives in characterization-principles.md + an always-on memory (state intent, quiz on blanks, echo resolved config before any run). A skill turns that convention into a walked procedure that carries the authoritative knob vocabulary, so the intent quiz is concrete and the echo-back is a filled spec — not prose. The config surface (tests/characterization/matrix/*.yaml — 20+ arms, namespaced is.*/proxy.* knobs; plus server_behavior params) churns constantly, which is exactly when a wrong assumption silently builds the wrong test.

Scope

New .claude/skills/test-author/SKILL.md covering two families:

  • char-matrix YAML (tests/characterization/matrix/*.yaml) — class config|fault; the full is.* (client, cold-relaunch) / proxy.* (server, config-on-connect) knob reference from internal/charmatrix/spec.go; axes: cartesian vs groups/compare A/B.
  • server-behavior (tests/server_behavior/server_*_test.go) — the baseline-plus-tolerance contract shape + sb_common_test.go helpers.

Procedure: classify family/class → contract quiz (behavior · platform · constant target · what varies · pass/fail DATA signal) → draft → echo resolved config → --dry-run verify, before any run.

Boundaries

  • Defers correctness rules to characterization-principles.md (links, doesn't restate).
  • Follows CONVENTIONS.md (no-guessing, pass-signal-is-data, Bash-first).
  • Out of scope: running tests (make characterize-* / harness char matrix), the unattended loop (sweep), analysis (forensics), and Roku.

Acceptance

  • Skill is invoked on "write/add a characterization test / new matrix / server-behavior case / A/B comparison" and NOT on run/analyse asks.
  • It produces a spec, echoes the resolved knobs, and ends with the exact --dry-run / -run verify command.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions