|
| 1 | +# @intentsolutions/audit-harness |
| 2 | + |
| 3 | +Deterministic test-enforcement toolkit. Companion to the `audit-tests` and `implement-tests` Claude Code skills — but usable standalone in any repo that wants hash-pinned, escape-scanned, AI-proof quality gates. |
| 4 | + |
| 5 | +## What it is |
| 6 | + |
| 7 | +A small CLI wrapping 6 deterministic scripts: |
| 8 | + |
| 9 | +| Command | Purpose | |
| 10 | +|---|---| |
| 11 | +| `audit-harness verify` | Verify hash-pinned artifacts haven't changed since `--init` | |
| 12 | +| `audit-harness init` | Pin the current state of engineer-owned policy files | |
| 13 | +| `audit-harness list` | Show pinned files | |
| 14 | +| `audit-harness escape-scan --staged` | Detect AI attempts to lower test thresholds, delete tests, bypass architecture rules | |
| 15 | +| `audit-harness arch` | Run language-appropriate architecture-rule checker (dependency-cruiser / import-linter / ArchUnit / deptrac / arch-go) | |
| 16 | +| `audit-harness bias` | Count common test-bias patterns | |
| 17 | +| `audit-harness gherkin-lint` | Advisory Gherkin quality check | |
| 18 | +| `audit-harness crap` | CRAP (Complexity × Coverage) scorer — Python, Go, JS/TS, Rust | |
| 19 | + |
| 20 | +## Install |
| 21 | + |
| 22 | +```bash |
| 23 | +pnpm add -D @intentsolutions/audit-harness |
| 24 | +# or: npm install --save-dev @intentsolutions/audit-harness |
| 25 | +# or: yarn add --dev @intentsolutions/audit-harness |
| 26 | +``` |
| 27 | + |
| 28 | +## Quick usage |
| 29 | + |
| 30 | +### Pre-commit hook (`.husky/pre-commit`) |
| 31 | + |
| 32 | +```bash |
| 33 | +#!/usr/bin/env sh |
| 34 | +pnpm exec audit-harness escape-scan --staged |
| 35 | +pnpm exec audit-harness verify |
| 36 | +``` |
| 37 | + |
| 38 | +### CI workflow (`.github/workflows/ci.yml`) |
| 39 | + |
| 40 | +```yaml |
| 41 | + containment: |
| 42 | + runs-on: ubuntu-latest |
| 43 | + steps: |
| 44 | + - uses: actions/checkout@v6 |
| 45 | + - uses: pnpm/action-setup@v5 |
| 46 | + - uses: actions/setup-node@v6 |
| 47 | + with: { node-version: '20', cache: 'pnpm' } |
| 48 | + - run: pnpm install --frozen-lockfile |
| 49 | + - run: pnpm exec audit-harness verify |
| 50 | + - run: pnpm exec audit-harness escape-scan --range origin/main..HEAD |
| 51 | +``` |
| 52 | +
|
| 53 | +### Engineer workflow — change a policy threshold |
| 54 | +
|
| 55 | +```bash |
| 56 | +# 1. Edit tests/TESTING.md to change coverage.line from 80 to 75 |
| 57 | +# 2. Re-init to accept the change |
| 58 | +pnpm exec audit-harness init |
| 59 | +# 3. Commit the updated manifest alongside the policy change |
| 60 | +git add tests/TESTING.md .harness-hash |
| 61 | +git commit -m "chore(test): lower coverage floor to 75" |
| 62 | +``` |
| 63 | +
|
| 64 | +## The containment model |
| 65 | +
|
| 66 | +The harness enforces this rule: **policy changes must be conscious, not silent.** |
| 67 | +
|
| 68 | +Engineer-owned files (`tests/TESTING.md`, `features/*.feature`, `.dependency-cruiser.cjs`, `stryker.conf.json`, etc.) are hashed into a manifest. Any diff that changes their content without a fresh `audit-harness init` is caught by pre-commit / CI and **REFUSED**. |
| 69 | + |
| 70 | +AI agents remain useful (they can read policy, they can implement within constraints). What they can't do is silently weaken the constraints. That's the entire design. |
| 71 | + |
| 72 | +See `audit-tests/references/philosophy.md` in the companion skill for the full rationale. |
| 73 | + |
| 74 | +## The 7-layer testing taxonomy |
| 75 | + |
| 76 | +This harness sits inside a larger framework: |
| 77 | + |
| 78 | +``` |
| 79 | +L7 Acceptance / RTM / Personas / Journeys ← WHAT are we proving? |
| 80 | +L6 E2E / BDD / Visual regression ← User-level guarantees |
| 81 | +L5 Perf / Security (SAST/DAST) / A11y / Chaos ← Non-functional |
| 82 | +L4 Integration / Contract / Migration ← Infrastructure wiring |
| 83 | +L3 Unit + Coverage + Mutation + Arch + CRAP ← Code-level correctness ← audit-harness lives here |
| 84 | +L2 Static analysis / Lint / Types / Secrets ← Read-only scanning |
| 85 | +L1 Git hooks / CI enforcement ← The cheapest gate ← audit-harness enables this |
| 86 | +``` |
| 87 | + |
| 88 | +The harness commands serve L1 (escape-scan in pre-commit + CI) and L3 (CRAP, architecture, bias, hash-pin). |
| 89 | + |
| 90 | +## Exit codes |
| 91 | + |
| 92 | +Important for CI scripting: |
| 93 | + |
| 94 | +| Exit | Command | Meaning | |
| 95 | +|---|---|---| |
| 96 | +| 0 | any | Clean | |
| 97 | +| 1 | escape-scan | CHALLENGE — requires engineer-approved comment | |
| 98 | +| 2 | verify | `HARNESS_TAMPERED` — pinned file changed | |
| 99 | +| 2 | escape-scan | REFUSE — pipeline halted | |
| 100 | +| 3 | verify | No manifest (fresh repo, not an error) | |
| 101 | + |
| 102 | +## Language support |
| 103 | + |
| 104 | +Most scripts are language-agnostic (shell + regex). CRAP has per-language backends: |
| 105 | + |
| 106 | +| Language | CRAP | Arch | Notes | |
| 107 | +|---|---|---|---| |
| 108 | +| Python | radon + coverage.py | import-linter | full support | |
| 109 | +| JS/TS | complexity-report + c8 | dependency-cruiser | full support | |
| 110 | +| Go | gocyclo + go test -cover | arch-go | full support | |
| 111 | +| Rust | rust-code-analysis + tarpaulin | (custom) | coverage integration pending | |
| 112 | +| Java/Kotlin | — | ArchUnit | via language-native tooling | |
| 113 | +| .NET | — | ArchUnitNET | via language-native tooling | |
| 114 | +| PHP | — | deptrac | via language-native tooling | |
| 115 | + |
| 116 | +## License |
| 117 | + |
| 118 | +MIT — see [LICENSE](./LICENSE). |
| 119 | + |
| 120 | +## Related |
| 121 | + |
| 122 | +- [`audit-tests` Claude Code skill](https://github.com/jeremylongshore/audit-harness#related) — diagnostic pipeline that uses this harness |
| 123 | +- [`implement-tests` Claude Code skill](https://github.com/jeremylongshore/audit-harness#related) — filesystem-mutating installer that installs this harness as part of L1/L3 setup |
| 124 | + |
| 125 | +## Versioning |
| 126 | + |
| 127 | +SemVer. Breaking changes to the CLI surface bump major; new commands bump minor; bug fixes bump patch. |
| 128 | + |
| 129 | +## Contributing |
| 130 | + |
| 131 | +This is infrastructure code. Changes need to be conservative. Before opening a PR: |
| 132 | + |
| 133 | +1. Read `audit-tests/references/philosophy.md` (in the companion skill) to understand the escape-grammar design |
| 134 | +2. Run `bash scripts/escape-scan.sh --staged` on your own diff — yes, the harness tests itself |
| 135 | +3. Add test cases if you're adding a new pattern to escape-scan or a new command to the CLI |
0 commit comments