Skip to content

Commit 3562a94

Browse files
chore(test): install audit-harness v0.1.0 (P6 batch)
Per IS Testing SOP and VPS-as-the-home Priority 6 (OPS-z9b). Vendored install + CLAUDE.md § Testing baseline + 000-docs entry. Pattern matches the canonical pilot in jeremylongshore/ Hybrid-ai-stack-intent-solutions PR #4. Worktree-based install: main checkout had uncommitted work; install performed via git worktree at /tmp/ai-devops-intent-solutions-p6 to avoid disturbing operator's in-progress changes.Refs jeremylongshore/intentsolutions-vps-runbook#2 jeremylongshore.com made me do it -claude intentsolutions.io
1 parent 8ce3dbc commit 3562a94

13 files changed

Lines changed: 1297 additions & 0 deletions

.audit-harness/CHANGELOG.md

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
# Changelog
2+
3+
All notable changes to `@intentsolutions/audit-harness` are documented here.
4+
5+
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
6+
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
7+
8+
## [0.1.0] — 2026-04-21
9+
10+
Initial release. Extracted from the `audit-tests` Claude Code skill v7.0.0 to enable in-repo enforcement without global skill installation.
11+
12+
### Added
13+
14+
- `audit-harness verify` — SHA-256 hash verification for pinned policy files
15+
- `audit-harness init` — initialize/re-init the `.harness-hash` manifest
16+
- `audit-harness list` — list pinned files
17+
- `audit-harness escape-scan` — detect AI escape patterns in a diff (coverage threshold lowering, test deletion, architecture bypasses, test skip markers)
18+
- `audit-harness arch` — dispatch language-appropriate architecture checker (dependency-cruiser / import-linter / ArchUnit / deptrac / arch-go)
19+
- `audit-harness bias` — count common test-bias patterns
20+
- `audit-harness gherkin-lint` — advisory Gherkin quality check
21+
- `audit-harness crap` — CRAP (Complexity × Coverage) scorer for Python, JS/TS, Go, Rust
22+
23+
### Key design decisions
24+
25+
- **Scripts stay as shell/python.** Not a TypeScript port — battle-tested implementations, language-portable, minimal dependencies.
26+
- **Thin Node CLI.** `bin/audit-harness.js` is a dispatcher only; all logic lives in `scripts/`.
27+
- **Policy-driven thresholds.** `escape-scan.sh` reads floors from `tests/TESTING.md` in the target repo, not from the script source.
28+
- **Zero runtime dependencies** beyond Node 18+, bash, and Python 3 (only if using `crap` command).

.audit-harness/LICENSE

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
MIT License
2+
3+
Copyright (c) 2026 Jeremy Longshore / Intent Solutions
4+
5+
Permission is hereby granted, free of charge, to any person obtaining a copy
6+
of this software and associated documentation files (the "Software"), to deal
7+
in the Software without restriction, including without limitation the rights
8+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9+
copies of the Software, and to permit persons to whom the Software is
10+
furnished to do so, subject to the following conditions:
11+
12+
The above copyright notice and this permission notice shall be included in all
13+
copies or substantial portions of the Software.
14+
15+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21+
SOFTWARE.

.audit-harness/README.md

Lines changed: 135 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,135 @@
1+
# @intentsolutions/audit-harness
2+
3+
Deterministic test-enforcement toolkit. Companion to the `audit-tests` and `implement-tests` Claude Code skills — but usable standalone in any repo that wants hash-pinned, escape-scanned, AI-proof quality gates.
4+
5+
## What it is
6+
7+
A small CLI wrapping 6 deterministic scripts:
8+
9+
| Command | Purpose |
10+
|---|---|
11+
| `audit-harness verify` | Verify hash-pinned artifacts haven't changed since `--init` |
12+
| `audit-harness init` | Pin the current state of engineer-owned policy files |
13+
| `audit-harness list` | Show pinned files |
14+
| `audit-harness escape-scan --staged` | Detect AI attempts to lower test thresholds, delete tests, bypass architecture rules |
15+
| `audit-harness arch` | Run language-appropriate architecture-rule checker (dependency-cruiser / import-linter / ArchUnit / deptrac / arch-go) |
16+
| `audit-harness bias` | Count common test-bias patterns |
17+
| `audit-harness gherkin-lint` | Advisory Gherkin quality check |
18+
| `audit-harness crap` | CRAP (Complexity × Coverage) scorer — Python, Go, JS/TS, Rust |
19+
20+
## Install
21+
22+
```bash
23+
pnpm add -D @intentsolutions/audit-harness
24+
# or: npm install --save-dev @intentsolutions/audit-harness
25+
# or: yarn add --dev @intentsolutions/audit-harness
26+
```
27+
28+
## Quick usage
29+
30+
### Pre-commit hook (`.husky/pre-commit`)
31+
32+
```bash
33+
#!/usr/bin/env sh
34+
pnpm exec audit-harness escape-scan --staged
35+
pnpm exec audit-harness verify
36+
```
37+
38+
### CI workflow (`.github/workflows/ci.yml`)
39+
40+
```yaml
41+
containment:
42+
runs-on: ubuntu-latest
43+
steps:
44+
- uses: actions/checkout@v6
45+
- uses: pnpm/action-setup@v5
46+
- uses: actions/setup-node@v6
47+
with: { node-version: '20', cache: 'pnpm' }
48+
- run: pnpm install --frozen-lockfile
49+
- run: pnpm exec audit-harness verify
50+
- run: pnpm exec audit-harness escape-scan --range origin/main..HEAD
51+
```
52+
53+
### Engineer workflow — change a policy threshold
54+
55+
```bash
56+
# 1. Edit tests/TESTING.md to change coverage.line from 80 to 75
57+
# 2. Re-init to accept the change
58+
pnpm exec audit-harness init
59+
# 3. Commit the updated manifest alongside the policy change
60+
git add tests/TESTING.md .harness-hash
61+
git commit -m "chore(test): lower coverage floor to 75"
62+
```
63+
64+
## The containment model
65+
66+
The harness enforces this rule: **policy changes must be conscious, not silent.**
67+
68+
Engineer-owned files (`tests/TESTING.md`, `features/*.feature`, `.dependency-cruiser.cjs`, `stryker.conf.json`, etc.) are hashed into a manifest. Any diff that changes their content without a fresh `audit-harness init` is caught by pre-commit / CI and **REFUSED**.
69+
70+
AI agents remain useful (they can read policy, they can implement within constraints). What they can't do is silently weaken the constraints. That's the entire design.
71+
72+
See `audit-tests/references/philosophy.md` in the companion skill for the full rationale.
73+
74+
## The 7-layer testing taxonomy
75+
76+
This harness sits inside a larger framework:
77+
78+
```
79+
L7 Acceptance / RTM / Personas / Journeys ← WHAT are we proving?
80+
L6 E2E / BDD / Visual regression ← User-level guarantees
81+
L5 Perf / Security (SAST/DAST) / A11y / Chaos ← Non-functional
82+
L4 Integration / Contract / Migration ← Infrastructure wiring
83+
L3 Unit + Coverage + Mutation + Arch + CRAP ← Code-level correctness ← audit-harness lives here
84+
L2 Static analysis / Lint / Types / Secrets ← Read-only scanning
85+
L1 Git hooks / CI enforcement ← The cheapest gate ← audit-harness enables this
86+
```
87+
88+
The harness commands serve L1 (escape-scan in pre-commit + CI) and L3 (CRAP, architecture, bias, hash-pin).
89+
90+
## Exit codes
91+
92+
Important for CI scripting:
93+
94+
| Exit | Command | Meaning |
95+
|---|---|---|
96+
| 0 | any | Clean |
97+
| 1 | escape-scan | CHALLENGE — requires engineer-approved comment |
98+
| 2 | verify | `HARNESS_TAMPERED` — pinned file changed |
99+
| 2 | escape-scan | REFUSE — pipeline halted |
100+
| 3 | verify | No manifest (fresh repo, not an error) |
101+
102+
## Language support
103+
104+
Most scripts are language-agnostic (shell + regex). CRAP has per-language backends:
105+
106+
| Language | CRAP | Arch | Notes |
107+
|---|---|---|---|
108+
| Python | radon + coverage.py | import-linter | full support |
109+
| JS/TS | complexity-report + c8 | dependency-cruiser | full support |
110+
| Go | gocyclo + go test -cover | arch-go | full support |
111+
| Rust | rust-code-analysis + tarpaulin | (custom) | coverage integration pending |
112+
| Java/Kotlin | — | ArchUnit | via language-native tooling |
113+
| .NET | — | ArchUnitNET | via language-native tooling |
114+
| PHP | — | deptrac | via language-native tooling |
115+
116+
## License
117+
118+
MIT — see [LICENSE](./LICENSE).
119+
120+
## Related
121+
122+
- [`audit-tests` Claude Code skill](https://github.com/jeremylongshore/audit-harness#related) — diagnostic pipeline that uses this harness
123+
- [`implement-tests` Claude Code skill](https://github.com/jeremylongshore/audit-harness#related) — filesystem-mutating installer that installs this harness as part of L1/L3 setup
124+
125+
## Versioning
126+
127+
SemVer. Breaking changes to the CLI surface bump major; new commands bump minor; bug fixes bump patch.
128+
129+
## Contributing
130+
131+
This is infrastructure code. Changes need to be conservative. Before opening a PR:
132+
133+
1. Read `audit-tests/references/philosophy.md` (in the companion skill) to understand the escape-grammar design
134+
2. Run `bash scripts/escape-scan.sh --staged` on your own diff — yes, the harness tests itself
135+
3. Add test cases if you're adding a new pattern to escape-scan or a new command to the CLI

.audit-harness/VERSION

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
v0.1.0
Lines changed: 143 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,143 @@
1+
#!/usr/bin/env bash
2+
# arch-check.sh — Wall 7 architecture-constraint dispatcher.
3+
#
4+
# Detects the primary language of the repo, invokes the appropriate
5+
# dependency / architecture checker with the project's rule pack, and
6+
# normalizes the exit code.
7+
#
8+
# Exit codes:
9+
# 0 — all rules pass
10+
# 1 — rule violations detected
11+
# 2 — no tool installed / no config / unsupported language
12+
#
13+
# Usage:
14+
# bash arch-check.sh # run from repo root
15+
# bash arch-check.sh --json # emit JSON summary to stdout
16+
# bash arch-check.sh --help
17+
18+
set -euo pipefail
19+
20+
ROOT="${ROOT:-$(pwd)}"
21+
JSON_OUT=0
22+
REPORT_DIR="${ROOT}/reports/arch"
23+
24+
usage() {
25+
sed -n '2,20p' "$0"
26+
exit 0
27+
}
28+
29+
for arg in "$@"; do
30+
case "$arg" in
31+
--json) JSON_OUT=1 ;;
32+
--help|-h) usage ;;
33+
*) echo "arch-check: unknown flag $arg" >&2; exit 2 ;;
34+
esac
35+
done
36+
37+
mkdir -p "$REPORT_DIR"
38+
39+
emit_result() {
40+
local tool="$1" status="$2" violations="$3" log="$4"
41+
if [[ "$JSON_OUT" -eq 1 ]]; then
42+
printf '{"tool":"%s","status":"%s","violations":%s,"log":"%s"}\n' \
43+
"$tool" "$status" "$violations" "$log"
44+
else
45+
echo "arch-check: tool=$tool status=$status violations=$violations"
46+
echo " log=$log"
47+
fi
48+
}
49+
50+
# 1. dependency-cruiser (JS/TS)
51+
if [[ -f "${ROOT}/.dependency-cruiser.js" || -f "${ROOT}/.dependency-cruiser.cjs" ]]; then
52+
LOG="${REPORT_DIR}/dep-cruiser.log"
53+
if command -v npx >/dev/null 2>&1; then
54+
if npx --no-install dependency-cruiser --validate --output-type err "${ROOT}/src" > "$LOG" 2>&1; then
55+
emit_result dependency-cruiser pass 0 "$LOG"
56+
exit 0
57+
else
58+
VIOL=$(grep -c "error" "$LOG" || echo 0)
59+
emit_result dependency-cruiser fail "$VIOL" "$LOG"
60+
exit 1
61+
fi
62+
else
63+
emit_result dependency-cruiser missing-tool 0 "$LOG"
64+
exit 2
65+
fi
66+
fi
67+
68+
# 2. import-linter (Python)
69+
if [[ -f "${ROOT}/.importlinter" ]] || grep -q "^\[importlinter\]" "${ROOT}/pyproject.toml" 2>/dev/null; then
70+
LOG="${REPORT_DIR}/import-linter.log"
71+
if command -v lint-imports >/dev/null 2>&1; then
72+
if (cd "$ROOT" && lint-imports) > "$LOG" 2>&1; then
73+
emit_result import-linter pass 0 "$LOG"
74+
exit 0
75+
else
76+
VIOL=$(grep -c "BROKEN" "$LOG" || echo 0)
77+
emit_result import-linter fail "$VIOL" "$LOG"
78+
exit 1
79+
fi
80+
else
81+
emit_result import-linter missing-tool 0 "$LOG"
82+
exit 2
83+
fi
84+
fi
85+
86+
# 3. deptrac (PHP)
87+
if [[ -f "${ROOT}/deptrac.yaml" ]]; then
88+
LOG="${REPORT_DIR}/deptrac.log"
89+
if [[ -x "${ROOT}/vendor/bin/deptrac" ]]; then
90+
if (cd "$ROOT" && vendor/bin/deptrac analyse --no-progress) > "$LOG" 2>&1; then
91+
emit_result deptrac pass 0 "$LOG"
92+
exit 0
93+
else
94+
VIOL=$(grep -Ec "violation" "$LOG" || echo 0)
95+
emit_result deptrac fail "$VIOL" "$LOG"
96+
exit 1
97+
fi
98+
else
99+
emit_result deptrac missing-tool 0 "$LOG"
100+
exit 2
101+
fi
102+
fi
103+
104+
# 4. arch-go
105+
if [[ -f "${ROOT}/arch-go.yml" ]]; then
106+
LOG="${REPORT_DIR}/arch-go.log"
107+
if command -v arch-go >/dev/null 2>&1; then
108+
if (cd "$ROOT" && arch-go) > "$LOG" 2>&1; then
109+
emit_result arch-go pass 0 "$LOG"
110+
exit 0
111+
else
112+
VIOL=$(grep -c "Violation" "$LOG" || echo 0)
113+
emit_result arch-go fail "$VIOL" "$LOG"
114+
exit 1
115+
fi
116+
else
117+
emit_result arch-go missing-tool 0 "$LOG"
118+
exit 2
119+
fi
120+
fi
121+
122+
# 5. ArchUnit (Java/Kotlin) — run via build tool
123+
if [[ -f "${ROOT}/build.gradle" || -f "${ROOT}/build.gradle.kts" ]] && \
124+
grep -rq "com.tngtech.archunit" "${ROOT}" --include="*.gradle*" 2>/dev/null; then
125+
LOG="${REPORT_DIR}/archunit.log"
126+
if [[ -x "${ROOT}/gradlew" ]]; then
127+
if (cd "$ROOT" && ./gradlew test --tests '*ArchitectureTest*' --tests '*ArchTest*') > "$LOG" 2>&1; then
128+
emit_result archunit pass 0 "$LOG"
129+
exit 0
130+
else
131+
VIOL=$(grep -Ec "violated|FAILED" "$LOG" || echo 0)
132+
emit_result archunit fail "$VIOL" "$LOG"
133+
exit 1
134+
fi
135+
else
136+
emit_result archunit missing-tool 0 "$LOG"
137+
exit 2
138+
fi
139+
fi
140+
141+
# No tool / config found
142+
emit_result none not-configured 0 "$REPORT_DIR/none.log"
143+
exit 2

0 commit comments

Comments
 (0)