|
| 1 | +# RealityCheck |
| 2 | + |
| 3 | +**Intent enforcement for agentic coding systems.** |
| 4 | + |
| 5 | +RealityCheck verifies whether an implementation faithfully realizes the declared intent of a system as expressed in a `SPEC.md` (contractual obligations) and a `PLAN.md` (declared execution steps). It answers one question: |
| 6 | + |
| 7 | +> *Did the code do what we said we would do, and only that?* |
| 8 | +
|
| 9 | +--- |
| 10 | + |
| 11 | +## Core Concepts |
| 12 | + |
| 13 | +| Term | Definition | |
| 14 | +|---|---| |
| 15 | +| **Drift** | Code behavior that exists without spec or plan authorization | |
| 16 | +| **Violation** | Code behavior that contradicts a declared constraint | |
| 17 | +| **Coverage** | Whether each spec/plan item is implemented in the code | |
| 18 | + |
| 19 | +If behavior exists without authorization, it is drift. |
| 20 | +If authorization exists without behavior, it is failure. |
| 21 | +If behavior contradicts authorization, it is violation. |
| 22 | + |
| 23 | +--- |
| 24 | + |
| 25 | +## Pipeline Position |
| 26 | + |
| 27 | +``` |
| 28 | +SPEC.md → SpecCritic → PLAN.md → PlanCritic → CODE → RealityCheck → Prism |
| 29 | +``` |
| 30 | + |
| 31 | +RealityCheck runs after planning and before code quality review. |
| 32 | + |
| 33 | +--- |
| 34 | + |
| 35 | +## Installation |
| 36 | + |
| 37 | +```bash |
| 38 | +go install github.com/dshills/realitycheck/cmd/realitycheck@latest |
| 39 | +``` |
| 40 | + |
| 41 | +Or build from source: |
| 42 | + |
| 43 | +```bash |
| 44 | +git clone https://github.com/dshills/realitycheck |
| 45 | +cd realitycheck |
| 46 | +go build ./cmd/realitycheck |
| 47 | +``` |
| 48 | + |
| 49 | +Requires `ANTHROPIC_API_KEY` to be set. |
| 50 | + |
| 51 | +--- |
| 52 | + |
| 53 | +## Usage |
| 54 | + |
| 55 | +```bash |
| 56 | +realitycheck check [path] [flags] |
| 57 | +``` |
| 58 | + |
| 59 | +### Required flags |
| 60 | + |
| 61 | +``` |
| 62 | +--spec <file> Path to SPEC.md |
| 63 | +--plan <file> Path to PLAN.md |
| 64 | +``` |
| 65 | + |
| 66 | +### Common flags |
| 67 | + |
| 68 | +``` |
| 69 | +--code-root <dir> Root directory to analyze (default: cwd) |
| 70 | +--format json|md Output format (default: json) |
| 71 | +--out <file> Write output to file instead of stdout |
| 72 | +--profile <name> Enforcement profile: general, strict-api, data-pipeline, library |
| 73 | +--strict No inferred intent; escalate drift severities |
| 74 | +--fail-on <verdict> Exit 2 if verdict >= level (ALIGNED|PARTIALLY_ALIGNED|DRIFT_DETECTED|VIOLATION) |
| 75 | +--severity-threshold <s> Filter output to findings at or above INFO|WARN|CRITICAL |
| 76 | +--model <id> Anthropic model ID (default: claude-opus-4-6) |
| 77 | +--offline Skip ANTHROPIC_API_KEY pre-flight check |
| 78 | +--verbose Print execution trace to stderr |
| 79 | +--debug Dump assembled prompt to stderr |
| 80 | +``` |
| 81 | + |
| 82 | +### Example |
| 83 | + |
| 84 | +```bash |
| 85 | +realitycheck check \ |
| 86 | + --spec specs/SPEC.md \ |
| 87 | + --plan specs/PLAN.md \ |
| 88 | + --code-root . \ |
| 89 | + --format md \ |
| 90 | + --fail-on DRIFT_DETECTED |
| 91 | +``` |
| 92 | + |
| 93 | +--- |
| 94 | + |
| 95 | +## Output |
| 96 | + |
| 97 | +### Verdicts |
| 98 | + |
| 99 | +| Verdict | Meaning | |
| 100 | +|---|---| |
| 101 | +| `ALIGNED` | Code matches spec and plan | |
| 102 | +| `PARTIALLY_ALIGNED` | Gaps or incomplete implementation | |
| 103 | +| `DRIFT_DETECTED` | Unauthorized behavior present | |
| 104 | +| `VIOLATION` | Code contradicts a declared constraint | |
| 105 | + |
| 106 | +### Scoring |
| 107 | + |
| 108 | +Score starts at 100 and decreases deterministically: |
| 109 | + |
| 110 | +- **−20** per CRITICAL finding |
| 111 | +- **−7** per WARN finding |
| 112 | +- **−2** per INFO finding |
| 113 | +- Clamped to `[0, 100]` |
| 114 | + |
| 115 | +Scoring is always computed locally — never by the LLM. |
| 116 | + |
| 117 | +### Exit codes |
| 118 | + |
| 119 | +| Code | Meaning | |
| 120 | +|---|---| |
| 121 | +| `0` | Success | |
| 122 | +| `2` | `--fail-on` threshold met | |
| 123 | +| `3` | Input error (missing flags, file not found) | |
| 124 | +| `4` | LLM / provider error | |
| 125 | +| `5` | LLM produced unrecoverable invalid output | |
| 126 | + |
| 127 | +### JSON output (excerpt) |
| 128 | + |
| 129 | +```json |
| 130 | +{ |
| 131 | + "tool": "realitycheck", |
| 132 | + "version": "0.1.0", |
| 133 | + "summary": { |
| 134 | + "verdict": "DRIFT_DETECTED", |
| 135 | + "score": 80, |
| 136 | + "critical_count": 0, |
| 137 | + "warn_count": 1, |
| 138 | + "info_count": 0 |
| 139 | + }, |
| 140 | + "drift": [ |
| 141 | + { |
| 142 | + "id": "DRIFT-001", |
| 143 | + "severity": "WARN", |
| 144 | + "description": "Undocumented retry loop in HTTP client", |
| 145 | + "evidence": [{ "path": "internal/client/client.go", "symbol": "retryRequest" }], |
| 146 | + "why_unjustified": "No spec or plan item authorizes automatic retries.", |
| 147 | + "recommendation": "Add to spec or remove." |
| 148 | + } |
| 149 | + ] |
| 150 | +} |
| 151 | +``` |
| 152 | + |
| 153 | +--- |
| 154 | + |
| 155 | +## Profiles |
| 156 | + |
| 157 | +Profiles modulate how the LLM interprets the spec and plan. |
| 158 | + |
| 159 | +| Profile | Description | |
| 160 | +|---|---| |
| 161 | +| `general` | Default balanced analysis | |
| 162 | +| `strict-api` | Any undeclared HTTP handler or outbound call is CRITICAL drift | |
| 163 | +| `data-pipeline` | Any undeclared write to an external store is CRITICAL drift | |
| 164 | +| `library` | Drift evaluated only on exported symbols | |
| 165 | + |
| 166 | +--- |
| 167 | + |
| 168 | +## Strict Mode |
| 169 | + |
| 170 | +`--strict` enables adversarial analysis: |
| 171 | + |
| 172 | +- Unclear coverage → `NOT_IMPLEMENTED` |
| 173 | +- Missing evidence → absent |
| 174 | +- WARN drift → CRITICAL, INFO drift → WARN |
| 175 | + |
| 176 | +--- |
| 177 | + |
| 178 | +## Architecture |
| 179 | + |
| 180 | +``` |
| 181 | +cmd/realitycheck/ CLI entry point (cobra) |
| 182 | +internal/schema/ Canonical data types |
| 183 | +internal/spec/ SPEC.md parser |
| 184 | +internal/plan/ PLAN.md parser |
| 185 | +internal/codeindex/ Code inventory (symbols, tests, manifests) |
| 186 | +internal/profile/ Enforcement profiles |
| 187 | +internal/llm/ LLM provider, prompt builder, response validator |
| 188 | +internal/coverage/ Coverage analysis helpers |
| 189 | +internal/drift/ Drift severity helpers |
| 190 | +internal/verdict/ Scoring and verdict logic |
| 191 | +internal/render/ JSON and Markdown renderers |
| 192 | +``` |
| 193 | + |
| 194 | +Symbol extraction is regex-based (no full AST). Supported languages: Go, JavaScript/TypeScript, Python, Rust. |
| 195 | + |
| 196 | +--- |
| 197 | + |
| 198 | +## Development |
| 199 | + |
| 200 | +```bash |
| 201 | +# Run all tests |
| 202 | +go test ./... |
| 203 | + |
| 204 | +# Run with race detector |
| 205 | +go test -race ./... |
| 206 | + |
| 207 | +# Run integration tests (uses mock LLM, no API key required) |
| 208 | +go test -race -tags=integration ./... |
| 209 | + |
| 210 | +# Build binary |
| 211 | +go build ./cmd/realitycheck |
| 212 | + |
| 213 | +# Lint |
| 214 | +go vet ./... |
| 215 | +``` |
| 216 | + |
| 217 | +--- |
| 218 | + |
| 219 | +## Security & Privacy |
| 220 | + |
| 221 | +- No telemetry emitted by default |
| 222 | +- Raw code is **never** sent to the LLM — only file paths, symbol names, and dependency manifest text |
| 223 | +- `--debug` prints the assembled prompt to stderr (no redaction needed since code content is absent) |
0 commit comments