Conformance sweep A3: prose tokenizer + subset/ref linters (warn-only)

Summary

Author the foundational pre-commit linter trio: a shared tokenizer module (_prose_tokens.py) plus the two consumer linters (validate_yaml_prose_subset.py for ADR-017's authoring subset; validate_prose_references.py for ADR-016's sentinel resolution + raw-<a> block + bare-camelCase block).

All three ship in warn-only mode and flip to block in the sweep-closing commit (C2). Stacks off A1 (validate_prose_references.py reads externalReferences[].id from the consumer schemas, which need the shared external-references.schema.json $ref to be authored — though the linter can read directly from the YAML rather than via schema indirection if the shared schema isn't yet $ref'd into consumers).

Parent / context

• Parent: [Epic]: Conformance-sweep plan for the YAML-content architectural ADRs (014–022) #237 (conformance-sweep epic)
• Implements:
  • ADR-017 D1 (allowed authoring tokens), D2 (disallowed-by-construction), D4 (lint design), D5 (shared tokenizer module at scripts/hooks/precommit/_prose_tokens.py)
  • ADR-016 D2 (sentinel grammar), D6 (linter), D8 (per-rule machine-enforcement table)
  • ADR-013 — pre-commit hook authoring pattern
• Branch target: main
• Base for the implementation PR: stacks off feature/A1-shared-schema (A1 = PR Conformance sweep A1: Implementation of external-references schema + framework patterns #243, currently OPEN) while A1 is in flight; rebases onto main once A1 merges, then this PR opens against main.

Tasks bundled

2.3.1 — Author shared tokenizer _prose_tokens.py

New module at scripts/hooks/precommit/_prose_tokens.py. Vanilla Python stdlib + pyyaml (already in requirements.txt); no new dependencies.

Exports: tokenize(text: str) -> list[Token] where Token is a dataclass or NamedTuple with kind (one of BOLD, ITALIC, SENTINEL_INTRA, SENTINEL_REF, TEXT, INVALID_HTML, INVALID_URL, INVALID_CAMELCASE_ID, etc.) and value (the raw string). The tokenizer is the single source of truth for the prose grammar (per ADR-017 D5); both consumer linters import it.

Recognized tokens (per ADR-017 D1):

• **bold** — asterisk delimiter; one nesting level inside (italic OK; another **bold** rejected)
• *italic* and _italic_ — both delimiters; sentinels do not nest into them
• {{idXxx}} — intra-document sentinel; identifier matches [A-Za-z][A-Za-z0-9]* and starts with a known prefix (risk, control, component, persona — the four entity-id prefixes from ADR-016 D2)
• {{ref:identifier}} — external-reference sentinel; identifier matches [A-Za-z0-9_-]+
• Plain text (TEXT)

Recognized rejection productions (per ADR-017 D2):

• Inline URLs in any form (raw http://, raw https://, [text](url) markdown link, ] followed by ()
• Raw HTML tags (any < followed by an alphabetic character or /)
• Markdown headings (#, ##, etc. at line start)
• Markdown list markers (- , * , 1. at line start) — including the folded-bullet drift heuristic from ADR-020 D4 (detects \s-\s patterns inside folded-scalar strings; ships warn-only initially, flips to block in C2)
• Code blocks (fenced ``` or indented), inline code (`code`)
• Images (![alt](url))
• Blockquotes (> at line start)
• Markdown pipe tables
• Bare camelCase identifiers matching (risk|control|component|persona)[A-Z]… outside a sentinel (caught here for validate_prose_references.py to consume; raw production output)

2.3.2 — Author validate_yaml_prose_subset.py

Per ADR-017 D4. New pre-commit hook wrapper at scripts/hooks/precommit/validate_yaml_prose_subset.py.

• Walks every prose field in risk-map/yaml/{components,controls,risks,personas}.yaml
• Reads prose fields by introspecting schemas (not hardcoded) so it tracks schema changes
• Imports the shared tokenizer (_prose_tokens.py)
• Accepts: **bold** (one nesting level), *italic*, _italic_, {{idXxx}} and {{ref:identifier}} (presence; resolution is validate_prose_references.py's job)
• Rejects: inline URL forms, raw HTML tags, markdown headings/lists/code/images/blockquotes/tables, plus the folded-bullet drift heuristic (ADR-020 D4)
• Mode: warn-only initially; flag/env-var to flip to block in C2
• Rejection format: validate-yaml-prose-subset: <file>:<entry-id>:<field>[<index>]: <reason> at <token-snippet> per ADR-017 D4

2.3.3 — Author validate_prose_references.py

Per ADR-016 D6. New pre-commit hook wrapper at scripts/hooks/precommit/validate_prose_references.py.

• Walks the same prose fields as validate_yaml_prose_subset.py
• Imports the shared tokenizer
• For each {{idXxx}} sentinel: reads identifier enums from risks/controls/components/personas.schema.json and asserts the ID resolves
• For each {{ref:identifier}} sentinel: reads the entry's own externalReferences[].id array from the YAML and asserts the identifier resolves
• For inline URLs (raw or [text](url)): rejects unconditionally
• For raw HTML tags: rejects (overlap with subset linter is acceptable; whichever message is more specific wins)
• For bare camelCase IDs (matching (risk|control|component|persona)[A-Z]\w* outside a sentinel): rejects
• Mode: warn-only initially; flag/env-var to flip to block in C2
• Rejection format: validate-prose-references: <file>:<entry-id>:<field>[<index>]: <reason>

2.3.11 partial — Pre-commit-config slot

Add the two new hooks to .pre-commit-config.yaml per ADR-013's pattern:

- id: validate-yaml-prose-subset
  name: validate: YAML prose authoring subset
  entry: python3 scripts/hooks/precommit/validate_yaml_prose_subset.py
  language: system
  files: ^risk-map/yaml/(components|controls|risks|personas)\.yaml$
  pass_filenames: true

- id: validate-prose-references
  name: validate: YAML prose references (sentinels + IDs)
  entry: python3 scripts/hooks/precommit/validate_prose_references.py
  language: system
  files: ^risk-map/yaml/(components|controls|risks|personas)\.yaml$
  pass_filenames: true

The validate-identification-questions hook from A5 lands separately.

Acceptance criteria

• scripts/hooks/precommit/_prose_tokens.py exists; exports tokenize(text) -> list[Token]; vanilla Python; no new deps
• scripts/hooks/precommit/validate_yaml_prose_subset.py exists; runs cleanly on current YAML in warn-only mode (warnings expected for all the to-be-migrated content — that's the point); block-mode flag wired but not active
• scripts/hooks/precommit/validate_prose_references.py exists; runs cleanly on current YAML in warn-only mode (warns on bare-ID mentions, inline URLs, raw <a> tags); block-mode flag wired
• Test coverage: shared tokenizer fixtures at scripts/hooks/tests/fixtures/prose_subset/ (per ADR-017 D5) — both linters read these fixtures so they cannot disagree on what a token is
• Both linter wrappers have their own test files at scripts/hooks/tests/test_validate_yaml_prose_subset.py and test_validate_prose_references.py
• .pre-commit-config.yaml slots for both hooks added; pre-commit run --all-files succeeds (warn-only output for current corpus violations)
• Bare-camelCase regex must distinguish "id appears as YAML field value" (legit) from "id appears in prose string" (block) — the tokenizer is context-aware; explicit fixture cases prove it. This is the highest-risk class for false positives — reviewer should explicitly verify both fixture cases pass.
• No new dependencies; uses stdlib + pyyaml

Out of scope

• Block-mode flip — that's C2 (sweep-close). This sub-PR ships warn-only.
• Identification-questions lint — A5's deliverable; uses the same pattern but a separate rule set.
• YAML content migration — the content-migration sub-PRs (B1, B2). Linters serve as the warn-mode authoring signal during the migration.
• Schema changes — A2's territory.
• validate_all_schemas.py extensions — A1 territory (already done; no changes needed here).

Dependencies

• A1 (PR Conformance sweep A1: Implementation of external-references schema + framework patterns #243) — validate_prose_references.py reads externalReferences[].id; the shared schema's existence on main is what makes this lookup well-defined for the published linter. Local development can stack off feature/A1-shared-schema while Conformance sweep A1: Implementation of external-references schema + framework patterns #243 is in review.

Closes via

PR #NNN (TBD)