template-manifest: single source of truth for template-owned files - #60
Merged
Conversation
…fixture
Single canonical enumeration of every file the template owns. To be
sourced by update-from-template.sh, check-template-version.sh, and
adopt.sh in subsequent commits; for now the file exists in isolation
and no consumer references it, so CI behaviour is unchanged.
Six arrays:
- TEMPLATE_SHARED_INFRA (32 files synced regardless of overlay)
- TEMPLATE_OVERLAY_CLAUDE (13 claude-code overlay files)
- TEMPLATE_OVERLAY_CURSOR (6 cursor overlay files)
- TEMPLATE_SUBSTITUTE_FILES (10 files needing {{REPO_NAME}} substitution)
- TEMPLATE_HOST_OWNED (CLAUDE.md, .gitignore, .claude/settings.json)
- TEMPLATE_ONE_SHOT (documentation only; never iterated)
Five accessors:
- lw_manifest_needs_substitution
- lw_manifest_assemble_active_files (agent mode + detection mode)
- lw_manifest_known_grant_type / _sentinel / _payload
Deliberate decisions documented inline:
- .gitignore is HOST_OWNED (not SHARED_INFRA) so update never overwrites
it; adopt's append-only grant maintains the sentinel block.
- TEMPLATE_SUBSTITUTE_FILES holds only the 10 paths whose template copy
actually contains {{REPO_NAME}}; the unit fixture verifies this and
catches the inverse drift (adding a file that has no marker).
- posttooluse-hook.sh and ensure-wiki.py are excluded from
SUBSTITUTE_FILES with reasons; tracked as separate issues.
scripts/test/tests/unit/manifest-shape/ enforces the cross-array
invariants:
- All arrays non-empty.
- HOST_OWNED entries have well-formed grants with known op types.
- HOST_OWNED is disjoint from every sync array.
- SUBSTITUTE_FILES is a subset of the sync arrays (no orphans).
- Every SUBSTITUTE entry actually contains {{REPO_NAME}} in the
template tree.
- Every manifest path exists in the template tree (catches the drift
that bit PR #51 twice).
- assemble_active_files counts match shared/overlay sums in all six
modes (agent={claude-code,cursor,none} x detection={claude,cursor,none}).
Mutation-tested locally: forcing a bogus op type, an orphan substitute,
an overlap, or a missing-tree path each produces exactly the expected
fail.
Refs: roadmap chunk 1 of 6.
scripts/update-from-template.sh now sources scripts/lib/template-manifest.sh and consumes the active file set via lw_manifest_assemble_active_files (detection mode: empty agent arg, repo_root populated). The five inline arrays (ALWAYS_FILES, ONE_SHOT_FILES, CLAUDE_FILES, CURSOR_FILES, SUBSTITUTE_FILES) and the local needs_substitution() function are deleted; the substitution check now calls lw_manifest_needs_substitution. Two behaviour changes follow from the manifest's canonical content: 1. .gitignore is host-owned (TEMPLATE_HOST_OWNED), so update no longer syncs it. A post-sync advisory flags when the host's .gitignore differs from the template's so back-porting is opt-in. Adopt's append-only grant continues to maintain the wiki sub-repo rule. 2. Two overlay templates that were in ADD_ALLOWLIST but missing from CLAUDE_FILES (posttooluse-hook.sh added in PR #51, ensure-wiki.py added in PR #50) now reach existing projects on the next update. Category B adopted projects gain these on their first post-refactor update; this is the drift the manifest exists to close, and it is the explicit goal. Net effect on the assembled file list (template detected as both .claude/ and .cursor/ host): old 33 ALWAYS + 11 CLAUDE + 6 CURSOR = 50; new 32 SHARED + 13 OVERLAY_CLAUDE + 6 OVERLAY_CURSOR = 51. Doc-block at the head of the script trimmed: the old per-file enumeration is replaced with a pointer to the manifest, and the TEMPLATE_HOST_OWNED rationale for .gitignore is documented inline. Smoke fixture template-bootstrap: the four parallel-pair grep assertions that asked both sync scripts to list wiki-write-protocol paths are replaced by two grep assertions on the manifest. By-construction guarantee from a shared source is a stronger contract than count- agnostic grep on each script: a path listed once in the manifest cannot drift between consumers. Mutation-tested: removing the manifest file makes the script exit 1 with a loud "No such file" error. Refs: roadmap chunk 2 of 6.
scripts/check-template-version.sh now sources scripts/lib/template- manifest.sh and assembles the active file list via lw_manifest_assemble_active_files (detection mode). The five inline arrays and the local needs_substitution() function are deleted, matching the migration in update-from-template. HAS_CLAUDE and HAS_CURSOR remain locally computed from filesystem checks; they appear in the summary block and are independent of the manifest accessor's internal decision (the two are intentionally kept in lockstep by mirroring the same conditions). The .gitignore advisory mirrors update-from-template's: drift is surfaced for review, never marked as out-of-date, since .gitignore is host-owned (TEMPLATE_HOST_OWNED) and the host owns when to back-port. Net diff: -74 lines (arrays + helper) / +30 lines (manifest source + advisory). Behaviour unchanged for any existing project except that the same +2 overlay templates (posttooluse-hook.sh, ensure-wiki.py) now appear in the drift report. That mirrors update-from-template's delta from the same manifest. Mutation-tested: removing the manifest file makes the script exit 1 with a loud "No such file" error. Refs: roadmap chunk 3 of 6.
scripts/adopt.sh now sources scripts/lib/template-manifest.sh and:
- Assembles the ADD set via lw_manifest_assemble_active_files in agent
mode (empty repo_root, AGENT drives overlay inclusion). Replaces the
inline ADD_ALLOWLIST array (28 entries) with the manifest's
SHARED_INFRA + agent-selected overlay union.
- Iterates TEMPLATE_HOST_OWNED for the default-grants branch, replacing
the inline DEFAULT_GRANTS array.
- Calls lw_manifest_known_grant_{type,sentinel,payload} at the four
classification + apply sites, replacing the three local known_grant_*
helpers (each was 5-20 lines of case lookup).
Behaviour delta vs the previous adopt.sh:
agent=claude-code: ADD count grows from 28 to 45 (+17 files). The new
entries close the drift that PR #50 and PR #51 patched manually:
wiki/agents/README.md
wiki/agents/wiki-write-protocol.md
wiki/agents/claude-code/README.md
features/README.md
scripts/wiki-write-protocol/{README.md,protocol.sh,sandbox.sh,run-all.sh}
scripts/wiki-write-protocol/scenarios/01..09/run.sh (9 scenarios)
These have always been in update-from-template's ALWAYS_FILES; they
just never reached projects that adopted (rather than instantiated).
Adopters from now on get them on first adopt; existing adopters get
them on their first post-refactor update-from-template run (the
Category B migration path documented in the roadmap).
agent=none: ADD count = 32 (TEMPLATE_SHARED_INFRA only). The plus-13
overlay files are correctly suppressed.
All 430 adopt integration tests pass unchanged. The plan predicted
this: the assertions check file presence + stderr/manifest patterns,
none of them assert exact ADD counts.
Mutation-tested: deleting the manifest file makes adopt print a loud
"No such file" + "command not found" on stderr and classify 0 files
in the dry-run, which a reviewer cannot miss. The script does not
abort because adopt deliberately uses 'set -uo pipefail' (no -e) so
that the dry-run report still surfaces partial classifications.
Net diff: -122 lines (inline arrays + three helpers) / +37 lines
(source line, while-read accessor loop, doc-block).
Refs: roadmap chunk 4 of 6.
scripts/test/tests/smoke/manifest-convergence/ is the load-bearing test for the consolidation: it stages two sandboxes that ought to produce the same on-disk file set if scripts/lib/template-manifest.sh is the single source of truth. Sandbox A: clone_template (forced to local-clone mode via MVP_TEMPLATE_LOCAL=$TEMPLATE_ROOT, so the source matches the branch under test), then instantiate.sh --agent=none. Produces the file tree that instantiate-driven projects ship with. Sandbox B: virgin git repo with fake-github origin, then adopt.sh --apply --agent=claude-code. Produces the file tree that adopt-driven projects ship with. Assertions: 1. For every path in expected-claude.txt (manifest assembler output in agent mode), B has the file on disk. Catches the drift PR #50 and PR #51 patched manually: a path listed in the manifest that adopt fails to install. 2. For every path in TEMPLATE_SHARED_INFRA that exists in both A and B (i.e. SHARED files, allowlisted asymmetric paths excluded), cmp -s reports byte-equality. Catches the inverse: adopt installs a different version than instantiate. 3. B contains no files outside the manifest's enumerated set, modulo a tight allowlist of runtime artifacts (.git/, wiki/<name>.wiki/, wiki/WIKI-INDEX.md from init-wiki, host-authored README.md and HOST_OWNED targets, hook scripts installed by setup.sh --hook). 4. At least 20 SHARED_INFRA paths must be reported as byte-equal, not just "no failures". Catches the inverse failure mode where the allowlist accidentally eats every path and the test silently verifies nothing. Mutation-tested with the discipline-of-failure protocol: - Adding a fake path to TEMPLATE_SHARED_INFRA produces one specific failure: "B has manifest-listed path: fake/nonexistent.md". - Making adopt write divergent content for wiki/agents/README.md produces "A and B byte-equal on SHARED_INFRA path: wiki/agents/ README.md" exactly. Other paths stay green. Note on update-from-template: the test does NOT stage a third sandbox running adopt + update because the update path requires a clonable remote (network or a bare mirror) and the hermetic harness has no clean way to stage one. The adopt-vs-instantiate comparison alone proves the manifest is canonical for the two divergent code paths; the update path reuses the same assembler in detection mode and is verified by the unit fixture (scripts/test/tests/unit/manifest-shape/) plus the existing update-from-template behaviour checks. Refs: roadmap chunk 5 of 6.
The Contributing back section now closes with a one-line maintainer note: to add a template-synced file, edit scripts/lib/template- manifest.sh and nothing else. References the unit and smoke fixtures that enforce the contract. Refs: feature/template-manifest-consolidation, chunk 6 of 6.
This was referenced Jun 28, 2026
psaboia
marked this pull request as draft
June 28, 2026 15:31
psaboia
marked this pull request as ready for review
June 28, 2026 17:24
psaboia
added a commit
that referenced
this pull request
Jul 2, 2026
…clines Found while verifying the branch has no derived-project side effects: this harness ships to derived projects via "Use this template", and there manifest-convergence's patch declines to stage (clone_template's issue-#15 guard refuses a derived TEMPLATE_ROOT, correctly) — but the four staging assertions ran anyway and failed. Observed in a real derived simulation: 4 spurious FAILs; reproduced identically with origin/main's harness under the shipped pinned-MVP_TEMPLATE_LOCAL CI condition, so this predates the default-source work (derived CI has been red on this test since the fixture landed in PR #60). Fix: guard on the staged sandbox like the five sibling smoke tests — [ ! -d "$A" ] means the patch declined (it exits 0 without staging), so skip with the reason instead of asserting staging that can never hold in a derived checkout. Observed contrasts: derived simulation goes 4 fail -> 0 fail / 6 skip under BOTH CI conditions (bare default-source and pinned var); the template repo still exercises the test for real (manifest-convergence alone: 53 pass, 0 fail, 0 SKIP); full suite 949 pass, 0 fail.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Consolidate the three parallel inline arrays (
ALWAYS_FILESinupdate-from-template.sh, the duplicated copy incheck-template-version.sh, andADD_ALLOWLISTinadopt.sh) into a single canonical manifest atscripts/lib/template-manifest.sh. All three scripts now source the manifest and consume the same accessors. Adding a new template-synced file becomes one edit in one place; the drift that PR #50 and PR #51 each had to patch manually is now structurally impossible.Closes the "Drift between
ADD_ALLOWLISTandALWAYS_FILES" Open Question on the Adopt-Existing-Repo-Design wiki page.What the manifest contains
Six arrays, every one bash-3.2 portable (no associative arrays):
TEMPLATE_SHARED_INFRA(32 paths): synced regardless of overlay.TEMPLATE_OVERLAY_CLAUDE(13 paths): claude-code overlay files.TEMPLATE_OVERLAY_CURSOR(6 paths): cursor overlay files.TEMPLATE_SUBSTITUTE_FILES(10 paths): need{{REPO_NAME}}substitution at copy time.TEMPLATE_HOST_OWNED(3 entries:path|op-type):CLAUDE.md|managed-block,.gitignore|append-only,.claude/settings.json|merge.TEMPLATE_ONE_SHOT(5 paths): documentation only; never iterated.Five accessors:
lw_manifest_needs_substitution,lw_manifest_assemble_active_files(agent mode + detection mode),lw_manifest_known_grant_{type,sentinel,payload}.Behaviour delta
1.
.gitignorebecomes host-ownedWas in
ALWAYS_FILES(updateoverwrote it when the host's matched the template's hash). Now inTEMPLATE_HOST_OWNED|append-only(updatenever touches it;adopt's append-only grant still maintains the wiki sub-repo rule).update-from-template.shandcheck-template-version.shprint a one-line advisory when the host's.gitignorediffers from the template's, so back-porting new rules is opt-in.2. Adopt installs +17 files that were drifting
The previous
ADD_ALLOWLISThad 28 entries;ALWAYS_FILEShad 33. The 17 paths that were inALWAYS_FILESbut missing fromADD_ALLOWLISTnow reach adopters:Counts:
adopt --agent=claude-codeADD grows from 28 to 45.adopt --agent=noneADD = 32 (SHARED only).3. Two formerly-orphaned overlay templates now reach derived projects
wiki/agents/claude-code/templates/posttooluse-hook.sh(added in PR #51) andensure-wiki.py(added in PR #50) were inADD_ALLOWLISTbut not inCLAUDE_FILES. Both are now inTEMPLATE_OVERLAY_CLAUDE, soupdate-from-template.shsyncs them too.Migration story for existing projects
Category A — instantiated before this refactor (e.g. team-ai-Engineering, p28-behavioral-test). Next
update-from-template.sh: identical behaviour, plus a one-line.gitignoreadvisory if the host's differs from template's. No action required.Category B — adopted between PR #51 and this refactor. Next
update-from-template.sh: the report lists 17 new files inChanged:. Review the dry-run, accept the additions, commit. Their.gitignoreandCLAUDE.mdare not touched (adopt installed them with sentinel blocks; update doesn't own them).Category C — never updated since adoption/instantiation. Same as A or B on first update.
No regression possible: every file in any existing host's current set remains in the new manifest. The union grows; it does not shrink.
Operational implications
Combined with PR #55 (default grants) and PR #59 (
--github-wikiflag), adopt is now sufficient as a single-command bootstrap. The recipe simplifies:.llm-wiki-adopt-grants.ymlfirstadopt --apply --github-wiki(PR #59)update-from-template.shafter adopt to install the missing filesALWAYS_FILESbut notADD_ALLOWLIST)ALWAYS_FILESANDADD_ALLOWLISTscripts/lib/template-manifest.shupdate-from-template.shbecomes what its name promised: pure incremental sync of template improvements. The bootstrap split between adopt and update is gone.The minimum onboarding recipe simplifies to:
Hosts adopted between PR #51 and this merge will see ~17 files listed as
Changed:in their first post-mergeupdate-from-template.shdry-run (wiki-write-protocol scripts plus a few agent docs). The list is strictly additive; theirCLAUDE.md,.gitignore, and.claude/settings.jsonare not touched.Test plan
scripts/test/tests/unit/manifest-shape/(99 assertions): every array non-empty,HOST_OWNEDdisjoint from sync arrays, everySUBSTITUTEentry actually contains{{REPO_NAME}}in the template tree, every manifest path exists on disk, all six accessor modes return the expected file counts.scripts/test/tests/smoke/manifest-convergence/(51 assertions): stages two sandboxes (instantiate.sh --agent=nonevsadopt.sh --apply --agent=claude-code) and asserts byte-equality onTEMPLATE_SHARED_INFRA. Mutation-tested: forcing adopt to write divergent content for one file fails exactly that path's assertion.manifest-shape: bogus op type → 1 specific FAIL; fake manifest path → 1 specific FAIL; substitute entry without marker → 1 specific FAIL. Each invariant discriminates.template-manifest.shmakesupdate-from-template.shexit 1 with loud "No such file" andadopt.shprint "command not found" + classify 0 files.--github-wiki):adopt --applyinstalled all expected files, sentinel blocks injected into CLAUDE.md / .gitignore / .claude/settings.json, wiki sub-repo materialized locally, Claude Code session honored the/wiki-experimentoverlay path end-to-end (created the page, updated index/Home/log, passed the Verification Gate, two-commit-per-log-entry rule respected).Maintainer note
README.mdContributing back section now closes with: to add a template-synced file, editscripts/lib/template-manifest.shand nothing else. The two test fixtures enforce the contract.Deliberate exclusions
TEMPLATE_SUBSTITUTE_FILESdoes NOT include three claude-code templates that lack{{REPO_NAME}}markers in their template copy (claude-md-snippet.md,session-start-hook.sh,memory-seed.md); listing them would lie about the contract.posttooluse-hook.shandensure-wiki.pyare excluded with the comment pointing to a follow-up issue: the${REPO_NAME}reference inposttooluse-hook.shis inside a single-quoted heredoc thatsetup.shships verbatim, so substituting it here would not reach the installed hook. To be addressed in a focused follow-up.Commits
update-from-template.shsources manifest, drops 5 inline arrays, .gitignore advisorycheck-template-version.shsame migration, preserves HAS_CLAUDE/HAS_CURSOR for the reportadopt.shsources manifest, drops ADD_ALLOWLIST + DEFAULT_GRANTS + three known_grant_* helpersWiki commit:
0379715oncrcresearch/llm-wiki-memory-template.wiki— resolves the Open Question onAdopt-Existing-Repo-Designwith a resolution note pointing at the manifest and the two fixtures, plus log entry.Out of scope (follow-ups)
posttooluse-hook.sh:${REPO_NAME}literal in a single-quoted heredoc thatsetup.shships verbatim, so substitution at sync time would not reach the installed hook. Tracked in posttooluse-hook.sh: ${REPO_NAME} literal in single-quoted heredoc never substituted #61.scripts/kg/exists in the template repo (build scripts, sparql queries, README, fixtures) but is not enumerated in the manifest, soclaude-md-snippet.md's### Knowledge Graphreference points at a pipeline that derived projects never receive. Pre-existing defect surfaced during this PR's manual adoption test on a FUNSD scratch clone, where the Claude Code session honestly flagged the gap. Design decision needed (ship the executable subset and gitignorebuild//.cache/, or remove the reference from the snippet). Tracked in scripts/kg/ exists in template but is missing from the manifest; CLAUDE.md hints at it #62.adopt --applyends with aReview the result with: git status && git diffline but noNext stepsblock, so a first-time adopter is not told that there are two git repos with separate flows (host vs wiki sub-repo), thatinit-wiki --githubadded a local commit that the team will not see until someone runsgit -C wiki/<name>.wiki/ push, or what slash commands to try in Claude Code. UX-polish surface surfaced by the same FUNSD manual adoption pass. Tracked in instantiate.sh & adopt.sh: closing-message guidance + CLAUDE.md content + README announcement gaps after bootstrap #63.ADOPTING.mdat the repo root orAdopter-Runbookwiki page that an external collaborator can follow end-to-end without reading the design docs. The existing surfaces (README, Adopt-Existing-Repo-Design wiki page, per-overlay READMEs) target different audiences; none answers "I have a GitHub project, what do I do step by step." Documentation half of the coherence that this PR's manifest consolidation made auditable; companion to instantiate.sh & adopt.sh: closing-message guidance + CLAUDE.md content + README announcement gaps after bootstrap #63's script-sideNext stepsblock. Tracked in ADOPTING.md + Adopter-Runbook wiki page: end-to-end instructions for external adopters #64.adopt.sh'slw_diefor--agent=cursorreferences closed issue adopt.sh: let an existing repository adopt llm-wiki-memory #6 (the original design issue, closed when adopt landed). The link goes to a closed issue that does not describe the deferred cursor work. Trivially fixable (one-line update to point at a live anchor). Tangentially related to this PR: PR template-manifest: single source of truth for template-owned files #60'slw_manifest_assemble_active_filesalready supportsagent=cursor; only adopt's parser refuses it. Tracked in adopt.sh: lw_die for --agent=cursor references closed issue #6 (broken anchor) #65..gitignoredivergence — the run-time advisory is enough; rare manual back-port.Co-authored-by: Priscila Saboia Moreira pmoreira@nd.edu