Proposal: opt-in "coverage & discovery" feature (candidate registers · registry census · coverage audit · stub rule) + an any-means reciprocity gate fix

[paulrbrenner]

Filed by @paulrbrenner from a fleet of 5 derived wikis (4 running this capability, 1 deliberately without), with the orchestrator agent. Nothing has been changed in this template repo — this is a request for your evaluation.

TL;DR

We built and battle-tested a five-part "coverage & discovery" capability across four
derived wikis. It makes an evolving-landscape wiki self-correcting against the field
instead of dependent on the right keyword being in a search string. It is orthogonal to
the wiki mechanics you already ship (KG, gate, wiki-write-protocol) and layers cleanly on
top. We propose packaging it as an opt-in template feature (features/coverage-discovery/,
using the feature mechanism you already have) so any derived project can enable and
parameterize it. This doc explains the components, the evidence, the design decisions worth
preserving, where it does not apply, and the risks — so you can decide whether/how to
adopt it upstream.

We are not asking you to take our project-specific content; we are proposing the
generic pattern be available to all derived wikis.

---

1. Motivation — three failure modes of a landscape wiki

A wiki that tracks a fast-moving external field (research literature, a program landscape)
degrades in three ways that internal lint never catches:

1. Keyword-dependent scanning misses novelty. Confirming search strings find what you
   already know to look for. New entities appear under terms not in the strings.
2. The concept taxonomy ossifies. A fixed backbone (mechanisms, domains, pillars) has
   no home for emerging hypotheses, so they get dropped rather than parked.
3. "Named in prose, never paged" leaks. An intervention/entity mentioned on a page but
   never given its own node is invisible to the KG and to future queries.

The internal lint checks health (orphans, dead links, frontmatter). None of it asks
"what exists in the field that we don't cover?"

2. What we built and proved

Piloted on a Long COVID research wiki, then replicated to three others (AI-for-science,
defense-AI, human-AI-team-science). Concrete results:

• Registry census caught real gaps mechanically. Long COVID: a ClinicalTrials.gov
  census (656 interventional studies) surfaced ensitrelvir, taVNS, pyridostigmine, AXA1125
  — none of which depended on a keyword. Defense-AI: a USAspending census returned 157
  AI-relevant DoD awards, 63 untracked. AI-for-science: a HuggingFace proxy returned 183
  untracked science-relevant models/datasets.
• The "do not mass-stub" discipline held. Of the untracked long tail, only the clear
  in-scope leaks were stubbed (Long COVID: 12; Defense-AI: 0 — all 63 deferred);
  the rest defer to a quarterly audit. No KG flooding.
• The candidate watchlist + promotion rule behaved. Emerging mechanisms/themes were
  parked and not prematurely promoted (e.g. MCAS/small-fiber neuropathy strengthened but
  kept as candidates pending the documented threshold).
• We found and fixed a real gate hole (worth upstreaming as-is). The stub→backbone
  link was expressed via a tag/body-link, which the typed-edge-only reciprocity check never
  fired on — 14 pages silently missed their back-reference and the gate still passed. Fix:
  a reciprocity criterion that triggers for a link by ANY means (tag, body link, OR
  typed edge). This is a generic gate improvement independent of the rest of the proposal.

3. The five components (generalized)

1. Registry census + diff. Pull a canonical enumeration of the domain's tracked
   objects; diff against what the wiki pages; untracked in-scope items become candidate
   stubs. Registry is a pluggable adapter: ClinicalTrials.gov (medical), USAspending/SAM
   (federal programs), Papers-with-Code/HuggingFace (AI methods/models), or none (pure
   literature → this folds into the audit). Runs per ingest cycle.
2. Open taxonomy + candidate register + promotion rule. Declare the backbone
   non-exhaustive; a lightweight Candidate-*-Watchlist holding page (one line per
   hypothesis, not a page each); a documented promotion rule (e.g. ≥2 independent
   cohorts/sources OR 1 Tier-1 review) that logs an auditable promotion event.
3. Discovery + adjacent-field search families. Alongside the confirming strings: a broad
   periodic sweep, a rotating candidate-census over watchlist terms, and adjacent-field
   queries (overlapping literatures where signals surface first).
4. Quarterly external-coverage audit (distinct from lint). Diff the wiki's inventory
   against (a) taxonomies in recent field review articles and (b) the census. Misses flow
   into the registers (watchlist entry or candidate stub, tagged with the audit id) + a
   dated log entry. No parallel gap-ticket system.
5. Stub rule + hardened reciprocity gate. Every named entity/concept that is the subject
   of a claim (or surfaced by census/review) gets ≥ a stub node (graded lowest-confidence,
   lifecycle: candidate, citing what named it). The gate enforces: (i) named→resolves,
   and (ii) any-means reciprocity (see §2).

4. Proposed template integration — an opt-in feature

Your repo already has the feature mechanism (features/, feature.json,
enable-feature.sh, install-feature.sh) with agent-comms as precedent. We propose:

features/coverage-discovery/ shipping:

• SCHEMA_ section fragments: lifecycle: candidate addition, open-taxonomy statement,
  promotion-rule template, stub invariant, discovery search-parameter families.
• verification-gate fragments: the named→resolves + any-means reciprocity criteria
  (with a hub-page exemption — hubs legitimately list many children).
• A census.py skeleton with a small registry-adapter interface and reference adapters
  (clinicaltrials, usaspending, papers-with-code/huggingface, and a none mode).
• A /coverage-audit command template (quarterly).
• /ingest step fragments: census + stub-sweep.
• A Candidate-Watchlist.md.template.

Recommendation: the any-means reciprocity gate fix (§2, last bullet) should land in
the base template's verification-gate.md regardless of whether the full feature is
adopted — it's a correctness fix to the reciprocity check you already ship.

5. Per-project parameterization (the config surface)

A feature.json/config would capture the ~4 domain knobs: the backbone-taxonomy name
(mechanisms / domains / pillars / concepts), the registry adapter (or none), the
adjacent fields, and the promotion threshold. Everything else is generic.

6. Design decisions worth preserving (hard-won)

• Census is maintenance-side, not scan-side — it must read local wiki state and be
  deterministic; scan agents surface discovery leads, they don't run the census.
• Do-not-mass-stub — the census defers its long tail to the quarterly audit; only clear
  in-scope leaks are stubbed live. Without this, the KG floods.
• Watchlist (holding page) for few/conceptual items; individual stubs for many/1:1 items.
  Mechanisms → one watchlist page; interventions/programs → individual KG nodes.
• Any-means reciprocity — do not restrict the reciprocity check to typed edges.

7. Where it does NOT apply

• Bounded / single-source corpora (e.g. a wiki seeded from one lab's own output): there
  is no open external field to be complete against, so the feature should stay off. This
  is why one of our five wikis deliberately does not use it.
• Pure-literature domains with no registry: enable it with registry: none so Discipline gates + Verification Gate + PostToolUse advisory hook #1 folds
  into the audit (Attribute every wiki log entry to a user, one commit per log entry #4) rather than shipping a census script.

8. Risks, costs, mitigations

• KG flooding → the do-not-mass-stub rule (mandatory in the feature).
• Census feed reliability → adapters are best-effort and labeled; a feed being down is
  non-blocking (proven — SAM.gov/Papers-with-Code were unreachable in runs without failing).
• Per-domain census maintenance → the adapter interface keeps domain logic small; most
  projects use one existing adapter or none.
• Reciprocity false-positives on hubs → hub-page exemption in the criterion.

9. Backward compatibility

Opt-in. Existing wikis are unaffected until they run enable-feature.sh coverage-discovery.
The one exception we'd argue for shipping unconditionally is the reciprocity gate fix (§4).

10. What we're asking you to evaluate / decide

1. Do you want this capability in the template at all, or should it stay per-project?
2. If yes: package as a feature (our proposal) vs. fold into base SCHEMA_/gate?
3. Adopt the any-means reciprocity gate fix into base verification-gate.md now,
   independent of the rest?
4. If you want it, we can prepare a PR against your current manifest/feature structure
   (template-manifest.sh / features/) — tell us the shape you prefer and we'll conform
   to your conventions rather than impose ours.

Evidence / reference: working implementations live in five derived wikis (four with the
feature, one deliberately without); happy to walk through the LongCovid pilot, the live
census outputs, and the gate-hole post-mortem on request.

[paulrbrenner]

Follow-up from field-testing: decompose the audit cadence (weekly ↔ quarterly is too disparate).

Running the capability across four wikis surfaced a gap the original proposal glosses: it defines a weekly fast loop and a quarterly /coverage-audit, but nothing in between. Two things fall into that ~12-week gap:

1. Promotion-ready candidates sit as thin stubs. When a watchlist candidate clears its threshold mid-quarter, it's flagged eligible but held for the audit — so for up to a quarter a concept that has earned a full page has no proper home for incoming weekly evidence, and queries get a stub instead of a synthesis.
2. The census's ambiguous in-scope long tail stays untracked. The weekly stub-sweep only pages the clear leaks and defers the ambiguous ones to the audit, so an in-scope-but-not-obvious item can wait a quarter.

The fix isn't "run the audit weekly" (the review-taxonomy diff yields no fresh signal most weeks). It's to split the audit by natural cadence:

• Weekly — fast loop (already there): scan + discovery + census + stub-sweep.
• Monthly — light reconciliation (new): promotions + census-triage. Cheap; no review-article parsing.
• Quarterly — deep backstop (the existing audit): the full review-taxonomy diff + deep reconciliation.
• Event-driven: when the weekly ingest ingests a new Tier-1 review/survey, run the taxonomy-diff against that review immediately rather than waiting for the calendar.

Net: promotion/triage latency drops from ~a quarter to ~a month, the expensive taxonomy diff stays quarterly, and fresh surveys get diffed on arrival. Suggest the feature ship a coverage-reconcile (monthly) command distinct from coverage-audit (quarterly), with the middle cadence configurable per project. We've drafted this for our four wikis and can fold it into any PR against your feature structure.

[psaboia]

Thanks @paulrbrenner — this is great work, and the fact that you ran it on four wikis with a control before bringing it here makes it much easier to evaluate. Answers to your §10 questions below, plus a heads-up on what our feature machinery can and can't absorb today.

Q3 — the reciprocity gate fix: confirmed on our side; send that PR first.
We checked the base verification-gate.md: the back-reference criterion (line 43) triggers only on frontmatter typed edges; tag/body links pass with resolution-only checking. Your 14-page silent miss matches what we ship today. One scope decision to settle in the PR, though: any-means reciprocity changes the cost of every ingest in every derived project (every body link would demand a back-reference; the hub exemption helps but doesn't make that free). The alternative is keeping the typed-edge trigger and adding a SCHEMA convention that semantic relations must use typed edges — your stub→backbone case then fails a cheaper convention check. Your field data favors any-means for landscape wikis; if you have data on false-positive rates on non-landscape pages, bring it to the PR and we'll decide there. Either way: send that PR now, separate from everything else.

Q1/Q2 — yes to the capability, as an opt-in feature. One caveat first: the current feature mechanism can't install most of what §4 lists. The install_feature surface today is one code directory, one tests directory, one CI workflow, and one CLAUDE.md section. It cannot:

1. Ship slash commands (/coverage-audit, /coverage-reconcile) — no file-level install into an existing .claude/commands/. Precedent: Add /ask slash command for agent-comms cross-agent consultation #80 shipped /ask as a base-template command with a runtime guard instead.
2. Touch the wiki repo at all — SCHEMA_ fragments and Candidate-Watchlist.md.template have no delivery mechanism; the installer only writes to the project repo.
3. Extend verification-gate.md — it's a synced file (TEMPLATE_SHARED_INFRA), so local edits become sync drift; its "Project-specific criteria" section is the intended extension point but currently conflicts with sync — a conflict your proposal surfaced.
4. Reach adopted projects or update existing installs — feature directories aren't in the sync manifest, and there's no feature-update mechanism yet (tracked as Feature update mechanism (RFC #13 Etapa 3): changes to feature sources never reach existing installs #83).

So the practical path is phased:

• Phase 0 (now): the gate-fix PR above.
• Phase 1 (infrastructure, on us / co-designed): extend the feature mechanism — file-level installs into existing dirs, a wiki-side payload type, and sentinel-based fragment injection for gate/SCHEMA (the lw_inject_block machinery the overlays already use is the natural base). These unblock every future feature, not just yours.
• Phase 2 (your PR): features/coverage-discovery/ per docs/adding-a-feature.md, with the config surface you outlined (§5) in feature.json, adapters best-effort as you designed, and the cadence decomposition from your follow-up comment (weekly / monthly reconcile / quarterly audit / event-driven) folded in as the default shape, with the middle cadence configurable.

Q4 — shape guidance for that PR: feature.json + code/ (census skeleton + adapters) + tests/ (harness fixture per scripts/test/README.md) + ci/. Until Phase 1 lands, commands can follow the #80 pattern (base-shipped, runtime-guarded, inert without the feature). Anything the feature adds to the template tree outside features/ must be listed in scripts/lib/template-manifest.sh — see the maintainer note in README §9.

We'd like your input on the Phase 1 design — the payload-type shapes especially. We'll open that issue and tag you before you build Phase 2.

[chrissweet]

Sharing an empirical observation from a comparison across knowledge graphs before endorsing where this goes next.

The graph pattern. Comparing two of the coverage-discovery wikis you shared (AI-Sci-Disc-Mem, SavoieAgentKnowledge) against a research wiki using the same llm-wiki-colab ontology (markov_embeddings_and_rag), one pattern stands out:

Wiki	Pages	SourceSummary	Synthesis	Real typed-tension edges
AI-Sci-Disc-Mem	137	115 (84%)	0	0
SavoieAgentKnowledge	110	91 (83%)	0	0
markov_embeddings_and_rag	51	5 (10%)	18 (35%)	13

Same ontology, same edge vocabulary (extends, supports, criticizes) documented in the Edge-Types page, same reciprocity gate. Just very different usage patterns. Both extends uses in the two coverage-discovery wikis are the SCHEMA-to-Theory-X boilerplate example. All 13 typed-tension edges in the research wiki are domain-authored (e.g., Dense-RAG-Scale-Collapse criticizes RAG-Sanity-Check, Claims-Plus-Walk supports Markov-Retrieval-Tutorial).

How we are reading it. Not an authoring failure. LLMs given a workflow of "read source, summarize, cross-link to related pages" reliably produce SourceSummary pages, but do not reliably do the cross-reading required to type an extends or criticizes edge in the same session. James Omegaice has seen the same non-determinism from the read side: LLMs given a wiki do not always read what you tell them to. Single-shot instructions to "add typed edges" are unreliable, and the missing annotation layer visible in the graph data is a natural consequence.

This does not undermine the failure modes listed in the issue. Keyword-dependent scanning, taxonomy ossification, and named-in-prose-never-paged are all real. The graph pattern suggests a prior failure mode alongside them: pages exist without ever being reasoned across, and the SourceSummary layer grows without a corresponding synthesis layer.

Complementary experiment alongside #84. Building on our RWR/Markov-RAG line of work, where reliability lives in distributions rather than single invocations: run repeated compare-and-contrast passes over selected page groups (siblings under a shared hub, tag-overlap pairs, co-mention pairs). Each pass proposes typed edges or new pages with lifecycle: candidate. Individual passes are unreliable; the distribution across many fills out the annotation layer. Coverage becomes measurable on-substrate: every SourceSummary page either earns a typed-tension edge to another page within N passes or moves to lifecycle: dormant. No outside registry required for that metric.

If the experiment produces the annotation-layer growth we expect, some of the coverage-discovery machinery may compose differently than the Phase 2 scope currently anticipates. Specifically, the census could feed synthesis prompts rather than stub-file directly, and the watchlist mechanism could dissolve into lifecycle: candidate pages that participate in the discipline layer natively. If it does not produce that growth, Phase 2 as scoped is closer to right. Either result is useful information for the phased plan.

To run the experiment we would need the downloaded PDFs (or whatever raw source form is available) for the SourceSummary pages, plus wiki access to one of the five projects of Paul's choice, ideally as a branch or fork so nothing on a main branch is touched. Reaching out separately with materials logistics. Happy to share the pass log and any proposed typed edges back before anything lands.

Endorsing #84 as Phase 0. Proposing this experiment as a probe alongside it: the result feeds directly into what shape Phase 2 will most benefit from.

[paulrbrenner]

Closing the loop here rather than leaving you waiting — thank you for the detailed phased plan, which was more than we expected.

We're not going to send the Phase-0 gate-fix PR, and we won't be pursuing Phase 2. The reason is organizational rather than technical: our group is evaluating an overhaul that would make the llm-wiki template function as an agentic tool instead of every wiki vendoring a full copy of the template code repo. Since that removes the very distribution model this proposal was designed around — a features/ payload installed into each derived project — it didn't seem right to have you build Phase-1 installer infrastructure for a feature shaped by assumptions that may be about to change. Better to say so now than after you'd invested in it.

Two things that may still be useful to you, with no obligation attached:

1. The reciprocity gap you confirmed is real and independent of everything else here. Your base verification-gate.md triggers the back-reference criterion only on frontmatter typed edges, so a semantic relation expressed as a tag or body link passes with resolution-only checking. That's what produced our 14-page silent miss. Whatever shape the overhaul takes, that hole is worth closing — and your open scope question (any-means vs. requiring typed edges by convention) is the right one to settle. We still have no false-positive data from non-landscape pages, so we can't answer it for you.
2. The field data stands on its own. Five wikis ran this capability for several weeks: a registry census caught interventions that keyword search missed; a candidate register with an explicit promotion rule kept an evolving taxonomy honest; and a "named entity resolves to at least a stub" invariant closed the named-in-prose-never-paged leak. The cadence decomposition in the comment above (weekly / monthly reconcile / quarterly audit / event-driven) is the part we'd most recommend carrying forward in any redesign, since the naive weekly-vs-quarterly split was the one thing that clearly didn't work.

Happy to share implementations, measurements, or post-mortems if any of it helps the agentic-tool design. Otherwise, feel free to close this.