Feature: Cognate-Assist & Polymorphism Analysis Pipeline

[TarahAssistant]

Feature Request — Cognate-Assist & Polymorphism Analysis Pipeline (PARSE)

Author: drafted from the 2026-05-30 SK 82-concept analysis session
Status: Feature request / design spec (pre-MC). Build-ready once approved.
Scope: four capabilities — cognate-set assignment, loan diagnostics,
polymorphism coding, and sensitivity analysis (MP4 + single-speaker risk).
Assumed-correct inputs (OUT of scope, provided upstream): per-speaker form extraction,
survey-overlap resolution, noise filtering, and phonetic variant collapse. This feature
assumes the annotation data already contains correct per-speaker forms on the chosen tier.
Primary users: the PhD analyst (Lucas) working in Compare mode; downstream consumer is
a Bayesian phylogenetic pipeline (BEAST2/BEASTling) + the thesis methods chapter.

---

0. One-paragraph summary

Add a backend "cognate analysis engine" + a Compare-mode review layer that, for each concept:
(A) Cognate-set assignment — proposes cognate-class groupings from the per-speaker forms
using a computational cognate detector, and lets the analyst confirm/override them;
(B) Loan diagnostics — flags loanwords by matching forms against contact-language reference
data + loan lists; (C) Polymorphism coding — derives the coding where a speaker that produced
forms in ≥2 classes is coded into each (binary / cladistic-ambiguity style); and
(D) Sensitivity analysis — computes the MP4 majority-state resolution (Canby et al. 2024)
and a single-speaker-risk flag per state. All outputs persist in parse-enrichments.json and
drive both the Compare UI and a phylogenetic-matrix export.

The design principle (Häuser et al. 2024): the machine proposes, the expert disposes.
Nothing auto-commits a cognate judgment; every proposal is a reviewable suggestion.

---

1. Motivation

Today, cognate decisions in PARSE are made entirely by hand and stored as raw A/B/C speaker
lists under manual_overrides.cognate_sets. There is no:

• algorithmic proposal to seed the grouping (analyst starts from a blank grid),
• loan flagging surfaced in the workflow,
• automatic polymorphism coding (analyst manually reasons about multi-class speakers),
• sensitivity analysis (MP4) — which the protocol documents mark as mandatory for thesis use,
• single-speaker-risk surfacing,
• export to a phylogenetic matrix format.

The 2026-05-30 session did all four capabilities below by hand for 82 concepts. ~70% of that
labour is mechanizable. This feature turns the per-concept work from "author from scratch" into
"review a proposal," and makes the thesis-required sensitivity analysis a button instead of a
manual pass.

---

2. Definitions (lock terminology)

Term	Definition
Cognate class / state	A set of forms descended from one etymon. Labelled A, B, C… per concept.
Cognate set	The mapping {state_letter: [speakers]} for one concept.
Polymorphism	One speaker/doculect attests forms in ≥2 cognate classes for the same meaning.
Binary cognate encoding	One presence/absence character per class; a speaker gets 1 in each class it attests (Greenhill/ABVD tradition).
Cladistic ambiguity	In likelihood calc, a polymorphic taxon has likelihood = 1 for each observed state.
MP4	Canby et al. (2024): for each polymorphic speaker on a character, retain only the state most frequent across the whole dataset for that character.
Singleton state	A class attested by exactly one speaker (standard practice flags/removes these).
Survey-overlap ids	Multiple numeric concept_ids for the same gloss/elicitation (e.g. hair = 1, 249, 250, 599). Decisions must be written under all of them.
Loan	A borrowed form (Arabic/Persian/Turkish in SK). Recorded but excludable from inherited-signal runs.

---

3. Existing PARSE surfaces this builds on (verified 2026-05-30)

• python/compare/cognate_compute.py — has load_annotations(annotations_dir) returning
  forms_by_concept (the FormRecord lists Compare/cognate use) + a cognate_compute_preview
  path. This is the integration point for the proposal engine.
• parse-enrichments.json — manual_overrides.cognate_sets[<concept_id>] = {A:[...],B:[...], C:[...],D:[...],E:[...]}; also manual_overrides.speaker_flags, canonical_lexemes; root
  cognate_sets/similarity (often empty until computed); lexeme_notes.
• parseui-compare-notes-v1.json — flat { "<concept_id>": "<note string>" } = the Compare
  per-concept general-notes field.
• concepts.csv — id, concept_en, source_item, source_survey, custom_order. Drives gloss↔id
  and survey-overlap resolution.
• Contact-language reference (CLEF / config/sil_contact_languages.json) — Arabic/Persian
  reference forms (script + IPA) per concept; the loan-matching corpus.
• Compare UI — currently renders cognate states A–E (5 max). ⚠️ Bird/sand in the SK set
  need 6 (A–F); whether the grid renders a 6th state is an OPEN question (see §11).

Build note: the spec must be verified against current code before implementation — these
citations are from a 2026-05-30 read and may have drifted.

---

4. Inputs and outputs (data contract)

Input (assumed correct, provided upstream — see scope)

Per concept, the per-speaker form set on the chosen tier:

{
  "concept_id": "338",
  "gloss": "sand",
  "survey_overlap_ids": ["338"],
  "tier": "ipa",
  "forms_by_speaker": {
    "Qorv01": ["šin"],
    "Fail03": ["šin", "māsa"],
    "Badr01": ["šin", "rix"],
    "...": ["..."]
  }
}

Output (this feature produces)

{
  "concept_id": "338",
  "cognate_sets": { "A": ["Qorv01","Kalh01","Kalh02","Fail03","Badr01"],
                    "B": ["Fail01"], "C": ["Fail03","Saha01"],
                    "D": ["Badr01"], "E": ["Khan02","Mand01"], "F": ["Qasr01"] },
  "state_meta": {
     "A": {"gloss_label":"šin/šen","etymon":"inherited","loan":false,"n_speakers":5,"singleton":false},
     "E": {"gloss_label":"qom/qum","etymon":"Arabic/Persian","loan":true,"n_speakers":2,"singleton":false},
     "F": {"gloss_label":"xak-i biaban","etymon":"Persian phrase","loan":true,"n_speakers":1,"singleton":true}
  },
  "polymorphic_speakers": {"Fail03":["A","C"], "Badr01":["A","D"]},
  "mp4": {"resolved_state_by_speaker": {"Fail03":"A","Badr01":"A"}, "dataset_majority_state":"A"},
  "single_speaker_risk_states": ["B","D","F"],
  "provenance": {"proposed_by":"lexstat","confirmed_by":"manual","model_version":"...","ts":"..."}
}

---

5. Requirement A — Cognate-set assignment (proposal + review)

Goal: given forms_by_speaker, propose {A:[...],B:[...]} groupings; analyst confirms/edits.

A.1 Proposal engine (backend) — pluggable, multiple selectable methods.

• Run a computational cognate detector per concept over the speaker forms. The detector is
  selectable (not a single hard-wired algorithm) so the analyst can compare methods and
  cross-check agreement. Support at minimum these methods (see §5a for the SIL-Cog-derived three):
  • Phonetic — pairwise phonetic-similarity threshold.
  • Blair — regular-correspondence + "similar-segment" categories (survey-oriented; well
    suited to closely-related dialect data like SK).
  • Dolgopolsky — sound-class matching on leading segment pairs.
  • LexStat / SCA — the existing LingPy pipeline in cognate_compute.py if present;
    otherwise SCA (sound-class alignment) edit distance as a lighter fallback.
• Cluster forms into classes by a tunable per-method threshold.
• Output: a proposed cognate_sets + a per-pair similarity matrix + a confidence per cluster +
  the method used. Optionally run >1 method and surface an agreement indicator (methods that
  agree on a grouping raise confidence; disagreement flags the concept for closer review).
• The proposal is written to a preview/staging field, NOT directly to manual_overrides.

A.2 Review affordance (frontend, Compare).

• Show each speaker's form with its proposed state badge (visually distinct from a
  confirmed badge — e.g. dashed outline / "suggested").
• Analyst actions per concept: accept-all, or drag/reassign a form to a different state,
  create a new state, merge two states, split a state.
• On confirm, the proposal graduates into manual_overrides.cognate_sets (with provenance).
• Never overwrite an existing human decision silently; if a proposal differs from a stored
  decision, show a diff and require explicit accept.

A.3 Survey-overlap write fan-out.

• When a decision is confirmed for a gloss, write the identical cognate_sets to every
  survey-overlap id (hair → 1, 249, 250, 599). One confirm, N id writes. Resolve the id set
  from concepts.csv (same concept_en, or an explicit overlap map like survey-overlap.json).

Automation level: algorithm-assisted, human-confirmed. (Häuser 2024: do not auto-commit.)

---

5a. Requirement A (detail) — cognate-identification methods (SIL Cog)

The proposal engine (A.1) must offer the three cognate identifiers SIL Cog provides, plus the
LexStat/SCA option. They are complementary — each errs differently — so exposing all of them
(and an agreement check across them) is the point. Verified against Cog's docs/wiki, 2026-05.

Shared pre-processing (all methods rely on it):

• IPA segmentation — split each word into C/V segments (assumed available from upstream).
• Word alignment — align segment pairs between two words by phonetic distance (Cog uses
  built-in IPA knowledge of segment similarity; ALINE/Levenshtein-class alignment acceptable).
• Statistical sound-correspondence inference — estimate, per environment, the probability
  that a segment in variety X corresponds to a segment in variety Y. This is run as part of the
  engine and feeds the cognate decision (see Blair) and can also surface recurrent
  correspondences for the analyst (complements the hand-built 04a correspondence table).
• Iterative convergence loop — align → infer correspondences → identify cognates → repeat
  until stable, rather than a single pass. (Cog's design; materially improves results.)

Method 1 — Phonetic.

• Cognate iff pairwise phonetic similarity ≥ a threshold. Simplest; no correspondence model.
• Pros: fast, no training. Cons: conflates regular sound change with chance similarity; the
  literature criticises pure edit-distance/phonetic methods for exactly this. Use as a baseline.

Method 2 — Blair.

• Assigns each aligned segment pair to a category based on (a) whether it is a regular
  correspondence (from the inferred correspondence model) and (b) whether the two segments are
  "similar" (analyst-specified similar-segment sets, or a phonetic threshold). Cognacy is
  decided from the category profile across the word.
• Purpose-built for language survey of closely related varieties — the SK dialect case.
  Recommended default for this dataset; a good middle ground between naive inspection and the
  full comparative method.

Method 3 — Dolgopolsky.

• Uses Dolgopolsky sound classes (groups of sounds that historically interchange). Cognacy
  is judged from how many leading segment pairs fall in the same sound class (classic rule:
  agreement on the first two consonantal classes ⇒ likely cognate).
• Pros: robust, language-independent, fast; strong for a first-pass screen. Cons: coarse; ignores
  vowels and later segments.

Selection + agreement.

• Analyst picks a method per run (default Blair for SK); engine records which method produced
  each proposal in provenance.proposed_by.
• Optional multi-method run: compute Phonetic + Blair + Dolgopolsky (+ LexStat) and show an
  agreement score per concept. High agreement → low-risk auto-accept candidate; disagreement
  → route to manual review. (Still expert-confirmed; never auto-commit — Häuser 2024.)

Config (per method, tunable, documented defaults):

• Phonetic: similarity threshold.
• Blair: similar-segment sets (or phonetic threshold), regular-correspondence source, category weights.
• Dolgopolsky: sound-class table (Dolgopolsky default), how many leading pairs must match.
• LexStat/SCA: cluster threshold, scorer.

Source: SIL Cog — methods verified from software.sil.org/cog + sillsdev/cog wiki
("Comparing Word Lists") + README, 2026-05. Cog's aligner internals are not publicly documented
beyond "phonetic distance per segment pair," so the alignment algorithm choice is left to the
implementer (ALINE or Levenshtein-with-IPA-weights both acceptable).

Out of Cog but worth noting as future visualization (NOT required here): Cog also produces
UPGMA / neighbor-joining dendrograms, radial trees, network graphs, similarity matrices, and
geographic maps. The phylogenetics path here goes through the NEXUS/CLDF export (§9) into BEAST,
so in-app trees are optional/nice-to-have, not part of this request.

---

6. Requirement B — Loan diagnostics

Goal: flag which states/forms are borrowings (Arabic/Persian/Turkish), surface in review,
and make them excludable from inherited-signal exports.

B.1 Matching (backend).

• For each form, compare against the CLEF contact-language reference (sil_contact_languages.json)
  for that concept (Arabic, Persian, optionally Turkish): script + IPA similarity above a
  threshold ⇒ candidate loan.
• Plus a curated loan wordlist (the q/ʕ/ʿ phonotactic diagnostics + known SK loan inventory
  from the protocol docs) as a second signal.
• Output: per form/state a loan_candidate boolean + donor_language + evidence
  (which reference matched, similarity score, which diagnostic fired).

B.2 Surfacing (frontend).

• Loan-candidate states/forms get a "loan?" chip in Compare, with hover evidence.
• Analyst confirms/dismisses → persists to manual_overrides.speaker_flags (or a new
  borrowing_flags field) keyed by concept_id + speaker + state.

B.3 Effect on export.

• Confirmed loans are tagged so the matrix export (§9) can emit an inherited-only variant
  (loans dropped) alongside the full matrix.

Automation level: semi-automatable — matching is scriptable; the final loan call is a
review decision (machine surfaces evidence, human confirms).

---

7. Requirement C — Polymorphism coding

Goal: mechanically derive the binary/multistate coding from the confirmed cognate sets.

C.1 (backend, deterministic).

• A speaker present in ≥2 states for a concept is polymorphic: it contributes 1 to each
  state's binary character (cladistic ambiguity convention).
• Emit polymorphic_speakers: {speaker: [states]} per concept.
• Emit the binary character matrix: one column per (concept, state); cell = 1 if speaker in
  that state else 0; polymorphic speakers have multiple 1s within a concept block.
• Singleton states (1 speaker) are computed but tagged (see Requirement D) — standard practice
  optionally removes them; make removal a toggle, not a default deletion.

C.2 (frontend).

• Compare shows a small "polymorphic" indicator on speakers coded into >1 state, with a tooltip
  listing the states + forms.

Automation level: fully scriptable given Requirement A output. No human input needed beyond
the already-confirmed sets.

---

8. Requirement D — Sensitivity analysis (MP4 + single-speaker risk)

Goal: the mandatory sensitivity analysis + confidence metadata.

D.1 MP4 (backend, deterministic). For each concept:

• dataset_majority_state = the state with the most speakers across the concept (ties: tunable
  rule — default lowest-letter, document it).
• For each polymorphic speaker, mp4_resolved_state = the single most-frequent state that the
  speaker actually attests (i.e. intersect speaker's states with global frequency ranking).
• Produce an MP4-resolved matrix: each polymorphic speaker collapsed to its one MP4 state,
  emitted as a parallel matrix to the full polymorphic matrix.

D.2 Single-speaker risk (backend, deterministic).

• A state attested by exactly one speaker ⇒ singleton=true, risk = High.
• A polymorphic speaker whose only evidence for a state is itself ⇒ flagged.
• Surface as a single_speaker_risk_states list + per-row risk level in the report.

D.3 (frontend).

• A per-concept "sensitivity" panel: shows the full grouping vs the MP4-collapsed grouping
  side by side, and lists singleton/high-risk states.

Automation level: fully scriptable — pure counting over the confirmed matrix.

---

9. Phylogenetic-matrix export (downstream, ties it together)

Not one of the four core capabilities but the reason they exist; include as a sibling deliverable.

• Export the binary cognate-class matrix in NEXUS (BEAST2/BEASTling) and CLDF
  (FormTable + CognateTable) formats.
• Emit both the full polymorphic matrix AND the MP4-resolved matrix (mandatory sensitivity).
• Emit an inherited-only variant (loans dropped) as a third sensitivity.
• Include a provenance header (config, thresholds, model version, date, speaker set, tier policy).

---

10. API / backend surface

Expose as chat tools (per the PARSE MCP/chat-tool pattern) + REST routes:

• cognate_propose(concept_ids[], config) → proposed sets + similarity + confidence (preview only).
• loan_scan(concept_ids[]) → loan candidates + evidence.
• cognate_derive(concept_ids[]) → polymorphism coding + MP4 + risk for already-confirmed sets.
• cognate_export(format, sensitivity) → NEXUS / CLDF matrix.
• All read via cognate_compute.load_annotations; all write atomically to parse-enrichments.json
  with backup + os.replace (never partial in-place). Survey-overlap fan-out handled in the
  write layer.

---

11. Persistence / schema changes

• Extend manual_overrides with: state_meta, borrowing_flags, provenance per concept.
• Add a cognate_proposals (preview/staging) top-level key so suggestions never collide with
  confirmed manual_overrides.cognate_sets.
• OPEN: cognate-state cap. Current Compare renders A–E (5). SK already needs A–F (6) for
  bird+sand. Decide: (a) make the state count dynamic (render from Object.keys of the cognate
  dict — preferred, future-proof), or (b) hard-cap and force a merge. This must be resolved
  for the SK data and is a likely small parse-front-end change. Grep the Compare component
  (ParseUI.tsx / ConceptSidebar.tsx / conceptGrouping.ts) for the label source.
• Notes: requirements A–D should be able to write a generated per-concept summary to
  parseui-compare-notes-v1.json (append under a marker, preserve existing), exactly like the
  2026-05-30 manual pass did.

---

12. Edge cases (must handle, all seen in the real SK data)

• Mixed-meaning elicitations. Some glosses pool distinct senses under one concept (fly =
  verb "to fly" + noun "fly/insect"; you = sg/pl/reflexive/sentence forms across 10 survey
  ids). Automation must detect and flag these (multiple survey ids with divergent meanings;
  forms that don't cluster) and route to a concept-split decision — NOT silently force one set.
• Wrong-organ / mis-elicitation (liver→"kidney", dog→"camel"): can't be auto-resolved;
  flag low-confidence singletons for human review.
• ASR/English artifacts in the data (out-of-scope to filter per assumption, but the engine
  should be robust: a non-phonotactic outlier should land as a flagged singleton, not silently
  create a spurious state).
• Survey-overlap consistency: all ids for a gloss must end with identical sets; export must
  dedupe so a concept isn't double-counted in the matrix.
• Empty / no-form speakers: a speaker with no token for a concept = absence (0 in all
  states), not a state.
• 6+ states (see §11).

---

13. Acceptance criteria

• Given correct forms, cognate_propose returns a grouping for every concept; analyst can
  accept/override in Compare; confirmed sets persist under all survey-overlap ids.
• loan_scan flags the known SK loans (qalb, ʿasal, qirmiz, zānū, āftāb, parande, ṭayr, etc.)
  with evidence; analyst confirm/dismiss persists.
• cognate_derive reproduces, from the confirmed sets, the polymorphism coding + MP4 + risk
  that match the hand-authored 2026-05-30 report (use it as a golden regression fixture).
• cognate_export emits valid NEXUS + CLDF, full + MP4 + inherited-only variants, with provenance.
• 6-state concepts (bird, sand) render correctly in Compare (after the §11 decision).
• All writes atomic + backed up; existing manual_overrides/notes preserved.
• Validation: vitest (Compare DOM: N state badges for N-state cognate dict) + pytest (engine:
  golden-fixture diff against the 2026-05-30 report) + tsc + build. No browser/parse-run per ban.

Golden regression fixture

The 2026-05-30 outputs are the test oracle:

• reports/SK_Polymorphism_Decision_Report.csv (782 rows, 82 concepts)
• reports/sk_cognate_sets_82.json (111 ids)
• reports/sk_summary_notes_82.json
cognate_derive over the confirmed sets MUST reproduce the polymorphism/MP4/risk columns; any
  diff is a bug in the engine OR a flagged human override.

---

14. Suggested lanes / phasing (for when this becomes an MC feature)

• Lane A (parse-back-end): engine — cognate_propose (LexStat/SCA), loan_scan (CLEF +
  loanlist), cognate_derive (Requirements C+D), atomic survey-overlap writer, golden-fixture
  pytest.
• Lane B (parse-front-end): Compare review layer — proposed-vs-confirmed badges, reassign/
  merge/split, loan chips, polymorphism indicator, sensitivity panel; dynamic state count (§11).
• Lane C (parse-back-end): matrix export (NEXUS/CLDF, 3 sensitivities) + provenance.
• Lane D (parse-coordinator): wire the chat tools/MCP, docs, end-to-end audit against the
  golden fixture.
  Build A→B in parallel behind the data contract in §4; C after A; D last.

---

15. Open questions for Lucas

1. State cap: dynamic state count (preferred) or hard-cap-and-merge? (§11)
2. Cognate-identifier methods: implement all of Phonetic + Blair + Dolgopolsky + LexStat/SCA
   (recommended, per §5a), or a subset? Default method for SK = Blair — confirm? Is LexStat
   already available in cognate_compute.py, or implement the Cog-style methods fresh?
3. Loan policy: separate borrowing_flags field, or reuse speaker_flags?
4. Concept-split workflow for mixed elicitations (fly, you) — in-scope here or separate feature?
5. Export targets — NEXUS + CLDF both, or just what BEASTling needs?
6. Tie-break rule for MP4 dataset-majority (lowest-letter default ok?).

---

16. References (from the protocol docs)

• Canby, Evans, Ringe & Warnow (2024) — polymorphism handling, MP4. Trans. Phil. Soc. 122(2).
• Häuser, Jäger & Stamatakis (2024) — integrating out subjectivity in synonym selection. arXiv:2404.19328.
• Greenhill, Blust & Gray (2008) — ABVD; binary cognate encoding.
• List/Forkel et al. — LingPy, CLDF, LexStat, EDICTOR.
• SIL Cog — Phonetic / Blair / Dolgopolsky cognate identifiers + statistical sound-correspondence
  inference. software.sil.org/cog ; sillsdev/cog wiki "Comparing Word Lists" ; README (2026-05).
• Dolgopolsky (1986) — sound classes for cognate screening.
• Kondrak — ALINE algorithm; "Identification of Cognates and Recurrent Sound Correspondences".
• SK_Polymorphism_Handling_Protocol_for_Bayesian_Phylogenetics.md (project protocol).
• 04a_phonetic_correspondences.md / part2_sk.md (SK-specific correspondence + precedent).