Skip to content

Commit 99f4144

Browse files
authored
Unify same-epoch fork resolution through distributed convergence (#1293)
* test(cgka-engine): pin the pre-unification fork-resolution routes Characterization evidence for Slice C (deleting the pairwise same-epoch fork-resolution route). Before the route disappears, pin what it does and prove the surviving convergence route already agrees with it. Rule agreement: the two fork-resolution rules agree eventually THE #1285 claim (option-c plan, N5), previously untested: a committer whose OWN commit won the pairwise race — rival root parked ConvergenceDeferred, never applied — must still reach the terminal state every convergence-resolved device reaches once valid depth disagrees with the ordering key. The test drives both routes side by side on one epoch-1 UpdateGroupData fork: A resolves pairwise (own commit wins, rival parked at its source epoch), an observer control resolves the identical fork through convergence and first agrees with A (ordering key at depth 1), the rival branch then grows two follow-on commits deeper, and both devices run their convergence passes. Verdict, pinned GREEN: A's pass re-admits the PARKED pairwise loser as branch input, reorgs A onto the fleet's deeper branch, and A, the observer, and the rival author agree on members, epoch, and group name; the parked root becomes Processed. This is the go/no-go evidence that deleting the pairwise route (Slice C) is deduplication of an already-agreeing rule, not a behavior change. Event fidelity: the exact event set of a convergence apply over an own commit U4 event fidelity on the post-#1285 base, both arms of the stored-convergence seam, driven through the deterministic router-flip fixture (now parameterized so either the inbound sibling or the device's own commit wins the ordering): - keep-own: the settling pass emits exactly the losing sibling's withdrawal pair (CommitRolledBack + GroupStateInvalidated at the source epoch) and never names the kept own commit. The pre-#1285 hazard — no checkpoint meant the kept own commit had no disposition and the superseded-processed sweep spuriously invalidated it — is closed by #1285's checkpoint materialization; this pins the clean behavior. - replace-own: exactly the two unattributed membership deltas at the selected tip plus one withdrawal pair naming the own commit by its stored id at its source epoch; the displaced own commit is parked ConvergenceDeferred at its source epoch, agreeing with the pairwise route's #1285 parking. This is the application-visible event contract the pairwise-route deletion (Slice C) must preserve. * test(cgka-conformance-simulator): pin route equivalence across the fork-resolution routes One same-epoch UpdateGroupData fork, resolved by every route that exists today in a single portable vector: the live committer's pairwise fork-recovery seam (alice, the deterministic ordering loser), post-restart convergence over hydrated storage (bob, the ordering winner, restarted while both commits are withheld so his in-memory committed_from guard is gone), and the observer convergence selector (carol, dave). The report trace confirms each mechanism empirically: alice records the single ForkRecoveryObservation and no convergence decision; bob and both observers each record a two-candidate convergence decision selecting the identical branch. Expected outcomes stay semantic (converged epoch/count, exact canonical equivalence, no pending work, one recovery) — no selected_branch_id or trace-byte pins, and the strict reliability tail mirrors add_strict_reliability_oracle. This is the Slice B acceptance pin the pairwise-route deletion (Slice C) must preserve unchanged. The vector is authored at conformance_version 0.9.11, the released version master's release prep (388a1d87) moved the tree to; canonical_vector_fixtures_match_generated_traces enforces the match and every other fixture already carries 0.9.11. * feat(cgka-engine)!: unify same-epoch fork resolution through distributed convergence Two routes existed for the same input: a committer that advanced from the fork's source epoch resolved a same-epoch rival PAIRWISE (the we_committed_from conjuncts in ingest.rs routing into ForkRecoveryManager::resolve), while every other member — and the same committer after a restart — adjudicated the identical rival through distributed convergence. Slice B proved the two rules agree eventually (committers_losing_rival_is_reconsidered_onto_the_fleets_deeper_branch), pass, and the restarted-committer pin already drove the convergence route end to end. The pairwise route was therefore a duplicate depth-blind fast-path over machinery that handles the general case. Delete it: - routing: drop the we_committed_from conjunct from the convergence admission gate (a committer's rival is admitted by the same retained source-epoch anchor as an observer's) and delete the WrongEpoch arm's pairwise block, its ingest retry loop, and the direct-seam ForkRecovered/GroupStateInvalidated emission pair. Slice A's loud fail-closed halt now also covers a live committer whose anchor is gone. - orphaned machinery, each grep-verified: ForkRecoveryManager (resolve, parking, incumbents, ForkResolution, the fork_resolution audit emission), the probe (probe_commit_ordering_metadata_for_recovery / ForkProbeError), forked_epoch_fail_closed (nothing in the engine produces EngineError::ForkedEpoch any more), the send/apply-time fork-{epoch}-{n}-{hash} snapshot captures, the fork- hydration scan (newest_fork_snapshot), DurableGroupEvolution.recovery_snapshot (serde-JSON rows tolerate the removed field), the snapshot half of PendingCommitCleanupGuard, and the whole committed_from surface in EpochManager (map, we_committed_from, owns_committed_from rollback discipline, prune) plus EpochManager::detect_fork, whose only caller was the deleted fail-closed helper. - the pending-publish origin-commit id — the one job the manager still did for confirm/fail bookkeeping — moves to a plain Engine::pending_origin_commits map (track/peek/take). Peel fallback keeps its coverage without the fork- snapshots: every seam that advances past an epoch retains a durable openmls-retained-anchor-N at that epoch first (confirm pre-merge, the inbound apply, join, the convergence apply), and both snapshot families were pruned by the same max_rewind_commits window, so available_past_peel_snapshots now reads the anchors alone. New raw-wrapper discipline replaces the pairwise-era retirement: once a peeled message is durably buffered as convergence input, its raw transport wrapper is retired Processed (mark_raw_transport_message_processed_if_awaiting_retry) instead of being reset Retryable and re-peeled on every later publish-cycle replay; replay_buffered_messages now respects verdicts ingest committed during the call. Known cost, deliberate: a live committer's rival resolution now waits for the convergence pass (P4 quiescence, ~1.1 s measured pre-#1285); sends queue during it. No expedite is added here. Test dispositions (intent preserved unless the subject was the deleted seam): - retargeted to the convergence route, outcomes unchanged: concurrent_invites_recover_to_deterministic_winner, strict_cutover_legacy_add_cannot_displace_valid_fork_incumbent (the materialization-time strict-cutover gate now does the rejecting; the forbidden candidate retires EpochInvalidated), convergence_privileged_remove_beats_grinding_ordinary_self_update, buffered_losing_fork_commit_raw_row_is_retired_after_confirm_replay, the two inbound-apply fault tests in auto_commit_atomicity (the fault now fires inside the settling pass's apply), and the concurrent-rename family in update_group_data (the winner's pass may emit a no-op invalidation naming the never-applied sibling; the pinned invariant is that it never withdraws its own origin). - renamed, pairwise vocabulary retired (former names in doc comments): pairwise_incumbent_defers_to_deeper_convergence_branch → incumbent_committer_defers_to_deeper_convergence_branch, pairwise_candidate_win_leaves_old_incumbent_reconsiderable → rival_win_leaves_displaced_own_commit_reconsiderable, pairwise_loser_is_reconsidered_onto_the_fleets_deeper_branch → committers_losing_rival_is_reconsidered_onto_the_fleets_deeper_branch, restarted_committer_routes_same_epoch_sibling_into_convergence (drops the _not_fork_recovery suffix), buffered_self_evicted_row_stays_failed_not_swept_processed_on_replay → buffered_losing_rival_content_row_is_never_swept_canonical_on_replay (the SelfEvicted-at-replay shape is unreachable once the rival buffers before eviction settles; the guarded invariant — a losing rival's content row is never relabeled canonical — survives). - deleted with their subject: invite_wrap_failure_releases_pre_commit_recovery_snapshot (no snapshot exists to leak; the sibling staged-commit-cleanup test keeps the guard covered), the two committed_from rollback-discipline tests in epoch_manager (the map is gone), newest_fork_snapshot unit tests. - repurposed: confirmed_commits_prune_fork_recovery_snapshots_to_rewind_horizon → confirmed_commits_prune_retained_anchor_snapshots_to_rewind_horizon, pinning the boundedness of the snapshot family that now carries the admission role. GroupEvent::ForkRecovered stops being emitted here; the variant itself and its simulator/app/FFI surface are removed in the follow-up commit. Forensics read-compat is untouched: AuditEventKind::ForkResolution and incident-replay keep parsing historical exports. Rebase and formatting fixups folded in: - fork_detection.rs:1121 kept the pre-#1377 hardcoded EpochId(3) after this resolution moved the test past the pinned rewind horizon (terminal_epoch = V1_MAX_REWIND_COMMITS + 1). Lines 1091/1112 already used terminal_epoch; only the trailing epoch assert was left stale. - router_flip_fixture_arranged picked up a call that exceeds the line budget, so cargo fmt --check failed the fast-ci gate. Formatting only; no test logic changed. * feat(traits)!: retire GroupEvent::ForkRecovered and migrate every consumer Nothing emits ForkRecovered since the pairwise fork-resolution route was deleted — the convergence route's rollback pair (CommitRolledBack + GroupStateInvalidated) is the only commit-rollback signal. Pre-production stance: delete the variant end to end rather than carry dead vocabulary. - traits: drop the GroupEvent::ForkRecovered variant (doc references on CommitRolledBack / GroupStateInvalidated rewritten), its snapshot test and event_fork_recovered.snap. - marmot-app / marmot-uniffi: remove the match arms and the FFI GroupEventKindFfi::ForkRecovered variant; the app's tombstoning already keyed exclusively on GroupStateInvalidated. - simulator: remove ForkRecoveryObservation / RecoveryOrderingKeyObservation, ClientObservation.recoveries, the fork_recovered event counter, TraceExpectation::{ClientRecoveries, RecoverySummary} and their comparators, OracleBehavior::ForkRecovered (with its evidence counter, expectation mapping, and recommended-behavior entries), the ScenarioReport.recovery_observations projection and the campaign recovery_observation_count metric — with no seam able to produce a recovery observation, every one of these expectations became unobservable (strict-oracle trap: a vector carrying one could only fail). - incident-replay synth: the two fork-incident vector shapes assert a settled convergence_decision at the post-fork tip (epoch 2) instead of a recovery summary — resolution evidence, not just final-state equality (Rule 4 intent preserved). Historical-export PARSING is untouched: EventKind::ForkResolution and the classifier still read old JSONL, and marmot-forensics keeps AuditEventKind::ForkResolution with a doc note that only pre-unification versions emit it. - vectors re-derived against the unified engine (inputs unchanged, expectations only): route-equivalence.v1 (the Slice B acceptance pin, flipped: committer, observer, and restarted committer all resolve through convergence; recovery_summary count:1 → settled convergence_decision at tip 2), group-data-fork-recovery.v1, concurrent-invite-fork-recovery.v1, and both incident vectors get the same recovery_summary → convergence_decision substitution; publish-fail.v1 and three-client-message-exchange.v1 drop the fork_recovered:0 counters and empty recoveries arrays from their expected traces. Manifest coverage/next notes and SCENARIOS.md updated to the unified-route wording. - convergence-chaos family cases drop their recovery_summary expectations; the converged-equality and client-state oracles remain the acceptance surface. deliberate-fork-recovery harness tests keep their outcome assertions and lose only the recovery-observation bookkeeping (scenario_report_records_trace_log_recoveries_and_failures → scenario_report_records_trace_log_and_failures). Consumer migrations folded in, each one a fixture or claim that the retired vocabulary left stranded. Cross-route recovery re-authored over post-unification observables. Both cross-route fixtures arrived from master carrying `client_recoveries` and `recovery_summary`. They no longer parse (`unknown variant 'client_recoveries'`), which failed three tests outright rather than weakening them. Equivalence of what is checked, old -> new: - `recovery_summary { count: 1, recovered_epoch: 2 }` — the group-level claim that every client agreed on one recovery -> four per-client `convergence_decision` pins at `selected_tip_epoch: 3`. Per-client, not client-less: a client-less pin is satisfied by any single observer, which is the opposite of what the vector claims. Tip 3 is the settled decision; epoch 2 was the intermediate the deeper branch supersedes. - `client_recoveries { client: alpha, source_epoch: 1, recovered_epoch: 2 }` -> `client_epoch_changes` for alpha, exactly `[1->2, 2->3]`. This is an exact-list comparison, so it pins the same two scalars positionally: alpha leaves the epoch-1 fork source and lands on epoch 2. The trailing 2->3 is strictly additional. - `winner_differs_from_invalidated: true` -> per-client `group_profile` pins at `zeta-branch-depth-two`. Alpha's own displaced commit named the group `alpha-root`, so alpha projecting the winner's name is that claim stated per client. `selected_branch_id` stays unpinned by house rule: it digests randomized MLS bytes and changes every run. - the withdrawal itself -> `assert_own_commit_was_withdrawn` requires alpha's own root to carry both `commit_rolled_back` and `group_state_invalidated_superseded`, the ledger's handles for the `CommitRolledBack` + `GroupStateInvalidated` pair the engine emits together. The pre-existing all-client `Invalidated` disposition check stays; `Invalidated` alone is reachable from other transitions, so it was never the withdrawal claim on its own. - end-state equality keeps the existing `clients_exactly_equivalent`. `conformance_version` and `vector_version` are untouched; that schema change is additive within the existing shape. The pairwise fork-recovery route claim is retired. `fork_recovery.rs` is gone and ingest no longer calls `.resolve_fork_candidate(`, so the route inventory's source-marker audit failed on both of its cited production sites. Retire the claim instead of re-pointing it at the unified convergence site: that site is already registered as `stored_convergence`, so re-pointing would give one seam two route claims and count its evidence twice — the exact inflation the audit exists to catch. The inventory has no tombstone record, and its drift tests require registered routes and matrix rows to match one-to-one, so retirement is removal from `DecisionRouteId`, the inventory, and the table, with a prose retired-routes record in the matrix that says what happened, why it was not re-pointed, and where the evidence went. The two properties the route carried survived the unification as renamed engine regressions, `incumbent_committer_defers_to_deeper_convergence_branch` and `rival_win_leaves_displaced_own_commit_reconsiderable`; they are now campaign evidence for `stored_convergence`, whose adopted rule absorbs the reconsiderable-loser clause so the citation matches a stated rule. The abstract route-lifecycle mutant keeps its witness but drops its dependence on the deleted route label: it now terminalizes any volatile route's provisional winner and is named `provisional_winner_terminalization`. Limitations that asserted a live pairwise route are reworded to the surviving vocabulary. Verified with `cargo test -p cgka-conformance-simulator --test route_assurance --test mutation_adequacy` (11 passed) plus crate fmt and clippy. incident-replay fork prose drops the retired `RecoverySummary`: two `fork.rs` doc comments still named that conformance `TraceExpectation` variant as the thing a synthesized vector asserts. Name the observables the arms actually pin today instead — the group-data arm's per-client `GroupProfile` pin and the membership arm's `member_count == 3` — which also corrects the second staleness in the same sentence: only the membership arm is winner-agnostic. The privacy claim is unchanged and still holds; it rests on synthetic labels, not on the assertion being winner-agnostic. app_runtime.rs's import block was hand-rewrapped during this conflict resolution and left under-filled; rustfmt re-wraps it here. * docs: state the unified fork-resolution route - cgka-engine AGENTS.md: fork handling now points at the convergence admission gate + distributed_convergence.rs; the ForkRecoveryManager "Done" section is superseded by the route-unification note; the inbound seam-parity convention drops the deleted fork-recovery seam (two seams remain); the mirror-invariant bullet loses the uncompensatable pairwise side effects; module map entries for fork_recovery.rs and the guard's snapshot half are gone. tests/AGENTS.md follows. - audit-logging.md: snapshot_created and fork_resolution are documented as historical read-compat kinds (emitted only by pre-unification versions; convergence_decision is the current resolution evidence). - relay-delivery-telemetry.md: the post-settle reorg description names the convergence rollback pair instead of ForkRecovered. - convergence-reliability-plan.md: Milestone 6 gets a status note — the #1236 committer-versus-observer route asymmetry no longer exists structurally; the 6.1 route-equivalence item enumerates the routes that still exist. - learnings.md: new dated entry for the unification (the 2026-05-04 fork-recovery entry stays as history). - CHANGELOG (unreleased/Changed): one rule for every member, checkpoints materialize own commits, loud fail-closed halt on missing in-horizon material, ~1 s committer-side resolution latency, ForkRecovered event + FFI variant removed, forensic read-compat retained. - distributed_convergence.rs / openmls_projection.rs: two in-code doc comments still explained own-commit supersession as a consequence of a restart clearing the in-memory `committed_from` guard. That guard is deleted, and the hazard they describe is not restart-specific — any same-epoch sibling adjudicated through stored convergence reaches it. Reworded to the general case; the invariants themselves are unchanged. * test(cgka-conformance-simulator): pin post-unification convergence outcomes per client Now that one route resolves every same-epoch fork, the vectors have to say so per client. Four escapes, each one a passing assertion that did not carry the claim its vector advertised. The settled convergence decision, pinned per client. route-equivalence.v1's convergence_decision expectation carried no client, and a client-less ConvergenceDecision is satisfied by any single observed client's settled decision. One participant adjudicating passed the vector — the opposite of what it claims to prove: that the live committer, the restarted committer over hydrated storage, and both passive observers all resolve the same fork through the one distributed-convergence route. The exact regression it would have missed is this branch's own predecessor, where alice resolved the fork on the pairwise seam and emitted no convergence decision at all while every state oracle still passed. Replace the single client-less entry with four per-client pins (alice, bob, carol, dave); the four subsume it. Only client and selected_tip_epoch are pinned — selected_branch_id digests randomized MLS bytes and changes on every run. The winning branch, pinned per client rather than on convergence. ClientsConverged only required the named clients to *share* a group name, never a specific one, so a group-data fork vector passed whichever branch won. Reproduced on the committed fork-recovery incident vector: swapping only the two branch names — so the branch the tip_committer tiebreak selects is the one named loser-branch — still yielded PASS. Nothing else closed it. The observation.group_name == WINNER_BRANCH check in incident-replay's accept path ran at synthesis time only and never guarded the persisted artifact canonical_scenarios replays in CI. Hanging the pin off ClientsConverged was tried first and rejected. ClientsConverged is a cross-client agreement assertion — it compares the first observation's name against the rest — so attaching a value pin to it conflates two questions, and the value it pins is one client's, silently the first named. #1299's GroupProfile expectation already pins a named client's exact projected name and description, and it asks per client, which is what "agreement is not attribution" actually requires: both committers sitting on the loser's branch is convergence too, and only a per-client pin says which branch each of them projects. So the winner is pinned through GroupProfile: - synth.rs's group-data arm emits one GroupProfile pin per committer at WINNER_BRANCH with the empty description the profile deterministically carries; the prose explaining why the name is the only stable winner observable (a group-data commit changes only the name, so the two branches agree on epoch and member count, while selected_branch_id digests randomized MLS envelopes and differs on every run) sits beside them. The membership arm stays winner-agnostic — a real export cannot attribute the surviving membership branch. - accept.rs drops the bespoke winner check: the label search and the acceptance gate become the one compare_observed_trace the membership arm already used, and the accept tests assert the property through the expectation the vector carries. - fork-recovery-incident.v1 re-derived through the synthesis path (accept() reproduced it, so the pins are live-verified); the expectation-only delta applied by hand because the recorded inputs would otherwise pick up the known AcknowledgeOutbound.selection serialization drift. With the pin flipped to loser-branch the CI replay now fails clients_not_converged — the escape above, closed. - route-equivalence.v1 pins all four clients to "bob branch" (stable across 6 report-CLI runs) and keeps clients_converged for the epoch/member-count agreement it does assert. Bidirectional decryptability, probed rather than assumed. route-equivalence proves four member shapes resolve one fork through the same route and land on identical canonical state, but every assertion it carried is a passive observation of that state. Exact equivalence and a settled decision per client say the four agree about the group; none of them exercises the group afterwards. A post-convergence key schedule that converged structurally while leaving some direction undecryptable passes the vector today. Add a ProbeBidirectionalDecryptability step over all four clients plus the matching expectation: each sends one logical event and every directed sender-to-recipient edge must be delivered and decrypted. Removing the step alone makes the fixture fail missing_bidirectional_decryptability_probe, so the expectation is load-bearing. Placement is not free: the probe is a mutating step, and the per-client convergence-decision pins read the latest observation, so probing after the terminal observe_exact empties all four (missing_convergence_decision x4). It goes immediately before it. * test(cgka-conformance-simulator): add the bounded convergence-pressure campaign Exercise the unified fork-resolution route under finite pressure: two admins race same-epoch profile commits, one application send per client is issued after the rival branch is ingested (inside the quiescence window the committers wait out), one committer restarts mid-resolution, and a bounded tail applies three self-updates, a profile commit, and an admin-policy commit. Seeds permute the sibling release order, the restarted committer, the window-send order, and the tail ordering. An explicit advance_time before the race activates controlled virtual time, so the quiescence window is a real deadline instead of the far-future tick shortcut; every settle after it is await_quiescence, whose watchdog budget is the bounded-time assertion. Bounded queue behavior is a scenario_inputs_pending resource assertion at peak pressure. The mutating decryptability probe runs before the terminal observe_exact the strict oracle appends. The campaign is finite by construction and claims nothing about progress under an unbounded self-update stream. The generator gate runs in CI. The runnable seeded gate is ignored: it fails on engine behavior, not on the campaign. An application message accepted while the group is resolving a same-epoch fork is queued durably and then stranded -- nothing re-arms the retained-intent drain once the pass completes, and the engine's conformance structural progress reports runnable_work = 0 with no next wake while pending_work.queued_outbound_intents > 0. Formatting fixup folded in: lib.rs's re-export block was hand-rewrapped during the ForkRecovered-retirement conflict resolution and left under-filled once these generators joined it; rustfmt re-wraps it. * test(cgka-engine): model production per-epoch transport sealing in Tier 2 Every Tier-2 test installs the pass-through `MockPeeler`, which hands each engine every message verbatim. Production transports do not: a kind-445 group message is sealed under the sender's current-epoch exporter secret and carries no epoch hint, so a device that never entered that epoch state sees opaque bytes and the peel returns `DecryptFailed`. `support::epoch_sealed_peeler::EpochSealedPeeler` models that boundary with the same AEAD and framing the Nostr binding uses (`nonce || ciphertext`, empty AAD, group key from `ctx.exporter_secret`). The engine already wraps outbound group messages through the peeler with the pre-commit context, so a test opts in by installing the peeler and routing the messages it already routes. `tests/epoch_sealed_transport.rs` owns the sealing model's own semantics (round-trip under the sealing context, opaque to a different epoch state at the same epoch NUMBER — the rival-branch shape) plus one end-to-end proof that uncontested delivery is unaffected. Infrastructure only; no engine behavior changes. The chacha20poly1305 dev-dependency lands with its Cargo.lock entry so the commit builds --locked on its own. * fix(cgka-engine): read a rival branch's traffic under that branch's state Unifying fork resolution through distributed convergence made branch selection the only route, and branch selection ranks on valid commit depth and app-witness score. Both are counted over inputs this device has STORED. A group message is sealed under the sender's current-epoch exporter secret and the transport carries no epoch hint, so once two members commit from the same epoch, every message either branch sends afterwards is opaque to the devices that adopted the other branch: it sits retained as PeelDeferred, contributes neither depth nor witnesses, and is retried only under contexts this device has itself entered -- which never include the rival branch. Each committer therefore scores only its own branch, every device settles "stable" on its own lineage, and the split is permanent. The deleted pairwise route was immune by construction: CommitOrderingKey is content-only. The engine tier could not see this. Its MockPeeler returns msg.payload verbatim, so every Tier-2 client has perfect cross-branch visibility -- exactly what production lacks. 263ed4b7 added an epoch-faithful peeler; the proving test built on it fails on the parent commit: test incumbent_adopts_a_deeper_rival_branch_whose_traffic_only_it_can_read ... FAILED thread '...' panicked at crates/cgka-engine/tests/epoch_sealed_transport.rs:318:5: assertion `left == right` failed: the incumbent must adopt the deeper rival branch left: 2 right: 3 and so does the fleet, on convergence-e2e-delivery/v1 seed 99 case 0: invariant_failures: [ {"kind": "clients_not_converged", "message": "clients [alice, bob, carol, frank] did not converge"}, {"kind": "clients_not_exactly_equivalent"}, {"kind": "bidirectional_decryptability_failed", "message": "... [alice->bob, alice->carol, alice->frank, bob->alice, carol->alice, frank->alice]"}] alice settled at epoch 3/6 members on effective_commit_depth while bob, carol and frank settled at epoch 2/5 members -- all "stable", holding retained rows none of them could read. Extra drive rounds changed nothing. Candidate branch states are part of a group's peel context, not a new mechanism. openmls_projection::candidate_branch_peel_contexts materializes each candidate branch under a SnapshotRollbackGuard and captures its tip's owned GroupContextSnapshot; because that snapshot derives the exporter secret eagerly, the transient branch state is rolled back before any peel runs against it, and no async work happens inside a restored state. retry_deferred_peels then offers those contexts to every retained row through the ordinary ingest seam, so dedup, classification, retirement, and the flood cap stay in one place -- the per-row bookkeeping both sweeps share now lives in reingest_deferred_peel_row. Peeling is only the visibility half. A message readable ONLY under a candidate branch belongs to a lineage this device has not adopted, so ingest routes it to the convergence seam and never to the direct apply. That rule is load-bearing, not tidiness: a commit at the same epoch number already entered convergence on the epoch test, but an APPLICATION message did not, and force-feeding one to canonical OpenMLS raised Err(Backend("process_message: ValidationError(UnableToDecrypt(AeadError))")) which the deferred-peel sweep turns into an aborted drain. The rule also keeps the seam honest: the peeled bytes are evidence of nothing on their own, the next pass's OpenMLS replay is what authenticates them, and a failed peel is silence rather than a verdict. PeelRecoverySource makes the provenance a type rather than a flag. The uncontested path pays nothing. The sweep returns before any context work on an empty backlog; collection stops at a cheap "two commits share a source epoch" test before any replay; at most MAX_CANDIDATE_BRANCH_PEEL_CONTEXTS (8) branches materialize, one bounded replay each under the pass's own ReplayBudget shape; and an inability to enumerate branches -- missing anchor, missing own-commit checkpoint, exhausted budget -- yields no contexts rather than an error, because the pass and not this helper owns every verdict. Losing commits keep their ConvergenceDeferred parking untouched. One consequence needed care. Because candidate branch states are part of the peel context, deferred_peel_context_fingerprint must fold in the stored commit graph: a newly retained rival commit adds a readable context while the live epoch and the retained-anchor set are both unchanged, and without that term the sweep gate stays armed exactly where it must not -- on a device holding its own branch while the rival branch's traffic sits unreadable. The empty-backlog early return moved ahead of the fingerprint so describing the context costs nothing when there is nothing to sweep. E10 registers MAX_CANDIDATE_BRANCH_PEEL_CONTEXTS in the convergence constant ledger, its plan rows, and policy_contract::CONSTANT_DECISIONS as an operational-resource, non-interfering constant. Review follow-ups folded in: five defects in the branch-relative peel seam, none of which change behavior on a healthy path and all of which fail silently on an unhealthy one. **The fail-closed rewind-probe detector was disabled.** `candidate_branch_ peel_contexts` reuses `retained_anchor_probe_snapshot_name`, so the sweep's rewind and the pass's rewind at the same `(group, epoch)` hash to the same snapshot name. `create_group_snapshot` is INSERT OR REPLACE and `recover_interrupted_retained_anchor_probe` fails closed only on MORE THAN ONE probe, so a second interrupted probe silently overwrote the first instead of producing the two rows the detector exists to catch. The sweep's probe now carries its own `openmls-branch-probe-` prefix and the recovery scan — renamed `recover_interrupted_rewind_probe`, since it no longer covers one rewind — matches both. Both new tests fail on the parent commit: test hydration_recovers_interrupted_candidate_branch_probe ... FAILED assertion `left == right` failed: hydrate must restore the pre-probe live state left: Group { ... name: "historical anchor" ... } right: Group { ... name: "healthy" ... } test hydration_fails_closed_on_two_interrupted_rewind_probes_of_different_kinds ... FAILED an ambiguous pair of interrupted probes must quarantine the group **`deferred_peel_context_fingerprint` hand-inlined the graph's state filter.** `stored_convergence_commit_digests` restated the match that `record_state_can_contribute_to_openmls_graph` owns — precisely the drift `seed_stored_openmls_graph_inputs` was extracted to prevent, per its own rustdoc. The failure mode is a mis-armed sweep gate with no test failing, so the predicate is now `pub(crate)` and the fingerprint calls it. **`CandidateBranchPeelContext::message_retention_seconds` was dead.** It was populated at three sites, and its only consumer is unreachable for a candidate-branch recovery because the provenance guard returns first. Rather than delete a field that a future caller would re-add, the retention moves onto `PeelRecoverySource::RetainedAnchor`, the only context that has one to offer — so a lineage this device has not adopted can no longer be asked for a retention policy at all. **Two doc claims were false.** Seeding is shared by the pass and by context collection, but they do not "see the identical graph": collection passes `admitted_message_ids: None` and the pass passes its frozen batch, so branch ids match only when both enumerated the same path. And the uncontested path pays the seeding scan, not "nothing" — it pays no *replay*. The contexts also run on a fresh `ReplayBudget` of the pass's shape, not the pass's own budget. **A swallowed enumeration halt was silent.** `MissingOwnCommitCheckpoint` and `ReplayBudgetExceeded` collapse to "no contexts", which reports a contested graph as uncontested — the one way this helper can hide a split-brain. Both now emit a `tracing::debug!` with a low-cardinality reason label and no identifiers. Reasoned about, deliberately unchanged: `MAX_DEFERRED_PEEL_ATTEMPTS = 32`. The fingerprint now moves on every stored commit, so a busy group spends the budget faster, but a row must sit unreadable across 32 distinct commit contexts to exhaust it and has fallen outside `max_rewind_commits` (5) long before that. Recorded on the constant. * fix(cgka-engine): count a contested sweep's evidence for every branch, once Branch-relative peel made a rival branch's traffic readable, but only the rival's. A deferred-peel sweep re-ingested rows one at a time, and each row took a route decided by WHICH CONTEXT had read it: a row readable only under a candidate branch hit the provenance guard and entered the convergence seam, while a row whose live peel now succeeded took the direct apply and was never admitted to any pass. The pass's input is its frozen membership, so the branch this device occupies contributed no app witnesses while the branch it had not adopted contributed all of them. Selection was biased OFF the branch the device held — the split-brain inverted, and the reason `adversarial-reliability/app-witness-value/v1` case 9 still diverged after the parent commit: invariant_failures: [ {"kind": "clients_not_converged", "message": "clients [alice, bob, carol] did not converge"}, {"kind": "client_state_mismatch", "message": "client carol state was epoch 2, members 4, payloads [eve-witness-9, frank-witness-9, david-witness-9], added [eve, frank, david], removed [frank, eve]"}, {"kind": "clients_not_exactly_equivalent"}] alice and bob settled on alice's branch (5 members); carol settled on bob's (4 members) and stayed there. With this commit all three settle on one branch and the convergence invariants clear. An application message is now evidence for the pass whichever context read it, for as long as the sweep has candidate branches in hand. Commits need no such rule: `commit_should_enter_convergence` already decides on epoch, which is provenance-blind. The uncontested path is untouched — `candidate_branch_peel_ contexts` returns nothing unless two branches actually compete, so `DeferredPeelSweep::is_contested` is false and every row keeps the ordinary ingest decision. The second half is that one sweep's rows are ONE evidence set. Each buffered row used to drain the group, so a pass could freeze on a PREFIX of the sweep and settle a verdict the remaining rows would have changed — then terminalize the losers of a decision taken on incomplete evidence: a rival app message peeled at row 1 is classified LosingBranch and persisted `EpochInvalidated`, which `record_state_is_canonicalization_input` never re-admits, and row 2 then makes that rival branch win. Commits survive this through their ConvergenceDeferred parking; application messages had no such protection. A contested sweep now buffers its whole batch and drains once, at the end. That also corrects what one sweep costs. The parent commit priced MAX_CANDIDATE_BRANCH_PEEL_CONTEXTS as "8 x 64 symmetric AEAD attempts on top of 8 replays"; with a drain per recovered row the real worst case was 64 full convergence drains, each up to MAX_CONVERGENCE_REPROCESSING_PASSES. One drain per sweep is what that rationale always claimed. No new test failures. `cargo nextest run -p cgka-engine --features test-policy-overrides` is 461/461. The three `canonical_scenarios` convergence e2e failures in the tree are unchanged by this commit — verified identical at 0984ff95, at the seam-defect commit, and here — and are the expectations acceptance item 2 exists to re-author. * test(cgka-conformance-simulator): demand branch retraction instead of a silent losing branch Branch-relative peel made a rival fork's traffic readable under that branch's own state (e0e5dd73 "read a rival branch's traffic under that branch's state"). Three convergence-E2E oracles were written while that was cryptographically impossible, and each read the new, honest output as failure. Pre-change RED: ---- convergence_e2e_from_peeler_ingest_to_group_events ---- panicked at tests/canonical_scenarios.rs:3131: carol invalidations: [EpochChanged { from: 1, to: 2 }, GroupStateChanged { change: MemberAdded { .. } }, CommitRolledBack { .. }, GroupStateInvalidated { reason: SupersededByBranchSelection }, MessageReceived { .. }, AppMessageInvalidated { reason: LosingBranch, decrypted_payload_ref: Some("sha256:52b9fc4f..") }, ..] ---- scenario_report_records_convergence_e2e_group_events ---- ---- convergence_e2e_delivery_family_runs_generated_variants ---- panicked at tests/canonical_scenarios.rs:3177: assertion `left == right` failed left: ["eve", "david", "grace"] right: ["david", "grace"] assert_canonical_application_event demanded that no AppMessageInvalidated ever appear. That held only while the losing branch's payload was undecryptable: it was dropped in silence because nothing could read it. Now it decrypts and convergence owes the application an explicit retraction. Old obligation -> new: - "no invalidation, ever" -> exactly one, reason LosingBranch - deliveries == [winner] -> unchanged, plus: no delivered message id may appear in any retraction, so a payload shown and then abandoned fails whether or not a retraction follows - EpochChanged 1 -> 2 -> unchanged assert_real_peeler_convergence_trace enumerated two allowed observation shapes. Its second arm accepted a client that ENDED the schedule at epoch 2 with five members, holding only bob's invite and bob's payload and no invalidation - the permanent split brain this branch fixes. A client handed both branches must settle on alice's, which carries two invites to bob's one, so depth decides it. That arm is deleted, not generalized. Old -> new for the surviving arm: - epoch 3, 6 members, [alice payload], epoch_changes [1->2, 2->3] -> all four unchanged - added == ["david","grace"], removed empty -> net diff == ["david","grace"], and every removal must be of a member this client had added. Whether the rival branch is adopted before the reorg is a delivery-order detail; where the client ends up is not. - app_invalidations empty -> exactly ["losing_branch"] The same test's `deferred-losing-transport` tail asserted that the losing branch's transport object stays retained and unpeeled forever, and that NoPendingWork therefore fails for both observers as a steady state - the same blindness arm two encoded. It now pins the opposite and stronger claim: zero stored_transport_deferred_messages, zero scenario_inputs_pending, and zero quiescence failures. The pending_work_remaining reporting path keeps independent coverage in no_pending_work_rejects_delayed_transport_then_accepts_drained_delivery and vector.rs's no_pending_work_requires_an_empty_exact_progress_snapshot. The report test's own `app_invalidation_observations.is_empty()` is arm two's claim at the report layer; it becomes the exact per-client retraction list. Verified: all 47 canonical_scenarios tests, report_runner 23/23, and cgka-engine 464/464 under test-policy-overrides. The delivery family and report spec were each run three times against the re-authored trace oracle (3/3 green) to rule out trading a deterministic red for a flaky one. * fix(cgka-engine): keep a contested sweep contested when branch enumeration halts `DeferredPeelSweep::is_contested` read contested-ness off the captured branch contexts, but `candidate_branch_peel_contexts` reached "no contexts" down five distinct paths and only ONE of them meant the graph was uncontested: 1. `!commits_share_a_source_epoch(...)` — genuinely uncontested; 2. rewinding to the retained anchor hits `SnapshotMissing` — contested; 3. a `MissingOwnCommitCheckpoint` / `ReplayBudgetExceeded` halt — contested; 4. fewer than two candidate paths survived replay — contested; 5. every `candidate_path_peel_context` returned `None`, or the budget broke before the first capture — contested. So the doc comment's "returns nothing unless two branches actually compete, so a non-empty set means ..." held in one direction only: a non-empty set proves a fork, an empty set proves nothing. The one thing this helper must never do is report a fork as healed exactly when this device stopped being able to see it — and reading contested-ness off an empty set did precisely that. The halt is not exotic either: enumeration funds itself with `ReplayBudget::for_pass(...)`, the same budget a pass gets, then pays for up to `MAX_CANDIDATE_BRANCH_PEEL_CONTEXTS` further whole-path replays the pass never pays, so it is structurally more likely to exhaust than the pass that follows it. Contested-ness now travels in its own field on a new `CandidateBranchPeel`, sourced from the shared-source-epoch check that already runs before any replay. Every return after that check is wrapped by `CandidateBranchPeel::contested_over`, so a halt can only ever lose the contexts, never the split. The two meanings stay split at the consumer, because they answer different questions: - `is_contested()` gates the DRAIN POLICY only. A contested sweep's recovered rows are one evidence set, so the drain waits for the whole batch instead of freezing a pass on a prefix of it. The end-of-sweep drain keys on the same predicate as `drain_policy`, so no row is ever deferred to a drain that does not run. - evidence routing keys on the new `has_branch_contexts()`. Routing live-readable application traffic into a pass whose branch enumeration just halted feeds evidence to an adjudicator that likely cannot read the rival branch either — same budget, same missing checkpoint. That restores no symmetry, it only adds noise. `recovered_from_candidate_branch` semantics are unchanged: it still tracks whether THIS row was read under an unadopted lineage. Cost asymmetry: a false positive on `contested` (a fork present, no surviving paths) costs one deferred drain instead of N immediate ones — strictly the safer direction, and bounded, since a resolved fork drops its loser out of the stored graph and the check goes quiet again. RED, from the new unit tests against the pre-fix single-field sweep: error[E0432]: unresolved import `crate::openmls_projection::CandidateBranchPeel` error[E0599]: no method named `has_branch_contexts` found for struct `DeferredPeelSweep<'a>` in the current scope error[E0599]: no method named `has_branch_contexts` found for struct `DeferredPeelSweep<'a>` in the current scope error: could not compile `cgka-engine` (lib test) due to 3 previous errors The old shape could not express "contested with nothing to read" at all, which is the defect stated as a type. Verified: `cargo nextest run -p cgka-engine --features test-policy-overrides` 466 passed / 0 failed (464 + 2 new); `just fast-ci` exit 0. * test(cgka-engine): drive both enumeration halts and pin the contested verdict The tests that shipped with the halt fix build `CandidateBranchPeel` by hand, so they pin only the CONSUMER: that `contested` drives the drain and `contexts` drives evidence routing. Nothing exercised the PRODUCER. A `candidate_branch_peel` regressed to report `contested: false` whenever a halt lost the contexts -- exactly the regression the fix exists to prevent -- left every one of them green. Drive the helper into both halts it names, over a real forked graph of real OpenMLS commits, and assert on its return value: - `MissingOwnCommitCheckpoint`: a two-way epoch-1 fork where this device committed one branch, then its post-merge checkpoint is released. MLS cannot replay an own path-bearing commit from the public wire echo, so enumeration loses the branch it is standing on. - `ReplayBudgetExceeded`: a same-epoch fork ground WIDTH ways, each rival a frontier path, plus DEPTH successors at the next epoch that enumeration must probe against every one of those paths. The budget is linear in the commit count, the probes are its product. A compile-time assertion holds the arithmetic so a change to either budget constant lands there rather than as a mystifying empty-context failure. Each test first runs the same graph without the halt and asserts it enumerates more than one context, so an empty set proves the halt rather than a graph that never had two branches to offer. The helper is `pub(crate)` and the contexts it returns carry derived exporter material, so this is an in-crate test module rather than a widened public surface. RED, with `contested_over` regressed to the pre-fix `contested: !contexts.is_empty()`: thread 'openmls_projection::candidate_branch_peel_halt_tests::a_lost_own_commit_checkpoint_halts_enumeration_and_keeps_the_fork' panicked at crates/cgka-engine/src/openmls_projection.rs:4513:9: the fork is in the stored graph, not in what enumeration managed to read thread 'openmls_projection::candidate_branch_peel_halt_tests::an_exhausted_replay_budget_halts_enumeration_and_keeps_the_fork' panicked at crates/cgka-engine/src/openmls_projection.rs:4573:9: the fork is in the stored graph, not in what enumeration managed to read test result: FAILED. 0 passed; 2 failed; 0 ignored; 0 measured; 88 filtered out The hand-built consumer tests stayed green through that same regression: test message_processor::ingest::tests::a_halted_enumeration_leaves_the_sweep_contested_with_nothing_to_read ... ok test message_processor::ingest::tests::an_uncontested_graph_drains_per_row ... ok test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 88 filtered out They keep their place -- the split they pin is a distinct property -- with a note pointing at where the producer is now covered. * test(cgka-engine): acknowledge the fixture's join event so a reopen cannot replay it `convergence_pass_that_keeps_the_own_commit_emits_no_own_withdrawal` and `convergence_pass_that_replaces_the_own_commit_withdraws_it_exactly_once` began failing on "buffering the rival must not be application-visible" after this branch was rebased onto master 85ef573b. What leaked, captured by binding the drained vector into the assertion message: [GroupJoined { group_id: .., via_welcome: .., welcomer: Some(..) }] Proven cause, and it is not the buffering. `a82524ca` (#1391) made `do_join_welcome` write its `GroupEvent::GroupJoined` to the durable pending-application-event outbox added by `01c09a9e` (#1366). That outbox is at-least-once: `drain_events` hands the event to the host but does not retire the row; only the host's `delete_pending_application_events` ack does, and `EngineBuilder::build` re-seeds `events_buf` from `list_pending_application_events` on every open. `router_flip_fixture_ arranged_with_creator` never acked the joiner's `GroupJoined`, so the `reopen_legacy_client` these two tests perform correctly re-delivered it, and it was sitting in the buffer before `ingest` was ever called. Both halves of that mechanism predate the rebase individually; only their combination -- the `GroupJoined` put landing on top of the durable outbox -- is new, which is why the tests passed at `01c09a9e` and at the pre-rebase tip `eeebcb2c`. The engine is right on every step here, so nothing in `src/` changes and the assertion keeps its exact strength: the arrangement was measuring a replayed event it had never consumed. The fixture now does what a host does -- takes the join notification and acks it -- mirroring the pattern already established at `tests/ingest.rs`'s welcome/reopen sites. A spurious event emitted by the ingest under test still fails the assertion. For a future bisector: the hand-resolved conflict in `message_processor/ingest.rs` during that rebase is exonerated. The line-level delta `eeebcb2c..582af852` on that file equals the delta `01c09a9e..85ef573b` exactly, modulo the four-space re-indent required because this branch removed the `loop` that wrapped the `MlsGroup::load` arm. Both of master's changes (#1379's `retryable_unpersisted_ingest_id` and #1367's `Option` refactor) survived intact and nothing else moved. 469/469 `cargo nextest run -p cgka-engine --features test-policy-overrides`. * test(cgka-conformance-simulator): state the losing-branch withdrawal as a rule The app-runtime cross-route characterization enumerated two participant-local invalidated-row shapes and rejected everything else. Observed RED, ~5 runs in 12 on this branch: ---- four_party_cross_route_recovery_characterizes_corrected_app_runtime_outcomes ---- panicked at tests/process_orchestrator.rs:1097: unexpected invalidated disposition shape: { "alpha": 1, "observer": 1, "yankee": 0, "zeta": 0, } What Observer withdraws, dumped from the app projection in two independent failing runs (10 and 11): client=alpha kind=1210 invalidated=true sender=<alpha> epoch=4 {"system_type":"group_renamed","data":{"actor":"<alpha>", "name":"alpha-root","old_name":"cross-route"}} client=observer kind=1210 invalidated=true sender="" epoch=4 {"system_type":"group_renamed","data":{ "name":"alpha-root","old_name":"cross-route"}} Both runs ended with all four participants at epoch 5 on `zeta-branch-depth-two`, and Yankee and Zeta held no invalidated row. Mechanism, not the peel. Nothing on the losing branch was decrypted: the row is a locally synthesized kind-1210 system row, stamped with `alpha-root`'s `origin_commit_id`, that `GroupStateInvalidated` tombstones when `emit_rolled_back_commits` withdraws the superseded commit. Observer's copy is unattributed because `emit_convergence_events` reaches the branch by replaying stored commits and emits profile deltas with `actor: None`, so it is not even the same canonical id as Alpha's self-attributed row. Observer therefore applied Alpha's commit through stored convergence and was later reorged off it — the #363 anti-lie contract firing on a participant that is not the losing committer. The test originated on master (#1395 / #1398) describing behavior before this branch unified same-epoch fork resolution through distributed convergence, when the observed shapes were the enumerated two. Enumerating shapes was the defect: which participants transiently hold the losing branch is a delivery-order detail, so the assertion now states the rule instead. Old -> new: - "all zero, or Alpha alone at one" -> Alpha withdraws its own confirmed alpha-root row EXACTLY when it ends off that branch. All-zero is no longer an unconditional escape; a converged Alpha silently keeping the row now fails, which is the shape the old assertion accepted. - Yankee and Zeta at zero -> unchanged, now with its reason: Zeta is the rival same-epoch committer and defers only to a deeper branch, and Yankee is two commits deep on the selected branch before `step-25` reaches it, so neither ever synthesizes the row. - Observer forced to zero -> at most one, and only after leaving the losing branch and advancing past the pre-fork baseline. This is the single relaxed clause, and it is paid for by the mandatory Alpha withdrawal above. - (new) no participant may show a withdrawn row as delivered, mirroring the retraction oracle in canonical_scenarios.rs. Every other assertion in the test is untouched, and it stays a characterization of a counterexample: the terminal-outcome arms below it still accept the two-branch splits. Mutation-checked the added Alpha clause by inverting its predicate; it goes RED on the real run, so the clause is live rather than vacuous. `cross-route-app-runtime-recovery/v1` has no JSON vector, and the sibling `cross-route-own-commit-recovery/v1` fixture expresses convergence decisions and group profiles rather than invalidated rows, so neither encodes the enumerated shape. The other `invalidated_message_ids.is_empty()` assertions in this file and in app_runtime_adapter.rs are on fork-free restart/offline scenarios. Verified: the named test 6/6 green (5/12 red before), process_orchestrator 11/11, route_assurance 8/8, cgka-engine 469/469 under test-policy-overrides, `just fast-ci` exit 0. * fix(cgka-engine): rank candidate branches before capping peel contexts `candidate_branch_peel_contexts_from_current` took the first `MAX_CANDIDATE_BRANCH_PEEL_CONTEXTS` entries of the BFS's completed-path vector. That vector is built in level order, so the prefix is the SHALLOWEST branches: eight one-commit rivals from a single member evict every branch two or more commits deep — and a deep branch is exactly the one carrying the post-fork traffic a retained row needs unsealed. Posting eight junk commits at one epoch is the whole attack. Candidates are now ordered by tip epoch descending, then by path-digest branch id, before the cap applies. `tip_epoch` is joined off `path_result.materialized` by `branch_id`, not by index: the two vectors are parallel by construction, but `materialized` is documented as allowed to come back short, and an index skew would silently rank one branch by another branch's depth. Both keys are content-derived, so peers holding the same evidence keep the same subset — the property that makes capping safe at all. Branch selection is untouched and stays uncapped. RED, before the ranking landed: thread '...a_deeper_branch_outranks_shallow_rivals_for_the_capped_contexts' panicked at crates/cgka-engine/src/openmls_projection.rs:4670:9: assertion `left == right` failed: a branch carried two commits deep holds traffic the one-commit rivals cannot unseal, so filling the cap with rivals must not evict it left: Some(2) right: Some(3) Two tests over a ten-branch fork (nine one-commit rivals plus one two-commit branch, one more branch than the cap): the cap is filled exactly and two peers holding the same evidence keep the same branch ids in the same order; and the deep branch survives the cap. The existing fixture only asserted `contexts.len() > 1`, so neither the truncation nor its cross-peer uniformity was pinned by anything. * docs(cgka-engine): correct why the candidate branch peel cap is eight The constant justified its value two ways, and both were false. "v1 `max_rewind_commits` is 5" bounds a path's LENGTH. This cap bounds the path COUNT, which is set by branching factor — and nothing bounds branching factor. "a graph wide enough to need more than eight branches is already failing the pass's own `ReplayBudget`" is arithmetically wrong. `ReplayBudget::for_pass` is `commits × (max_rewind + 1) × 4 + 32`, so at v1 it is `commits × 24 + 32`: a flat nine-way fork is nine commits, a budget of 248, and about nine probes — nowhere near exhaustion. This crate's own budget-exhaustion fixture has to stack WIDTH=10 with DEPTH=13 AND max_rewind_commits=0 to trip it. Width alone never gets there. State the real reason instead: candidate-path count is attacker-inflatable at one message per extra branch, each context costs a whole branch replay plus up to `MAX_DEFERRED_ROWS_PER_SWEEP` AEAD attempts, and the kept subset is content-derived — ranked by tip epoch then branch id — so every peer with the same evidence caps identically. Uniform, not divergent.
1 parent 85ef573 commit 99f4144

83 files changed

Lines changed: 6257 additions & 4337 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

Cargo.lock

Lines changed: 1 addition & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

crates/cgka-conformance-simulator/AGENTS.md

Lines changed: 10 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -57,8 +57,8 @@ the property-test registry. This file is the agent-facing model.
5757

5858
- **Module:** `src/family.rs`
5959
- **Role:** Deterministic generated scenario families. `generate_send_leave_family`,
60-
`generate_convergence_e2e_delivery_family`, `generate_convergence_chaos_family`, and
61-
`generate_admin_churn_family` record family name, generator
60+
`generate_convergence_e2e_delivery_family`, `generate_convergence_chaos_family`,
61+
`generate_admin_churn_family`, and `generate_bounded_convergence_pressure_family` record family name, generator
6262
version, seed, case index, runnable `ScenarioSpec`, and optional semantic expectations. `run_generated_case_report`
6363
adds generated metadata to report artifacts. Reliability families append a final global drain, exact canonical
6464
observation, and pending-work assertion; do not remove a red strict result merely because an earlier legacy
@@ -146,7 +146,7 @@ the property-test registry. This file is the agent-facing model.
146146
- **Role:** `ScenarioTrace`, observations, and semantic `TraceExpectation` checks. Records final epoch/member/payload
147147
facts plus member additions/removals, client convergence, epoch changes, app invalidations, exact canonical state,
148148
commit/proposal/application input dispositions, pending-work blockers, active decryptability matrices, and
149-
`ForkRecoveryObservation` entries from `GroupEvent::ForkRecovered`.
149+
settled `ConvergenceDecisionObservation` entries captured from the forensic recorder.
150150

151151
- **Module:** `src/policy_cases.rs`
152152
- **Role:** Serializable `PolicyCase` DTOs plus selection-reasoning helpers (`parse_policy_cases`, `reason_against`,
@@ -242,6 +242,12 @@ shapes (arms 2, 6, 7, 8, 9) from the seed, so distinct seeds exercise distinct a
242242
schedule-invariant convergence, rollback, and payload-set expectations stay fixed, so coverage grows with the seed
243243
without re-pinning vectors.
244244

245+
The `bounded-convergence-pressure/v1` generated family is the finite-pressure acceptance campaign for the unified
246+
fork-resolution route: a same-epoch commit race, application sends issued inside the quiescence window, a committer
247+
restart taken mid-resolution, and a bounded self-update/profile/admin tail. It activates controlled virtual time before
248+
the race, so every later settle must be `AwaitQuiescence`; the driver's watchdog budget is the bounded-time assertion.
249+
The campaign is finite by construction and claims nothing about progress under unbounded self-updates.
250+
245251
## How to add a new scripted scenario
246252

247253
1. Prefer a `ScenarioSpec` when the case should become portable or reportable.
@@ -262,7 +268,7 @@ Look at `three_client_happy_path_via_harness` for the canonical shape.
262268
2. Include `scenario_name`, `vector_version`, `conformance_version`, `seed`, `scenario`, and either `expected_trace` or
263269
`expected_outcomes`.
264270
3. Keep `ScenarioTrace` free of MLS bytes and Rust-only internals.
265-
4. Make recovery behavior observable through `ForkRecoveryObservation`, not just final membership.
271+
4. Make fork-resolution behavior observable through a settled `convergence_decision` expectation, not just final membership.
266272
5. Run `cargo test -p cgka-conformance-simulator canonical_vector_fixtures_match_generated_traces`.
267273

268274
## OpenMLS replay probes

crates/cgka-conformance-simulator/CONVERGENCE_ROUTE_MATRIX.md

Lines changed: 30 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -14,8 +14,7 @@ below is deliberately not treated as passing evidence.
1414
| Route id | Production owner | Independent model | Mutation sentinel | Current campaign evidence | Current limitation |
1515
| --- | --- | --- | --- | --- | --- |
1616
| `ordinary_ingest` | `message_processor/ingest.rs` | partial: `reference_convergence::evaluate` | covered: cutoff admission and output invalidation | observer vectors, both strict cross-route regressions, the shared engine/app/process restart journey, and the app-runtime cross-route characterization | Corrected reversible relay input can still leave a two-branch public/probe split; process/distributed execution and exact app-runtime state remain open. |
17-
| `pairwise_fork_recovery` | `message_processor/ingest.rs`, `fork_recovery.rs` | partial: bounded abstract route lifecycle | partial: sibling-model pairwise-loser terminalization | both strict cross-route regressions, app-runtime cross-route characterization, and pairwise reconsideration tests | The production-shaped app route does not reliably select the strict reference's winning branch; process/distributed and durable-transition permutations remain open. |
18-
| `stored_convergence` | `distributed_convergence.rs` | partial: selector plus lifecycle models | covered: comparator and frozen membership | both strict cross-route regressions, app-runtime cross-route characterization, restart properties, and convergence chaos | Route choice is not represented in the models, and app-runtime/process/distributed equivalence remains open. |
17+
| `stored_convergence` | `distributed_convergence.rs` | partial: selector plus lifecycle models | covered: comparator and frozen membership | both strict cross-route regressions, app-runtime cross-route characterization, restart properties, convergence chaos, and the two reconsideration regressions inherited from the retired pairwise route | Route choice is not represented in the models, and app-runtime/process/distributed equivalence remains open. |
1918
| `candidate_materialization` | `openmls_projection.rs` | partial: dependency/disposition model | covered: retention and frozen membership | engine and retained-history cross-route recovery plus OpenMLS replay | Exact cryptographic and durable-disposition evidence remains limited to engine-capable subjects; app-runtime/process/distributed equivalence remains open. |
2019
| `retained_history_replay` | `message_processor/ingest.rs`, `openmls_projection.rs` | partial: unequal-history lifecycle | covered: retention and witness deduplication | `cross-route-retained-history-recovery/v1`, `cross-route-app-runtime-recovery/v1`, retained-relay equality, offline journeys, and shared app/process offline full-history recovery | Reversible relay-wide app-runtime history removes the tombstone artifact but does not reliably produce public or active-chat equivalence; process, multi-relay, retention-boundary, and distributed evidence remain open. |
2120
| `crash_restart_recovery` | `engine.rs`, `distributed_convergence.rs` | covered: crash/restart lifecycle | covered: frozen membership and scheduler re-arm | both strict cross-route encrypted-SQLite regressions, the app-runtime cross-route restart, durable-phase kill properties, and the shared engine/app/process restart journey | The adversarial app-runtime topology covers one post-displacement restart; every durable transition plus process/distributed adapters remain open. |
@@ -46,6 +45,35 @@ still produced both complete equivalence and a two-branch protocol/probe split.
4645
The corrected-input characterization accepts only reviewed terminal surfaces;
4746
it is counterexample evidence, not passing assurance evidence.
4847

48+
Its losing-branch surface is stated as a rule rather than an enumeration of
49+
observed shapes, because which participants transiently hold the losing branch
50+
is a delivery-order detail. Every participant that applies the losing commit
51+
synthesizes that commit's kind-1210 rename row and owes it a withdrawal on
52+
leaving the branch — the losing committer unconditionally, an observer that
53+
adopted the branch before the selected one displaced it, and no one else.
54+
55+
## Retired routes
56+
57+
A deleted route is removed from the table and from `DecisionRouteId` rather
58+
than re-pointed at a surviving seam. Re-pointing would give one seam two route
59+
claims and count its evidence twice, which is exactly the inflation the source
60+
audit exists to prevent. The audit failing on a vanished marker is the intended
61+
trigger for that review, not an obstacle to it.
62+
63+
Retired by MDK #1293 (option C route unification): `pairwise_fork_recovery`,
64+
whose owners were `message_processor/ingest.rs` and `fork_recovery.rs`. The
65+
pairwise same-epoch route was deleted — `fork_recovery.rs` is gone and ingest no
66+
longer calls `.resolve_fork_candidate(` — and same-epoch rivals are now
67+
adjudicated by distributed convergence, which is already registered as
68+
`stored_convergence`. The route's two cited properties survived the unification
69+
as renamed engine regressions,
70+
`incumbent_committer_defers_to_deeper_convergence_branch` and
71+
`rival_win_leaves_displaced_own_commit_reconsiderable`, and are now cited as
72+
campaign evidence for `stored_convergence`; its reconsiderable-loser rule is
73+
folded into that route's adopted rule. The abstract route-lifecycle mutant it
74+
sponsored survives route-agnostically as `provisional_winner_terminalization`
75+
in the mutation catalog.
76+
4977
## Active production work
5078

5179
MDK #1329 has merged the fail-closed missing-anchor behavior and Welcome-repair

crates/cgka-conformance-simulator/MUTATION_MATRIX.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -16,4 +16,4 @@ observation and keeps this table exactly aligned with the executable catalog.
1616
| `publication_acknowledgement` | Accepted publication remains pending | Real `ReferenceModelSubject` acknowledgement plus structural pending-work observation | Lost/duplicate acknowledgement and no-pending-work checks |
1717
| `retained_history_expiration_boundary` | Exact retention boundary expires early | Shared reference evaluation plus canonicalizer boundary | Past-epoch, proposal-expiry, and retained-relay boundary tests |
1818
| `group_profile_projection` | Accepted profile commit advances while the public projection remains stale | Real engine-vector execution plus portable expected-value group-profile oracle | Engine vector `group-profile-update/v1`; separate app-runtime and process profile smoke tests cover those adapters |
19-
| `pairwise_losing_branch_terminalization` | Pairwise routing terminalizes its provisional winner instead of reconsidering the complete durable branch set | Bounded abstract route-lifecycle sentinel; not a production differential | Reconsiderable-loser engine regressions and the pending cross-route #1236 campaign |
19+
| `provisional_winner_terminalization` | A volatile route terminalizes its provisional winner instead of reconsidering the complete durable branch set | Bounded abstract route-lifecycle sentinel; not a production differential | Reconsiderable-loser engine regressions `incumbent_committer_defers_to_deeper_convergence_branch` and `rival_win_leaves_displaced_own_commit_reconsiderable`, plus the landed cross-route recovery campaigns |

crates/cgka-conformance-simulator/README.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -72,7 +72,7 @@ welcomes use NIP-59 gift wraps before the bus delivers them.
7272
action kinds are drift-checked against committed Scenario IR.
7373
- `mutation_adequacy` — eleven simulator-only single-rule mutants with minimal deterministic witnesses. The generated
7474
catalog kills selector order, witness dedup/admission, cutoff, frozen-state, scheduler re-arm, invalidation,
75-
publication-ack, retention-boundary, group-profile-projection, and pairwise-loser terminalization mutations without
75+
publication-ack, retention-boundary, group-profile-projection, and provisional-winner terminalization mutations without
7676
compiling mutation switches into production. Lifecycle
7777
mutants run through the shared Rust transition model; publication acknowledgement runs through a real independent
7878
reference subject and its structural pending-work observation; the profile mutant runs the production-shaped engine

0 commit comments

Comments
 (0)