Commit 99f4144
authored
Unify same-epoch fork resolution through distributed convergence (#1293)
* test(cgka-engine): pin the pre-unification fork-resolution routes
Characterization evidence for Slice C (deleting the pairwise same-epoch
fork-resolution route). Before the route disappears, pin what it does and
prove the surviving convergence route already agrees with it.
Rule agreement: the two fork-resolution rules agree eventually
THE #1285 claim (option-c plan, N5), previously untested: a committer whose
OWN commit won the pairwise race — rival root parked ConvergenceDeferred,
never applied — must still reach the terminal state every
convergence-resolved device reaches once valid depth disagrees with the
ordering key.
The test drives both routes side by side on one epoch-1 UpdateGroupData
fork: A resolves pairwise (own commit wins, rival parked at its source
epoch), an observer control resolves the identical fork through
convergence and first agrees with A (ordering key at depth 1), the rival
branch then grows two follow-on commits deeper, and both devices run their
convergence passes. Verdict, pinned GREEN: A's pass re-admits the PARKED
pairwise loser as branch input, reorgs A onto the fleet's deeper branch,
and A, the observer, and the rival author agree on members, epoch, and
group name; the parked root becomes Processed.
This is the go/no-go evidence that deleting the pairwise route (Slice C)
is deduplication of an already-agreeing rule, not a behavior change.
Event fidelity: the exact event set of a convergence apply over an own commit
U4 event fidelity on the post-#1285 base, both arms of the stored-convergence
seam, driven through the deterministic router-flip fixture (now parameterized
so either the inbound sibling or the device's own commit wins the ordering):
- keep-own: the settling pass emits exactly the losing sibling's withdrawal
pair (CommitRolledBack + GroupStateInvalidated at the source epoch) and
never names the kept own commit. The pre-#1285 hazard — no checkpoint meant
the kept own commit had no disposition and the superseded-processed sweep
spuriously invalidated it — is closed by #1285's checkpoint
materialization; this pins the clean behavior.
- replace-own: exactly the two unattributed membership deltas at the selected
tip plus one withdrawal pair naming the own commit by its stored id at its
source epoch; the displaced own commit is parked ConvergenceDeferred at its
source epoch, agreeing with the pairwise route's #1285 parking.
This is the application-visible event contract the pairwise-route deletion
(Slice C) must preserve.
* test(cgka-conformance-simulator): pin route equivalence across the fork-resolution routes
One same-epoch UpdateGroupData fork, resolved by every route that exists
today in a single portable vector: the live committer's pairwise
fork-recovery seam (alice, the deterministic ordering loser), post-restart
convergence over hydrated storage (bob, the ordering winner, restarted
while both commits are withheld so his in-memory committed_from guard is
gone), and the observer convergence selector (carol, dave).
The report trace confirms each mechanism empirically: alice records the
single ForkRecoveryObservation and no convergence decision; bob and both
observers each record a two-candidate convergence decision selecting the
identical branch. Expected outcomes stay semantic (converged epoch/count,
exact canonical equivalence, no pending work, one recovery) — no
selected_branch_id or trace-byte pins, and the strict reliability tail
mirrors add_strict_reliability_oracle.
This is the Slice B acceptance pin the pairwise-route deletion (Slice C)
must preserve unchanged.
The vector is authored at conformance_version 0.9.11, the released version
master's release prep (388a1d87) moved the tree to;
canonical_vector_fixtures_match_generated_traces enforces the match and
every other fixture already carries 0.9.11.
* feat(cgka-engine)!: unify same-epoch fork resolution through distributed convergence
Two routes existed for the same input: a committer that advanced from the
fork's source epoch resolved a same-epoch rival PAIRWISE (the
we_committed_from conjuncts in ingest.rs routing into
ForkRecoveryManager::resolve), while every other member — and the same
committer after a restart — adjudicated the identical rival through
distributed convergence. Slice B proved the two rules agree eventually
(committers_losing_rival_is_reconsidered_onto_the_fleets_deeper_branch),
pass, and the restarted-committer pin already drove the convergence route
end to end. The pairwise route was therefore a duplicate depth-blind
fast-path over machinery that handles the general case. Delete it:
- routing: drop the we_committed_from conjunct from the convergence
admission gate (a committer's rival is admitted by the same retained
source-epoch anchor as an observer's) and delete the WrongEpoch arm's
pairwise block, its ingest retry loop, and the direct-seam
ForkRecovered/GroupStateInvalidated emission pair. Slice A's loud
fail-closed halt now also covers a live committer whose anchor is gone.
- orphaned machinery, each grep-verified: ForkRecoveryManager (resolve,
parking, incumbents, ForkResolution, the fork_resolution audit
emission), the probe (probe_commit_ordering_metadata_for_recovery /
ForkProbeError), forked_epoch_fail_closed (nothing in the engine
produces EngineError::ForkedEpoch any more), the send/apply-time
fork-{epoch}-{n}-{hash} snapshot captures, the fork- hydration scan
(newest_fork_snapshot), DurableGroupEvolution.recovery_snapshot
(serde-JSON rows tolerate the removed field), the snapshot half of
PendingCommitCleanupGuard, and the whole committed_from surface in
EpochManager (map, we_committed_from, owns_committed_from rollback
discipline, prune) plus EpochManager::detect_fork, whose only caller
was the deleted fail-closed helper.
- the pending-publish origin-commit id — the one job the manager still
did for confirm/fail bookkeeping — moves to a plain
Engine::pending_origin_commits map (track/peek/take).
Peel fallback keeps its coverage without the fork- snapshots: every seam
that advances past an epoch retains a durable openmls-retained-anchor-N
at that epoch first (confirm pre-merge, the inbound apply, join, the
convergence apply), and both snapshot families were pruned by the same
max_rewind_commits window, so available_past_peel_snapshots now reads
the anchors alone.
New raw-wrapper discipline replaces the pairwise-era retirement: once a
peeled message is durably buffered as convergence input, its raw
transport wrapper is retired Processed
(mark_raw_transport_message_processed_if_awaiting_retry) instead of
being reset Retryable and re-peeled on every later publish-cycle replay;
replay_buffered_messages now respects verdicts ingest committed during
the call.
Known cost, deliberate: a live committer's rival resolution now waits
for the convergence pass (P4 quiescence, ~1.1 s measured pre-#1285);
sends queue during it. No expedite is added here.
Test dispositions (intent preserved unless the subject was the deleted
seam):
- retargeted to the convergence route, outcomes unchanged:
concurrent_invites_recover_to_deterministic_winner,
strict_cutover_legacy_add_cannot_displace_valid_fork_incumbent (the
materialization-time strict-cutover gate now does the rejecting; the
forbidden candidate retires EpochInvalidated),
convergence_privileged_remove_beats_grinding_ordinary_self_update,
buffered_losing_fork_commit_raw_row_is_retired_after_confirm_replay,
the two inbound-apply fault tests in auto_commit_atomicity (the fault
now fires inside the settling pass's apply), and the concurrent-rename
family in update_group_data (the winner's pass may emit a no-op
invalidation naming the never-applied sibling; the pinned invariant is
that it never withdraws its own origin).
- renamed, pairwise vocabulary retired (former names in doc comments):
pairwise_incumbent_defers_to_deeper_convergence_branch →
incumbent_committer_defers_to_deeper_convergence_branch,
pairwise_candidate_win_leaves_old_incumbent_reconsiderable →
rival_win_leaves_displaced_own_commit_reconsiderable,
pairwise_loser_is_reconsidered_onto_the_fleets_deeper_branch →
committers_losing_rival_is_reconsidered_onto_the_fleets_deeper_branch,
restarted_committer_routes_same_epoch_sibling_into_convergence
(drops the _not_fork_recovery suffix),
buffered_self_evicted_row_stays_failed_not_swept_processed_on_replay →
buffered_losing_rival_content_row_is_never_swept_canonical_on_replay
(the SelfEvicted-at-replay shape is unreachable once the rival buffers
before eviction settles; the guarded invariant — a losing rival's
content row is never relabeled canonical — survives).
- deleted with their subject:
invite_wrap_failure_releases_pre_commit_recovery_snapshot (no snapshot
exists to leak; the sibling staged-commit-cleanup test keeps the guard
covered), the two committed_from rollback-discipline tests in
epoch_manager (the map is gone), newest_fork_snapshot unit tests.
- repurposed: confirmed_commits_prune_fork_recovery_snapshots_to_rewind_horizon →
confirmed_commits_prune_retained_anchor_snapshots_to_rewind_horizon,
pinning the boundedness of the snapshot family that now carries the
admission role.
GroupEvent::ForkRecovered stops being emitted here; the variant itself
and its simulator/app/FFI surface are removed in the follow-up commit.
Forensics read-compat is untouched: AuditEventKind::ForkResolution and
incident-replay keep parsing historical exports.
Rebase and formatting fixups folded in:
- fork_detection.rs:1121 kept the pre-#1377 hardcoded EpochId(3) after this
resolution moved the test past the pinned rewind horizon (terminal_epoch =
V1_MAX_REWIND_COMMITS + 1). Lines 1091/1112 already used terminal_epoch;
only the trailing epoch assert was left stale.
- router_flip_fixture_arranged picked up a call that exceeds the line budget,
so cargo fmt --check failed the fast-ci gate. Formatting only; no test logic
changed.
* feat(traits)!: retire GroupEvent::ForkRecovered and migrate every consumer
Nothing emits ForkRecovered since the pairwise fork-resolution route was
deleted — the convergence route's rollback pair (CommitRolledBack +
GroupStateInvalidated) is the only commit-rollback signal. Pre-production
stance: delete the variant end to end rather than carry dead vocabulary.
- traits: drop the GroupEvent::ForkRecovered variant (doc references on
CommitRolledBack / GroupStateInvalidated rewritten), its snapshot test
and event_fork_recovered.snap.
- marmot-app / marmot-uniffi: remove the match arms and the FFI
GroupEventKindFfi::ForkRecovered variant; the app's tombstoning already
keyed exclusively on GroupStateInvalidated.
- simulator: remove ForkRecoveryObservation / RecoveryOrderingKeyObservation,
ClientObservation.recoveries, the fork_recovered event counter,
TraceExpectation::{ClientRecoveries, RecoverySummary} and their
comparators, OracleBehavior::ForkRecovered (with its evidence counter,
expectation mapping, and recommended-behavior entries), the
ScenarioReport.recovery_observations projection and the campaign
recovery_observation_count metric — with no seam able to produce a
recovery observation, every one of these expectations became
unobservable (strict-oracle trap: a vector carrying one could only
fail).
- incident-replay synth: the two fork-incident vector shapes assert a
settled convergence_decision at the post-fork tip (epoch 2) instead of
a recovery summary — resolution evidence, not just final-state
equality (Rule 4 intent preserved). Historical-export PARSING is
untouched: EventKind::ForkResolution and the classifier still read old
JSONL, and marmot-forensics keeps AuditEventKind::ForkResolution with a
doc note that only pre-unification versions emit it.
- vectors re-derived against the unified engine (inputs unchanged,
expectations only): route-equivalence.v1 (the Slice B acceptance pin,
flipped: committer, observer, and restarted committer all resolve
through convergence; recovery_summary count:1 → settled
convergence_decision at tip 2), group-data-fork-recovery.v1,
concurrent-invite-fork-recovery.v1, and both incident vectors get the
same recovery_summary → convergence_decision substitution;
publish-fail.v1 and three-client-message-exchange.v1 drop the
fork_recovered:0 counters and empty recoveries arrays from their
expected traces. Manifest coverage/next notes and SCENARIOS.md updated
to the unified-route wording.
- convergence-chaos family cases drop their recovery_summary
expectations; the converged-equality and client-state oracles remain
the acceptance surface. deliberate-fork-recovery harness tests keep
their outcome assertions and lose only the recovery-observation
bookkeeping (scenario_report_records_trace_log_recoveries_and_failures
→ scenario_report_records_trace_log_and_failures).
Consumer migrations folded in, each one a fixture or claim that the
retired vocabulary left stranded.
Cross-route recovery re-authored over post-unification observables. Both
cross-route fixtures arrived from master carrying `client_recoveries` and
`recovery_summary`. They no longer parse (`unknown variant
'client_recoveries'`), which failed three tests outright rather than
weakening them. Equivalence of what is checked, old -> new:
- `recovery_summary { count: 1, recovered_epoch: 2 }` — the group-level
claim that every client agreed on one recovery -> four per-client
`convergence_decision` pins at `selected_tip_epoch: 3`. Per-client, not
client-less: a client-less pin is satisfied by any single observer, which
is the opposite of what the vector claims. Tip 3 is the settled decision;
epoch 2 was the intermediate the deeper branch supersedes.
- `client_recoveries { client: alpha, source_epoch: 1, recovered_epoch: 2 }`
-> `client_epoch_changes` for alpha, exactly `[1->2, 2->3]`. This is an
exact-list comparison, so it pins the same two scalars positionally:
alpha leaves the epoch-1 fork source and lands on epoch 2. The trailing
2->3 is strictly additional.
- `winner_differs_from_invalidated: true` -> per-client `group_profile`
pins at `zeta-branch-depth-two`. Alpha's own displaced commit named the
group `alpha-root`, so alpha projecting the winner's name is that claim
stated per client. `selected_branch_id` stays unpinned by house rule: it
digests randomized MLS bytes and changes every run.
- the withdrawal itself -> `assert_own_commit_was_withdrawn` requires
alpha's own root to carry both `commit_rolled_back` and
`group_state_invalidated_superseded`, the ledger's handles for the
`CommitRolledBack` + `GroupStateInvalidated` pair the engine emits
together. The pre-existing all-client `Invalidated` disposition check
stays; `Invalidated` alone is reachable from other transitions, so it was
never the withdrawal claim on its own.
- end-state equality keeps the existing `clients_exactly_equivalent`.
`conformance_version` and `vector_version` are untouched; that schema
change is additive within the existing shape.
The pairwise fork-recovery route claim is retired. `fork_recovery.rs` is
gone and ingest no longer calls `.resolve_fork_candidate(`, so the route
inventory's source-marker audit failed on both of its cited production
sites. Retire the claim instead of re-pointing it at the unified
convergence site: that site is already registered as `stored_convergence`,
so re-pointing would give one seam two route claims and count its evidence
twice — the exact inflation the audit exists to catch. The inventory has no
tombstone record, and its drift tests require registered routes and matrix
rows to match one-to-one, so retirement is removal from `DecisionRouteId`,
the inventory, and the table, with a prose retired-routes record in the
matrix that says what happened, why it was not re-pointed, and where the
evidence went. The two properties the route carried survived the
unification as renamed engine regressions,
`incumbent_committer_defers_to_deeper_convergence_branch` and
`rival_win_leaves_displaced_own_commit_reconsiderable`; they are now
campaign evidence for `stored_convergence`, whose adopted rule absorbs the
reconsiderable-loser clause so the citation matches a stated rule. The
abstract route-lifecycle mutant keeps its witness but drops its dependence
on the deleted route label: it now terminalizes any volatile route's
provisional winner and is named `provisional_winner_terminalization`.
Limitations that asserted a live pairwise route are reworded to the
surviving vocabulary. Verified with `cargo test -p
cgka-conformance-simulator --test route_assurance --test mutation_adequacy`
(11 passed) plus crate fmt and clippy.
incident-replay fork prose drops the retired `RecoverySummary`: two
`fork.rs` doc comments still named that conformance `TraceExpectation`
variant as the thing a synthesized vector asserts. Name the observables the
arms actually pin today instead — the group-data arm's per-client
`GroupProfile` pin and the membership arm's `member_count == 3` — which
also corrects the second staleness in the same sentence: only the
membership arm is winner-agnostic. The privacy claim is unchanged and still
holds; it rests on synthetic labels, not on the assertion being
winner-agnostic.
app_runtime.rs's import block was hand-rewrapped during this conflict
resolution and left under-filled; rustfmt re-wraps it here.
* docs: state the unified fork-resolution route
- cgka-engine AGENTS.md: fork handling now points at the convergence
admission gate + distributed_convergence.rs; the ForkRecoveryManager
"Done" section is superseded by the route-unification note; the inbound
seam-parity convention drops the deleted fork-recovery seam (two seams
remain); the mirror-invariant bullet loses the uncompensatable pairwise
side effects; module map entries for fork_recovery.rs and the guard's
snapshot half are gone. tests/AGENTS.md follows.
- audit-logging.md: snapshot_created and fork_resolution are documented as
historical read-compat kinds (emitted only by pre-unification versions;
convergence_decision is the current resolution evidence).
- relay-delivery-telemetry.md: the post-settle reorg description names the
convergence rollback pair instead of ForkRecovered.
- convergence-reliability-plan.md: Milestone 6 gets a status note — the
#1236 committer-versus-observer route asymmetry no longer exists
structurally; the 6.1 route-equivalence item enumerates the routes that
still exist.
- learnings.md: new dated entry for the unification (the 2026-05-04
fork-recovery entry stays as history).
- CHANGELOG (unreleased/Changed): one rule for every member, checkpoints
materialize own commits, loud fail-closed halt on missing in-horizon
material, ~1 s committer-side resolution latency, ForkRecovered event +
FFI variant removed, forensic read-compat retained.
- distributed_convergence.rs / openmls_projection.rs: two in-code doc comments
still explained own-commit supersession as a consequence of a restart
clearing the in-memory `committed_from` guard. That guard is deleted, and
the hazard they describe is not restart-specific — any same-epoch sibling
adjudicated through stored convergence reaches it. Reworded to the general
case; the invariants themselves are unchanged.
* test(cgka-conformance-simulator): pin post-unification convergence outcomes per client
Now that one route resolves every same-epoch fork, the vectors have to say
so per client. Four escapes, each one a passing assertion that did not
carry the claim its vector advertised.
The settled convergence decision, pinned per client.
route-equivalence.v1's convergence_decision expectation carried no client,
and a client-less ConvergenceDecision is satisfied by any single observed
client's settled decision. One participant adjudicating passed the vector —
the opposite of what it claims to prove: that the live committer, the
restarted committer over hydrated storage, and both passive observers all
resolve the same fork through the one distributed-convergence route. The
exact regression it would have missed is this branch's own predecessor,
where alice resolved the fork on the pairwise seam and emitted no
convergence decision at all while every state oracle still passed. Replace
the single client-less entry with four per-client pins (alice, bob, carol,
dave); the four subsume it. Only client and selected_tip_epoch are pinned —
selected_branch_id digests randomized MLS bytes and changes on every run.
The winning branch, pinned per client rather than on convergence.
ClientsConverged only required the named clients to *share* a group name,
never a specific one, so a group-data fork vector passed whichever branch
won. Reproduced on the committed fork-recovery incident vector: swapping
only the two branch names — so the branch the tip_committer tiebreak
selects is the one named loser-branch — still yielded PASS. Nothing else
closed it. The observation.group_name == WINNER_BRANCH check in
incident-replay's accept path ran at synthesis time only and never guarded
the persisted artifact canonical_scenarios replays in CI.
Hanging the pin off ClientsConverged was tried first and rejected.
ClientsConverged is a cross-client agreement assertion — it compares the
first observation's name against the rest — so attaching a value pin to it
conflates two questions, and the value it pins is one client's, silently
the first named. #1299's GroupProfile expectation already pins a named
client's exact projected name and description, and it asks per client,
which is what "agreement is not attribution" actually requires: both
committers sitting on the loser's branch is convergence too, and only a
per-client pin says which branch each of them projects.
So the winner is pinned through GroupProfile:
- synth.rs's group-data arm emits one GroupProfile pin per committer at
WINNER_BRANCH with the empty description the profile deterministically
carries; the prose explaining why the name is the only stable winner
observable (a group-data commit changes only the name, so the two
branches agree on epoch and member count, while selected_branch_id
digests randomized MLS envelopes and differs on every run) sits beside
them. The membership arm stays winner-agnostic — a real export cannot
attribute the surviving membership branch.
- accept.rs drops the bespoke winner check: the label search and the
acceptance gate become the one compare_observed_trace the membership arm
already used, and the accept tests assert the property through the
expectation the vector carries.
- fork-recovery-incident.v1 re-derived through the synthesis path
(accept() reproduced it, so the pins are live-verified); the
expectation-only delta applied by hand because the recorded inputs would
otherwise pick up the known AcknowledgeOutbound.selection serialization
drift. With the pin flipped to loser-branch the CI replay now fails
clients_not_converged — the escape above, closed.
- route-equivalence.v1 pins all four clients to "bob branch" (stable across
6 report-CLI runs) and keeps clients_converged for the epoch/member-count
agreement it does assert.
Bidirectional decryptability, probed rather than assumed.
route-equivalence proves four member shapes resolve one fork through the
same route and land on identical canonical state, but every assertion it
carried is a passive observation of that state. Exact equivalence and a
settled decision per client say the four agree about the group; none of
them exercises the group afterwards. A post-convergence key schedule that
converged structurally while leaving some direction undecryptable passes
the vector today. Add a ProbeBidirectionalDecryptability step over all four
clients plus the matching expectation: each sends one logical event and
every directed sender-to-recipient edge must be delivered and decrypted.
Removing the step alone makes the fixture fail
missing_bidirectional_decryptability_probe, so the expectation is
load-bearing. Placement is not free: the probe is a mutating step, and the
per-client convergence-decision pins read the latest observation, so
probing after the terminal observe_exact empties all four
(missing_convergence_decision x4). It goes immediately before it.
* test(cgka-conformance-simulator): add the bounded convergence-pressure campaign
Exercise the unified fork-resolution route under finite pressure: two admins
race same-epoch profile commits, one application send per client is issued
after the rival branch is ingested (inside the quiescence window the committers
wait out), one committer restarts mid-resolution, and a bounded tail applies
three self-updates, a profile commit, and an admin-policy commit. Seeds permute
the sibling release order, the restarted committer, the window-send order, and
the tail ordering.
An explicit advance_time before the race activates controlled virtual time, so
the quiescence window is a real deadline instead of the far-future tick
shortcut; every settle after it is await_quiescence, whose watchdog budget is
the bounded-time assertion. Bounded queue behavior is a scenario_inputs_pending
resource assertion at peak pressure. The mutating decryptability probe runs
before the terminal observe_exact the strict oracle appends.
The campaign is finite by construction and claims nothing about progress under
an unbounded self-update stream.
The generator gate runs in CI. The runnable seeded gate is ignored: it fails on
engine behavior, not on the campaign. An application message accepted while the
group is resolving a same-epoch fork is queued durably and then stranded --
nothing re-arms the retained-intent drain once the pass completes, and the
engine's conformance structural progress reports runnable_work = 0 with no next
wake while pending_work.queued_outbound_intents > 0.
Formatting fixup folded in: lib.rs's re-export block was hand-rewrapped
during the ForkRecovered-retirement conflict resolution and left
under-filled once these generators joined it; rustfmt re-wraps it.
* test(cgka-engine): model production per-epoch transport sealing in Tier 2
Every Tier-2 test installs the pass-through `MockPeeler`, which hands each
engine every message verbatim. Production transports do not: a kind-445 group
message is sealed under the sender's current-epoch exporter secret and carries
no epoch hint, so a device that never entered that epoch state sees opaque
bytes and the peel returns `DecryptFailed`.
`support::epoch_sealed_peeler::EpochSealedPeeler` models that boundary with the
same AEAD and framing the Nostr binding uses (`nonce || ciphertext`, empty AAD,
group key from `ctx.exporter_secret`). The engine already wraps outbound group
messages through the peeler with the pre-commit context, so a test opts in by
installing the peeler and routing the messages it already routes.
`tests/epoch_sealed_transport.rs` owns the sealing model's own semantics
(round-trip under the sealing context, opaque to a different epoch state at the
same epoch NUMBER — the rival-branch shape) plus one end-to-end proof that
uncontested delivery is unaffected.
Infrastructure only; no engine behavior changes.
The chacha20poly1305 dev-dependency lands with its Cargo.lock entry so the
commit builds --locked on its own.
* fix(cgka-engine): read a rival branch's traffic under that branch's state
Unifying fork resolution through distributed convergence made branch
selection the only route, and branch selection ranks on valid commit
depth and app-witness score. Both are counted over inputs this device
has STORED. A group message is sealed under the sender's current-epoch
exporter secret and the transport carries no epoch hint, so once two
members commit from the same epoch, every message either branch sends
afterwards is opaque to the devices that adopted the other branch: it
sits retained as PeelDeferred, contributes neither depth nor witnesses,
and is retried only under contexts this device has itself entered --
which never include the rival branch. Each committer therefore scores
only its own branch, every device settles "stable" on its own lineage,
and the split is permanent. The deleted pairwise route was immune by
construction: CommitOrderingKey is content-only.
The engine tier could not see this. Its MockPeeler returns msg.payload
verbatim, so every Tier-2 client has perfect cross-branch visibility --
exactly what production lacks. 263ed4b7 added an epoch-faithful peeler;
the proving test built on it fails on the parent commit:
test incumbent_adopts_a_deeper_rival_branch_whose_traffic_only_it_can_read ... FAILED
thread '...' panicked at crates/cgka-engine/tests/epoch_sealed_transport.rs:318:5:
assertion `left == right` failed: the incumbent must adopt the deeper rival branch
left: 2
right: 3
and so does the fleet, on convergence-e2e-delivery/v1 seed 99 case 0:
invariant_failures: [
{"kind": "clients_not_converged", "message": "clients [alice, bob, carol, frank] did not converge"},
{"kind": "clients_not_exactly_equivalent"},
{"kind": "bidirectional_decryptability_failed",
"message": "... [alice->bob, alice->carol, alice->frank, bob->alice, carol->alice, frank->alice]"}]
alice settled at epoch 3/6 members on effective_commit_depth while bob,
carol and frank settled at epoch 2/5 members -- all "stable", holding
retained rows none of them could read. Extra drive rounds changed
nothing.
Candidate branch states are part of a group's peel context, not a new
mechanism. openmls_projection::candidate_branch_peel_contexts
materializes each candidate branch under a SnapshotRollbackGuard and
captures its tip's owned GroupContextSnapshot; because that snapshot
derives the exporter secret eagerly, the transient branch state is
rolled back before any peel runs against it, and no async work happens
inside a restored state. retry_deferred_peels then offers those contexts
to every retained row through the ordinary ingest seam, so dedup,
classification, retirement, and the flood cap stay in one place -- the
per-row bookkeeping both sweeps share now lives in
reingest_deferred_peel_row.
Peeling is only the visibility half. A message readable ONLY under a
candidate branch belongs to a lineage this device has not adopted, so
ingest routes it to the convergence seam and never to the direct apply.
That rule is load-bearing, not tidiness: a commit at the same epoch
number already entered convergence on the epoch test, but an APPLICATION
message did not, and force-feeding one to canonical OpenMLS raised
Err(Backend("process_message: ValidationError(UnableToDecrypt(AeadError))"))
which the deferred-peel sweep turns into an aborted drain. The rule also
keeps the seam honest: the peeled bytes are evidence of nothing on their
own, the next pass's OpenMLS replay is what authenticates them, and a
failed peel is silence rather than a verdict. PeelRecoverySource makes
the provenance a type rather than a flag.
The uncontested path pays nothing. The sweep returns before any context
work on an empty backlog; collection stops at a cheap "two commits share
a source epoch" test before any replay; at most
MAX_CANDIDATE_BRANCH_PEEL_CONTEXTS (8) branches materialize, one bounded
replay each under the pass's own ReplayBudget shape; and an inability to
enumerate branches -- missing anchor, missing own-commit checkpoint,
exhausted budget -- yields no contexts rather than an error, because the
pass and not this helper owns every verdict. Losing commits keep their
ConvergenceDeferred parking untouched.
One consequence needed care. Because candidate branch states are part of
the peel context, deferred_peel_context_fingerprint must fold in the
stored commit graph: a newly retained rival commit adds a readable
context while the live epoch and the retained-anchor set are both
unchanged, and without that term the sweep gate stays armed exactly
where it must not -- on a device holding its own branch while the rival
branch's traffic sits unreadable. The empty-backlog early return moved
ahead of the fingerprint so describing the context costs nothing when
there is nothing to sweep.
E10 registers MAX_CANDIDATE_BRANCH_PEEL_CONTEXTS in the convergence
constant ledger, its plan rows, and policy_contract::CONSTANT_DECISIONS
as an operational-resource, non-interfering constant.
Review follow-ups folded in: five defects in the branch-relative peel
seam, none of which change behavior on a healthy path and all of which
fail silently on an unhealthy one.
**The fail-closed rewind-probe detector was disabled.** `candidate_branch_
peel_contexts` reuses `retained_anchor_probe_snapshot_name`, so the sweep's
rewind and the pass's rewind at the same `(group, epoch)` hash to the same
snapshot name. `create_group_snapshot` is INSERT OR REPLACE and
`recover_interrupted_retained_anchor_probe` fails closed only on MORE THAN ONE
probe, so a second interrupted probe silently overwrote the first instead of
producing the two rows the detector exists to catch. The sweep's probe now
carries its own `openmls-branch-probe-` prefix and the recovery scan — renamed
`recover_interrupted_rewind_probe`, since it no longer covers one rewind —
matches both. Both new tests fail on the parent commit:
test hydration_recovers_interrupted_candidate_branch_probe ... FAILED
assertion `left == right` failed: hydrate must restore the pre-probe live state
left: Group { ... name: "historical anchor" ... }
right: Group { ... name: "healthy" ... }
test hydration_fails_closed_on_two_interrupted_rewind_probes_of_different_kinds ... FAILED
an ambiguous pair of interrupted probes must quarantine the group
**`deferred_peel_context_fingerprint` hand-inlined the graph's state filter.**
`stored_convergence_commit_digests` restated the match that
`record_state_can_contribute_to_openmls_graph` owns — precisely the drift
`seed_stored_openmls_graph_inputs` was extracted to prevent, per its own
rustdoc. The failure mode is a mis-armed sweep gate with no test failing, so
the predicate is now `pub(crate)` and the fingerprint calls it.
**`CandidateBranchPeelContext::message_retention_seconds` was dead.** It was
populated at three sites, and its only consumer is unreachable for a
candidate-branch recovery because the provenance guard returns first. Rather
than delete a field that a future caller would re-add, the retention moves onto
`PeelRecoverySource::RetainedAnchor`, the only context that has one to offer —
so a lineage this device has not adopted can no longer be asked for a retention
policy at all.
**Two doc claims were false.** Seeding is shared by the pass and by context
collection, but they do not "see the identical graph": collection passes
`admitted_message_ids: None` and the pass passes its frozen batch, so branch
ids match only when both enumerated the same path. And the uncontested path
pays the seeding scan, not "nothing" — it pays no *replay*. The contexts also
run on a fresh `ReplayBudget` of the pass's shape, not the pass's own budget.
**A swallowed enumeration halt was silent.** `MissingOwnCommitCheckpoint` and
`ReplayBudgetExceeded` collapse to "no contexts", which reports a contested
graph as uncontested — the one way this helper can hide a split-brain. Both now
emit a `tracing::debug!` with a low-cardinality reason label and no identifiers.
Reasoned about, deliberately unchanged: `MAX_DEFERRED_PEEL_ATTEMPTS = 32`. The
fingerprint now moves on every stored commit, so a busy group spends the budget
faster, but a row must sit unreadable across 32 distinct commit contexts to
exhaust it and has fallen outside `max_rewind_commits` (5) long before that.
Recorded on the constant.
* fix(cgka-engine): count a contested sweep's evidence for every branch, once
Branch-relative peel made a rival branch's traffic readable, but only the
rival's. A deferred-peel sweep re-ingested rows one at a time, and each row
took a route decided by WHICH CONTEXT had read it: a row readable only under a
candidate branch hit the provenance guard and entered the convergence seam,
while a row whose live peel now succeeded took the direct apply and was never
admitted to any pass. The pass's input is its frozen membership, so the branch
this device occupies contributed no app witnesses while the branch it had not
adopted contributed all of them. Selection was biased OFF the branch the device
held — the split-brain inverted, and the reason
`adversarial-reliability/app-witness-value/v1` case 9 still diverged after the
parent commit:
invariant_failures: [
{"kind": "clients_not_converged",
"message": "clients [alice, bob, carol] did not converge"},
{"kind": "client_state_mismatch",
"message": "client carol state was epoch 2, members 4,
payloads [eve-witness-9, frank-witness-9, david-witness-9],
added [eve, frank, david], removed [frank, eve]"},
{"kind": "clients_not_exactly_equivalent"}]
alice and bob settled on alice's branch (5 members); carol settled on bob's
(4 members) and stayed there. With this commit all three settle on one branch
and the convergence invariants clear.
An application message is now evidence for the pass whichever context read it,
for as long as the sweep has candidate branches in hand. Commits need no such
rule: `commit_should_enter_convergence` already decides on epoch, which is
provenance-blind. The uncontested path is untouched — `candidate_branch_peel_
contexts` returns nothing unless two branches actually compete, so
`DeferredPeelSweep::is_contested` is false and every row keeps the ordinary
ingest decision.
The second half is that one sweep's rows are ONE evidence set. Each buffered
row used to drain the group, so a pass could freeze on a PREFIX of the sweep
and settle a verdict the remaining rows would have changed — then terminalize
the losers of a decision taken on incomplete evidence: a rival app message
peeled at row 1 is classified LosingBranch and persisted `EpochInvalidated`,
which `record_state_is_canonicalization_input` never re-admits, and row 2 then
makes that rival branch win. Commits survive this through their
ConvergenceDeferred parking; application messages had no such protection.
A contested sweep now buffers its whole batch and drains once, at the end.
That also corrects what one sweep costs. The parent commit priced
MAX_CANDIDATE_BRANCH_PEEL_CONTEXTS as "8 x 64 symmetric AEAD attempts on top of
8 replays"; with a drain per recovered row the real worst case was 64 full
convergence drains, each up to MAX_CONVERGENCE_REPROCESSING_PASSES. One drain
per sweep is what that rationale always claimed.
No new test failures. `cargo nextest run -p cgka-engine --features
test-policy-overrides` is 461/461. The three `canonical_scenarios` convergence
e2e failures in the tree are unchanged by this commit — verified identical at
0984ff95, at the seam-defect commit, and here — and are the expectations
acceptance item 2 exists to re-author.
* test(cgka-conformance-simulator): demand branch retraction instead of a silent losing branch
Branch-relative peel made a rival fork's traffic readable under that
branch's own state (e0e5dd73 "read a rival branch's traffic under that
branch's state"). Three convergence-E2E oracles were written while that was
cryptographically impossible, and each read the new, honest output as
failure. Pre-change RED:
---- convergence_e2e_from_peeler_ingest_to_group_events ----
panicked at tests/canonical_scenarios.rs:3131:
carol invalidations: [EpochChanged { from: 1, to: 2 }, GroupStateChanged
{ change: MemberAdded { .. } }, CommitRolledBack { .. },
GroupStateInvalidated { reason: SupersededByBranchSelection },
MessageReceived { .. }, AppMessageInvalidated { reason: LosingBranch,
decrypted_payload_ref: Some("sha256:52b9fc4f..") }, ..]
---- scenario_report_records_convergence_e2e_group_events ----
---- convergence_e2e_delivery_family_runs_generated_variants ----
panicked at tests/canonical_scenarios.rs:3177:
assertion `left == right` failed
left: ["eve", "david", "grace"]
right: ["david", "grace"]
assert_canonical_application_event demanded that no AppMessageInvalidated
ever appear. That held only while the losing branch's payload was
undecryptable: it was dropped in silence because nothing could read it.
Now it decrypts and convergence owes the application an explicit
retraction. Old obligation -> new:
- "no invalidation, ever" -> exactly one, reason LosingBranch
- deliveries == [winner] -> unchanged, plus: no delivered message id may
appear in any retraction, so a payload shown
and then abandoned fails whether or not a
retraction follows
- EpochChanged 1 -> 2 -> unchanged
assert_real_peeler_convergence_trace enumerated two allowed observation
shapes. Its second arm accepted a client that ENDED the schedule at epoch 2
with five members, holding only bob's invite and bob's payload and no
invalidation - the permanent split brain this branch fixes. A client handed
both branches must settle on alice's, which carries two invites to bob's
one, so depth decides it. That arm is deleted, not generalized. Old -> new
for the surviving arm:
- epoch 3, 6 members, [alice payload], epoch_changes [1->2, 2->3]
-> all four unchanged
- added == ["david","grace"], removed empty
-> net diff == ["david","grace"], and every
removal must be of a member this client had
added. Whether the rival branch is adopted
before the reorg is a delivery-order detail;
where the client ends up is not.
- app_invalidations empty -> exactly ["losing_branch"]
The same test's `deferred-losing-transport` tail asserted that the losing
branch's transport object stays retained and unpeeled forever, and that
NoPendingWork therefore fails for both observers as a steady state - the
same blindness arm two encoded. It now pins the opposite and stronger
claim: zero stored_transport_deferred_messages, zero scenario_inputs_pending,
and zero quiescence failures. The pending_work_remaining reporting path
keeps independent coverage in
no_pending_work_rejects_delayed_transport_then_accepts_drained_delivery and
vector.rs's no_pending_work_requires_an_empty_exact_progress_snapshot.
The report test's own `app_invalidation_observations.is_empty()` is arm
two's claim at the report layer; it becomes the exact per-client retraction
list.
Verified: all 47 canonical_scenarios tests, report_runner 23/23, and
cgka-engine 464/464 under test-policy-overrides. The delivery family and
report spec were each run three times against the re-authored trace oracle
(3/3 green) to rule out trading a deterministic red for a flaky one.
* fix(cgka-engine): keep a contested sweep contested when branch enumeration halts
`DeferredPeelSweep::is_contested` read contested-ness off the captured
branch contexts, but `candidate_branch_peel_contexts` reached "no
contexts" down five distinct paths and only ONE of them meant the graph
was uncontested:
1. `!commits_share_a_source_epoch(...)` — genuinely uncontested;
2. rewinding to the retained anchor hits `SnapshotMissing` — contested;
3. a `MissingOwnCommitCheckpoint` / `ReplayBudgetExceeded` halt — contested;
4. fewer than two candidate paths survived replay — contested;
5. every `candidate_path_peel_context` returned `None`, or the budget
broke before the first capture — contested.
So the doc comment's "returns nothing unless two branches actually
compete, so a non-empty set means ..." held in one direction only: a
non-empty set proves a fork, an empty set proves nothing. The one thing
this helper must never do is report a fork as healed exactly when this
device stopped being able to see it — and reading contested-ness off an
empty set did precisely that. The halt is not exotic either: enumeration
funds itself with `ReplayBudget::for_pass(...)`, the same budget a pass
gets, then pays for up to `MAX_CANDIDATE_BRANCH_PEEL_CONTEXTS` further
whole-path replays the pass never pays, so it is structurally more likely
to exhaust than the pass that follows it.
Contested-ness now travels in its own field on a new `CandidateBranchPeel`,
sourced from the shared-source-epoch check that already runs before any
replay. Every return after that check is wrapped by
`CandidateBranchPeel::contested_over`, so a halt can only ever lose the
contexts, never the split.
The two meanings stay split at the consumer, because they answer
different questions:
- `is_contested()` gates the DRAIN POLICY only. A contested sweep's
recovered rows are one evidence set, so the drain waits for the whole
batch instead of freezing a pass on a prefix of it. The end-of-sweep
drain keys on the same predicate as `drain_policy`, so no row is ever
deferred to a drain that does not run.
- evidence routing keys on the new `has_branch_contexts()`. Routing
live-readable application traffic into a pass whose branch enumeration
just halted feeds evidence to an adjudicator that likely cannot read
the rival branch either — same budget, same missing checkpoint. That
restores no symmetry, it only adds noise.
`recovered_from_candidate_branch` semantics are unchanged: it still
tracks whether THIS row was read under an unadopted lineage.
Cost asymmetry: a false positive on `contested` (a fork present, no
surviving paths) costs one deferred drain instead of N immediate ones —
strictly the safer direction, and bounded, since a resolved fork drops
its loser out of the stored graph and the check goes quiet again.
RED, from the new unit tests against the pre-fix single-field sweep:
error[E0432]: unresolved import `crate::openmls_projection::CandidateBranchPeel`
error[E0599]: no method named `has_branch_contexts` found for struct `DeferredPeelSweep<'a>` in the current scope
error[E0599]: no method named `has_branch_contexts` found for struct `DeferredPeelSweep<'a>` in the current scope
error: could not compile `cgka-engine` (lib test) due to 3 previous errors
The old shape could not express "contested with nothing to read" at all,
which is the defect stated as a type.
Verified: `cargo nextest run -p cgka-engine --features test-policy-overrides`
466 passed / 0 failed (464 + 2 new); `just fast-ci` exit 0.
* test(cgka-engine): drive both enumeration halts and pin the contested verdict
The tests that shipped with the halt fix build `CandidateBranchPeel` by
hand, so they pin only the CONSUMER: that `contested` drives the drain and
`contexts` drives evidence routing. Nothing exercised the PRODUCER. A
`candidate_branch_peel` regressed to report `contested: false` whenever a
halt lost the contexts -- exactly the regression the fix exists to prevent
-- left every one of them green.
Drive the helper into both halts it names, over a real forked graph of real
OpenMLS commits, and assert on its return value:
- `MissingOwnCommitCheckpoint`: a two-way epoch-1 fork where this device
committed one branch, then its post-merge checkpoint is released. MLS
cannot replay an own path-bearing commit from the public wire echo, so
enumeration loses the branch it is standing on.
- `ReplayBudgetExceeded`: a same-epoch fork ground WIDTH ways, each rival a
frontier path, plus DEPTH successors at the next epoch that enumeration
must probe against every one of those paths. The budget is linear in the
commit count, the probes are its product. A compile-time assertion holds
the arithmetic so a change to either budget constant lands there rather
than as a mystifying empty-context failure.
Each test first runs the same graph without the halt and asserts it
enumerates more than one context, so an empty set proves the halt rather
than a graph that never had two branches to offer.
The helper is `pub(crate)` and the contexts it returns carry derived
exporter material, so this is an in-crate test module rather than a
widened public surface.
RED, with `contested_over` regressed to the pre-fix
`contested: !contexts.is_empty()`:
thread 'openmls_projection::candidate_branch_peel_halt_tests::a_lost_own_commit_checkpoint_halts_enumeration_and_keeps_the_fork' panicked at crates/cgka-engine/src/openmls_projection.rs:4513:9:
the fork is in the stored graph, not in what enumeration managed to read
thread 'openmls_projection::candidate_branch_peel_halt_tests::an_exhausted_replay_budget_halts_enumeration_and_keeps_the_fork' panicked at crates/cgka-engine/src/openmls_projection.rs:4573:9:
the fork is in the stored graph, not in what enumeration managed to read
test result: FAILED. 0 passed; 2 failed; 0 ignored; 0 measured; 88 filtered out
The hand-built consumer tests stayed green through that same regression:
test message_processor::ingest::tests::a_halted_enumeration_leaves_the_sweep_contested_with_nothing_to_read ... ok
test message_processor::ingest::tests::an_uncontested_graph_drains_per_row ... ok
test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 88 filtered out
They keep their place -- the split they pin is a distinct property -- with
a note pointing at where the producer is now covered.
* test(cgka-engine): acknowledge the fixture's join event so a reopen cannot replay it
`convergence_pass_that_keeps_the_own_commit_emits_no_own_withdrawal` and
`convergence_pass_that_replaces_the_own_commit_withdraws_it_exactly_once`
began failing on "buffering the rival must not be application-visible"
after this branch was rebased onto master 85ef573b.
What leaked, captured by binding the drained vector into the assertion
message:
[GroupJoined { group_id: .., via_welcome: .., welcomer: Some(..) }]
Proven cause, and it is not the buffering. `a82524ca` (#1391) made
`do_join_welcome` write its `GroupEvent::GroupJoined` to the durable
pending-application-event outbox added by `01c09a9e` (#1366). That outbox
is at-least-once: `drain_events` hands the event to the host but does not
retire the row; only the host's `delete_pending_application_events` ack
does, and `EngineBuilder::build` re-seeds `events_buf` from
`list_pending_application_events` on every open. `router_flip_fixture_
arranged_with_creator` never acked the joiner's `GroupJoined`, so the
`reopen_legacy_client` these two tests perform correctly re-delivered it,
and it was sitting in the buffer before `ingest` was ever called. Both
halves of that mechanism predate the rebase individually; only their
combination -- the `GroupJoined` put landing on top of the durable outbox
-- is new, which is why the tests passed at `01c09a9e` and at the
pre-rebase tip `eeebcb2c`.
The engine is right on every step here, so nothing in `src/` changes and
the assertion keeps its exact strength: the arrangement was measuring a
replayed event it had never consumed. The fixture now does what a host
does -- takes the join notification and acks it -- mirroring the pattern
already established at `tests/ingest.rs`'s welcome/reopen sites. A
spurious event emitted by the ingest under test still fails the assertion.
For a future bisector: the hand-resolved conflict in
`message_processor/ingest.rs` during that rebase is exonerated. The
line-level delta `eeebcb2c..582af852` on that file equals the delta
`01c09a9e..85ef573b` exactly, modulo the four-space re-indent required
because this branch removed the `loop` that wrapped the `MlsGroup::load`
arm. Both of master's changes (#1379's `retryable_unpersisted_ingest_id`
and #1367's `Option` refactor) survived intact and nothing else moved.
469/469 `cargo nextest run -p cgka-engine --features test-policy-overrides`.
* test(cgka-conformance-simulator): state the losing-branch withdrawal as a rule
The app-runtime cross-route characterization enumerated two participant-local
invalidated-row shapes and rejected everything else. Observed RED, ~5 runs in
12 on this branch:
---- four_party_cross_route_recovery_characterizes_corrected_app_runtime_outcomes ----
panicked at tests/process_orchestrator.rs:1097:
unexpected invalidated disposition shape: {
"alpha": 1, "observer": 1, "yankee": 0, "zeta": 0,
}
What Observer withdraws, dumped from the app projection in two independent
failing runs (10 and 11):
client=alpha kind=1210 invalidated=true sender=<alpha> epoch=4
{"system_type":"group_renamed","data":{"actor":"<alpha>",
"name":"alpha-root","old_name":"cross-route"}}
client=observer kind=1210 invalidated=true sender="" epoch=4
{"system_type":"group_renamed","data":{
"name":"alpha-root","old_name":"cross-route"}}
Both runs ended with all four participants at epoch 5 on
`zeta-branch-depth-two`, and Yankee and Zeta held no invalidated row.
Mechanism, not the peel. Nothing on the losing branch was decrypted: the row
is a locally synthesized kind-1210 system row, stamped with `alpha-root`'s
`origin_commit_id`, that `GroupStateInvalidated` tombstones when
`emit_rolled_back_commits` withdraws the superseded commit. Observer's copy is
unattributed because `emit_convergence_events` reaches the branch by replaying
stored commits and emits profile deltas with `actor: None`, so it is not even
the same canonical id as Alpha's self-attributed row. Observer therefore
applied Alpha's commit through stored convergence and was later reorged off it
— the #363 anti-lie contract firing on a participant that is not the losing
committer.
The test originated on master (#1395 / #1398) describing behavior before this
branch unified same-epoch fork resolution through distributed convergence, when
the observed shapes were the enumerated two. Enumerating shapes was the defect:
which participants transiently hold the losing branch is a delivery-order
detail, so the assertion now states the rule instead. Old -> new:
- "all zero, or Alpha alone at one"
-> Alpha withdraws its own confirmed alpha-root row EXACTLY when it
ends off that branch. All-zero is no longer an unconditional escape;
a converged Alpha silently keeping the row now fails, which is the
shape the old assertion accepted.
- Yankee and Zeta at zero
-> unchanged, now with its reason: Zeta is the rival same-epoch
committer and defers only to a deeper branch, and Yankee is two
commits deep on the selected branch before `step-25` reaches it, so
neither ever synthesizes the row.
- Observer forced to zero
-> at most one, and only after leaving the losing branch and advancing
past the pre-fork baseline. This is the single relaxed clause, and
it is paid for by the mandatory Alpha withdrawal above.
- (new) no participant may show a withdrawn row as delivered, mirroring the
retraction oracle in canonical_scenarios.rs.
Every other assertion in the test is untouched, and it stays a characterization
of a counterexample: the terminal-outcome arms below it still accept the
two-branch splits.
Mutation-checked the added Alpha clause by inverting its predicate; it goes RED
on the real run, so the clause is live rather than vacuous.
`cross-route-app-runtime-recovery/v1` has no JSON vector, and the sibling
`cross-route-own-commit-recovery/v1` fixture expresses convergence decisions
and group profiles rather than invalidated rows, so neither encodes the
enumerated shape. The other `invalidated_message_ids.is_empty()` assertions in
this file and in app_runtime_adapter.rs are on fork-free restart/offline
scenarios.
Verified: the named test 6/6 green (5/12 red before), process_orchestrator
11/11, route_assurance 8/8, cgka-engine 469/469 under test-policy-overrides,
`just fast-ci` exit 0.
* fix(cgka-engine): rank candidate branches before capping peel contexts
`candidate_branch_peel_contexts_from_current` took the first
`MAX_CANDIDATE_BRANCH_PEEL_CONTEXTS` entries of the BFS's completed-path
vector. That vector is built in level order, so the prefix is the
SHALLOWEST branches: eight one-commit rivals from a single member evict
every branch two or more commits deep — and a deep branch is exactly the
one carrying the post-fork traffic a retained row needs unsealed. Posting
eight junk commits at one epoch is the whole attack.
Candidates are now ordered by tip epoch descending, then by path-digest
branch id, before the cap applies. `tip_epoch` is joined off
`path_result.materialized` by `branch_id`, not by index: the two vectors
are parallel by construction, but `materialized` is documented as allowed
to come back short, and an index skew would silently rank one branch by
another branch's depth. Both keys are content-derived, so peers holding
the same evidence keep the same subset — the property that makes capping
safe at all. Branch selection is untouched and stays uncapped.
RED, before the ranking landed:
thread '...a_deeper_branch_outranks_shallow_rivals_for_the_capped_contexts'
panicked at crates/cgka-engine/src/openmls_projection.rs:4670:9:
assertion `left == right` failed: a branch carried two commits deep
holds traffic the one-commit rivals cannot unseal, so filling the cap
with rivals must not evict it
left: Some(2)
right: Some(3)
Two tests over a ten-branch fork (nine one-commit rivals plus one
two-commit branch, one more branch than the cap): the cap is filled
exactly and two peers holding the same evidence keep the same branch ids
in the same order; and the deep branch survives the cap. The existing
fixture only asserted `contexts.len() > 1`, so neither the truncation nor
its cross-peer uniformity was pinned by anything.
* docs(cgka-engine): correct why the candidate branch peel cap is eight
The constant justified its value two ways, and both were false.
"v1 `max_rewind_commits` is 5" bounds a path's LENGTH. This cap bounds
the path COUNT, which is set by branching factor — and nothing bounds
branching factor.
"a graph wide enough to need more than eight branches is already failing
the pass's own `ReplayBudget`" is arithmetically wrong.
`ReplayBudget::for_pass` is `commits × (max_rewind + 1) × 4 + 32`, so at
v1 it is `commits × 24 + 32`: a flat nine-way fork is nine commits, a
budget of 248, and about nine probes — nowhere near exhaustion. This
crate's own budget-exhaustion fixture has to stack WIDTH=10 with DEPTH=13
AND max_rewind_commits=0 to trip it. Width alone never gets there.
State the real reason instead: candidate-path count is attacker-inflatable
at one message per extra branch, each context costs a whole branch replay
plus up to `MAX_DEFERRED_ROWS_PER_SWEEP` AEAD attempts, and the kept
subset is content-derived — ranked by tip epoch then branch id — so every
peer with the same evidence caps identically. Uniform, not divergent.1 parent 85ef573 commit 99f4144
83 files changed
Lines changed: 6257 additions & 4337 deletions
File tree
- crates
- cgka-conformance-simulator
- src
- tests
- vectors
- generated-inputs
- incidents
- cgka-engine
- src
- message_processor
- tests
- support
- cli
- incident-replay
- src
- tests
- marmot-app/src
- client
- runtime
- marmot-forensics/src
- marmot-uniffi/src/conversions
- traits
- src
- tests
- snapshots
- docs
- marmot-architecture
- scripts
Some content is hidden
Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
57 | 57 | | |
58 | 58 | | |
59 | 59 | | |
60 | | - | |
61 | | - | |
| 60 | + | |
| 61 | + | |
62 | 62 | | |
63 | 63 | | |
64 | 64 | | |
| |||
146 | 146 | | |
147 | 147 | | |
148 | 148 | | |
149 | | - | |
| 149 | + | |
150 | 150 | | |
151 | 151 | | |
152 | 152 | | |
| |||
242 | 242 | | |
243 | 243 | | |
244 | 244 | | |
| 245 | + | |
| 246 | + | |
| 247 | + | |
| 248 | + | |
| 249 | + | |
| 250 | + | |
245 | 251 | | |
246 | 252 | | |
247 | 253 | | |
| |||
262 | 268 | | |
263 | 269 | | |
264 | 270 | | |
265 | | - | |
| 271 | + | |
266 | 272 | | |
267 | 273 | | |
268 | 274 | | |
| |||
Lines changed: 30 additions & 2 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
14 | 14 | | |
15 | 15 | | |
16 | 16 | | |
17 | | - | |
18 | | - | |
| 17 | + | |
19 | 18 | | |
20 | 19 | | |
21 | 20 | | |
| |||
46 | 45 | | |
47 | 46 | | |
48 | 47 | | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
49 | 77 | | |
50 | 78 | | |
51 | 79 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
16 | 16 | | |
17 | 17 | | |
18 | 18 | | |
19 | | - | |
| 19 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
72 | 72 | | |
73 | 73 | | |
74 | 74 | | |
75 | | - | |
| 75 | + | |
76 | 76 | | |
77 | 77 | | |
78 | 78 | | |
| |||
0 commit comments