chore(governance): maintainer-led direction policy + CODEOWNERS/SECURITY/PR template#124
Closed
heyoub wants to merge 581 commits into
Closed
chore(governance): maintainer-led direction policy + CODEOWNERS/SECURITY/PR template#124heyoub wants to merge 581 commits into
heyoub wants to merge 581 commits into
Conversation
Rename the public policy_gates example to caller_defined_gates so the packaged crate uses substrate/caller-defined gate language rather than policy phrasing. Update the examples index and traceability artifact list in lockstep. Remove tenant/customer-shaped fixture strings from packaged tests and docs, and keep the build-time banned-declaration guard focused on generic substrate boundaries rather than domain vocabulary that no longer appears in public examples.
Add a structural architecture lint that scans production core source for syncbat/runtime and PCP profile vocabulary. This keeps batpak core as the substrate boundary before the new runtime layer is added as a separate workspace crate. The lint is wired into cargo xtask structural and covered by unit tests for positive leaks, clean substrate terms, and production-source scoping.
Create the initial syncbat workspace crate as a separate runtime-layer home over batpak core. The first public surface is deliberately tiny: a typed syncbat receipt-extension namespace and helpers that compile to batpak receipt-extension keys and values. Keep default-members pinned to crates/core, but wire syncbat explicitly into xtask check, test, clippy, pre-commit, and ci. Extend the tooling structural lint so adding crates/syncbat requires those explicit gates while the core package remains the default Cargo surface.
Track B runtime slice. Runtime invocations now own one receipt emission path: successful handlers record completed receipts, handler failures record failed receipts, sink failures fail closed, and handlers no longer receive the receipt sink directly. Add syncbat-macros with #[syncbat::operation] descriptor/codegen sugar and trybuild diagnostics for invalid declarations. The macro emits descriptors and optional registration helpers; it does not own dispatch, policy, or runtime behavior. Harden the syncbat boundary lint so it scans syncbat-macros, ignores comments, and catches grouped imports of internal batpak store modules.
Track B stack layer rung. Adds the clawbat facade as cb kit vocabulary over syncbat, with #[cb::operation] re-export, PassRef and CapabilityRef validation, and tests proving declarations run through syncbat rather than a clawbat-owned runtime. Adds netbat as the nb boundary layer: route/module/introspection metadata over syncbat modules and cores, with no async runtime dependency, no batpak direct dependency, and no dispatch path. Extends syncbat with CheckoutFrame, Checkout, and CheckoutResult over the existing runtime-owned receipt path. Core::checkout re-resolves mounted descriptors so stale external checkout metadata cannot control runtime context or receipt fields. Generalizes family-layer enforcement: boundary lint now covers core/syncbat/syncbat-macros/clawbat/netbat, tooling contract requires explicit family-crate gates, and xtask check/test/clippy/ci/docs/coverage include syncbat, clawbat, and netbat while default-members stays core-only.
Add a real sync transport rung to netbat without giving it listener, thread, dispatch-policy, or batpak ownership. The new line protocol handles one already-accepted blocking stream with bounded request lines, validated operation names, hex-encoded byte payloads, stable success/error response frames, Interrupted-read retry, and output serialization limits. The transport dispatches exclusively through syncbat::Core checkout frames. Public RequestFrame values are revalidated at dispatch so callers cannot bypass configured limits by skipping the decoder. Hardens the netbat boundary tests from metadata-only coverage to 21 cases covering decode/dispatch/serve success, runtime errors, handler errors, line/input/output limits, malformed tokens and hex, CRLF/bare-CR handling, empty streams, Interrupted reads, EOF-framed requests, stable response encoding, and post-dispatch output-limit semantics. Also removes a clippy-blocking usize-to-u32 cast in the mmap fallback digest path by iterating seed values as u32 directly.
Validate operation descriptors and module names before they enter live registers, modules, or runtime builders while keeping OperationDescriptor::new const-friendly for declaration sites. The validation grammar rejects empty, overlong, whitespace/control/path/query/hash-shaped, and malformed dot-token names across operation names, schema refs, receipt kinds, module names, and handler names. Invalid descriptors and handler names now surface typed BuildError/RegisterValidationError variants instead of silently entering runtime state. Add focused descriptor and register projection tests covering accepted stable tokens, invalid descriptor fields, invalid modules, builder rejection, deterministic register ordering, duplicate rejection, cache/register equivalence, unknown checkout, and generated valid-name lookup.
- make endpoint, route, module exposure, and server mount construction validate boundary metadata - reject malformed operation names, paths, method labels, and duplicate method/path pairs - keep netbat as metadata/transport boundary only; syncbat still owns dispatch and batpak still owns records - add route validation tests for normalization, rejection cases, duplicate routes, and stable mount order
- carry batpak disk position and receipt extensions through BatpakReceiptFields - test StoreReceiptSink against a real Store append and queried index entry - prove syncbat signed receipt drawer metadata is committed as batpak receipt extensions while local drawer data stays local - test Core configured with StoreReceiptSink banks exactly one success receipt
- fail structural checks on upward stack dependencies across batpak, syncbat, clawbat, and netbat manifests - reject production async runtime dependencies in the batpak family manifests - scan syncbat, clawbat, and netbat runtime sources with syn for async functions, unsafe functions, and unsafe blocks - add integrity tests for dependency table parsing and runtime-shape detection
- reject pass/capability refs that start or end with separators - reject adjacent separator bytes while keeping the small ASCII userland vocabulary - drill the accepted/rejected ASCII byte space in clawbat facade tests - keep clawbat as declarative vocabulary over syncbat without adding runtime ownership
Cap uuid below the 1.21 getrandom 0.4 switch and cap tempfile below the 3.25 flexible >=0.3,<0.5 getrandom edge. Refresh standalone fixture locks so workspace and fixture dependency graphs stay on getrandom 0.3 and cargo-deny no longer emits the duplicate getrandom warning.
- Introduce 000_REPO_MAP.md for repository navigation and governance. - Add 001_BATPAK_SUBSTRATE.md to document the batpak substrate layer. - Create 002_SYNCBAT_RUNTIME.md detailing the sync-first runtime layer. - Add 003_CLAWBAT_KIT.md for the operation-kit facade documentation. - Introduce 004_NETBAT_NETWORK.md to describe the server/network boundary layer. - Add 010_USER_GUIDE.md as a human-first usage guide for batpak. - Create 020_TECHNICAL_REFERENCE.md for a compact technical reference. - Introduce 030_RELEASE_RUNBOOK.md outlining the release process. - Add 040_TESTING_DOCTRINE.md defining the operational testing doctrine. - Create 041_TESTING_LEDGER.md to record doctrine-bearing test suites. - Introduce 050_SECURITY.md for security reporting guidelines. - Add 060_CONTRIBUTING.md to outline the contribution process and environment setup. - Update .gitignore to refine ignored files and directories.
…odules - Add comprehensive Clippy linting rules to the workspace configuration in Cargo.toml. - Enable workspace linting for the clawbat, netbat, and syncbat crates. - Move Clippy linting rules from the core crate to the workspace level for consistency. - Introduce new route and transport modules in netbat, defining structures and error handling for boundary routes and TCP transport. - Update the builder in syncbat to improve module registration error handling. - Refactor staged command to correctly parse paths from output.
- centralize frame decode error mapping and implement Error for decode/extension errors - extract cold-start row vocabulary out of segment reach-throughs - extend workspace lint inheritance across tool and companion crates - add source citation structural checks and refresh stale ADR citations - fix renamed testing-ledger diagnostics, docs artifacts, and fuzz-chaos ignored execution - route fixture/trybuild targets into the repo target discipline and harden a subscription exhaustion test
Add verb-forward read_raw and explicit by_entity aliases while keeping the older surfaces available. Add Ctx as the preferred syncbat handler context spelling alongside Cx. Pin schema snapshot canonical bytes with a property/golden stability test and traceability invariant. Tighten integrity checks so ledger cargo-test filters must resolve to real tests, including nested integration modules, and changelog sections with breaking/removal language require migration notes.
Add the pre-1.0 correction strategy and vocabulary target, make Canal a real typed-reactor selector, add advisory public API/semver release checks, and tighten accidental public surface around ClockKey, DiskPos, and frontier naming.
- unify read-walk evidence with the plain query visibility snapshot so hidden ranges cannot diverge - route runtime raw event-kind packing through EventKind::as_raw_u16 - add symmetric NETBAT request encoding constants, helper, and boundary coverage - validate checkpoint ids, coordinate identity separators, and nonzero cursor gap capacity - update docs, changelog, traceability, public API baseline, fuzz model, and typestate fixtures Final gates: cargo xtask ci passed; cargo clippy --workspace --all-targets --all-features -- -D warnings passed; cargo test -p batpak --all-features passed; cargo xtask structural passed; cargo xtask public-api --strict --check-baseline passed. Co-authored-by: heyoub <212025917+heyoub@users.noreply.github.com>
…ever bound recovery (P1) Root-cause fix for the corrupt-footer recovery seam (closes the bug class). Round-3's crc_valid_frames_end still used the untrusted string_table_offset as an UPPER BOUND. If a CRC-failed SDX3 / legacy SDX2 / forged trailer offset landed INSIDE a later CRC-valid frame's header or payload (not at a frame boundary), validate_sidx_boundary_not_truncating didn't catch it (no frame begins there) and the walker stopped at that frame's start — silently dropping that CRC-valid frame and all later frames (pre-fix: a mid-frame hint recovered 1 of 6 frames). Data loss. Fix: for an UNTRUSTED boundary the trailer offset is garbage and is now discarded entirely. crc_valid_frames_end takes file_len (not the hint) and walks CRC-valid frames from frames_start to their NATURAL end — stopping at the first frame that fails (truncated / over-large len / frame_decode Err / tail past file_len). Footer/garbage bytes never decode as a CRC-valid frame, so the scan stops at the true frame-region end regardless of whether the garbage hint was too-low, mid-frame, or too-high. Recovers ALL CRC-valid frames (availability) while admitting zero non-CRC-valid bytes (integrity) — no offset value can truncate. - Removed validate_sidx_boundary_not_truncating (now redundant + a false-reject hazard; no production callers remained). - The frames_end < frames_start CorruptSegment lower-bound check is gated to TRUSTED boundaries only at all three sites (recovery.rs, full_scan.rs, the compaction copy) — a too-low UNTRUSTED hint recovers, not errors. - TRUSTED (CRC-authenticated SDX3) behavior unchanged: authoritative offset, lower-bound check active, mid-stream corruption before it still FailCloses. Tests: mid-frame / too-low / too-high untrusted hints all recover all frames (mid-frame + too-low fail on pre-fix code, verified by stash); trusted-corruption and no-footer fail-closed tests unchanged; SDX2 tail/non-tail + graceful fallback pass. segment 78, cold_start 3 + segment_scan_hardening 15, atomic_batch (hooks) 24, clippy -p batpak + structural-check clean. Oversize cap 928->986. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…corruption under an untrusted footer (P1) Completes the corrupt-footer recovery matrix. Round-4's crc_valid_frames_end recovered CRC-valid frames and stopped at the FIRST non-decodable frame as a clean EOF — which silently converted MID-STREAM corruption into EOF: a sealed segment [valid][corrupt][later valid][untrusted footer] recovered only the prefix, dropping the corrupt frame AND all later valid events, where a trusted/ no-footer scan would FailClosed. Resolution (the textbook EOF-vs-corruption disambiguation): a non-decodable position P is the true end of frames ONLY if nothing CRC-valid follows it. On reaching P, a look-ahead resync (crc_valid_frame_exists_after) probes every byte offset from P+1 to file_len via try_decode_frame_at: - a CRC-valid frame found after P -> P is mid-stream corruption -> CorruptSegment (fail closed; no silent truncation). - nothing valid after P -> P is the real frame-region end (footer or torn tail) -> recover prefix [frames_start..P]. False-positive guard: 8 zero bytes decode as a zero-length frame (CRC32 of empty == 0) and zeros are endemic in the footer entry table (prev_hash:[0;32]); reject claimed_len == 0 — real frames always carry a non-empty FramePayload. Cost: the resync only scans the tail after the last valid frame (the footer in the benign case), bounded by file_len. Applied at all three sites (recovery.rs, full_scan.rs, compaction copy); crc_valid_frames_end gained a segment_id for error context. Completes the matrix: trusted=authoritative+FailClosed; untrusted+intact (offset high/low/mid)=recover-all; untrusted+mid-stream-corruption=FailClosed (new); untrusted+torn-last-frame=recover-prefix; no-footer=FailClosed; SDX2-legacy=recover. Tests: untrusted_footer_mid_stream_corruption_fails_closed (fails on round-4 code, verified by stash) + untrusted_footer_torn_last_frame_recovers_prefix + 2 unit tests; cases 5/6 (trusted + no-footer FailClosed) and 1/2/4/7 unchanged. segment 80, cold_start+segment_scan_hardening 17, atomic_batch (hooks) 24, clippy + structural clean. Oversize cap 986->1161. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ully inert (P1, comprehensive)
Applies the seam's governing principle UNIVERSALLY (prior rounds fixed instances;
this audits every offset-use site): an unauthenticated SIDX trailer offset must
NEVER cause data loss OR a hard failure — it always degrades to CRC-valid-frame
recovery. Trust is determined FIRST (authenticated_string_table_offset / SDX3
CRC); a TRUSTED offset stays authoritative + bounds-checked; an UNTRUSTED offset
(CRC-failed, legacy SDX2, torn trailer, or a no-footer segment whose last bytes
coincidentally equal the magic) is fully inert — never used as a bound, never
validated as a hard error, recovery is purely crc_valid_frames_end.
Root-cause (Codex round-6): the Batch A upper-bound check
(string_table_offset > file_len-16 -> CorruptSegment) ran BEFORE trust was
determined, so a torn/garbage/coincidental-magic out-of-bounds offset hard-erred
cold start / compaction even though the CRC-valid frames were recoverable. Fix:
compute `trusted` first, gate that upper-bound hard error on `trusted`, and gate
the three lower-bound CorruptSegment checks (recovery.rs, full_scan.rs, the
compaction copy) on `!untrusted_boundary`. Audited all 12 sites that read/derive
the offset; each is now trust-gated (untrusted -> ignored + crc_valid_frames_end;
trusted -> authoritative). No path remains where an untrusted offset value can
make Store::open / compaction error or drop CRC-valid frames.
Tests: out-of-bounds untrusted offset recovers (fails on pre-fix, stash-verified);
torn SDX3 trailer recovers; no-footer-coincidental-magic recovers; PROPERTY
battery over adversarial offsets {0, 8, mid-frame, file_len-16, file_len,
u64::MAX/2} all recover (file_len shape fails pre-fix). All prior round 1-5
fail-closed + recover tests unchanged. segment 84, cold_start 3 +
segment_scan_hardening 21, atomic_batch (hooks) 24, recovery sweep 13, clippy +
structural clean. Oversize cap 1161->1406 (split tracked separately).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…-vs-fail-closed (P1) The untrusted-footer recovery (rounds 3-6) couldn't honor the caller's FailClosed tail policy: a torn/corrupt LAST committed frame followed by a corrupt footer was indistinguishable from "intact frames + footer" via the offset alone, so it silently recovered the prefix and dropped a committed event. Codex round-7. Resolution leverages existing infra (no new on-disk format, no new artifact): the SIDX footer's per-frame entry table is a self-authenticating MANIFEST. SidxEntry decode is CRC-independent, so even when the footer CRC fails we parse the entries as UNTRUSTED HYPOTHESES, then re-authenticate each against the recovered CRC-valid frames — an entry whose (frame_offset, frame_length, content-addressed event_hash) matches a real CRC-valid frame is corroborated regardless of the footer CRC (a forger/bit-flip cannot match a real frame's blake3). Decision (new recovery_manifest.rs): - a corroborated entry naming a committed frame at/after the recovery stop that is missing from the recovered set -> the manifest (anchored to this segment by >=1 corroborated entry) proves a trailing committed frame is lost -> FailClosed. - all corroborated entries map to recovered frames -> complete -> recover prefix. - entry table unparseable / zero corroborated -> no trustworthy signal -> fall back to the tail policy (RecoverTornTail recovers; FailClosed strict), same inert posture as an untrusted offset. Composes with the round-5 look-ahead (mid-stream corruption still fails closed) and the round-6 inert-offset handling; runs only on the untrusted path; trusted-SDX3 / no-footer paths untouched. Tail policy (FailClosed for sealed non-tail) threaded in for the fall-back case. Applied at recovery.rs, full_scan.rs, and the compaction copy. The checkpoint/mmap watermark would be a cleaner expected-head but is written only on orderly close() (stale/absent after a crash), so it cannot see the last frame; the entry table is the only seal-time per-frame manifest and it self-authenticates. Stash-verified: the torn-last-committed-frame case returns Ok(prefix) (silent drop) on the pre-fix primitive and Err(CorruptSegment) after. Tests: torn-last-frame + corroborated entry -> fail closed (both policies); intact frames + corrupt footer -> recover all; garbage/empty entry table -> fall back (no false fail-closed); SDX2 legacy -> recover; + a corroboration property test. All round 1-6 tests green (incl. untrusted_footer_torn_last_frame_recovers_prefix for an EMPTY manifest). Logic extracted to recovery_manifest.rs to keep mod.rs under the file budget. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…eate/fsync failure (P1) maybe_rotate_segment wrote the old segment's SIDX footer, wiped sidx_collector, synced the old segment, and advanced durable — THEN did mem::replace with create_with_created_ns(new_id)? (the new-segment file+dir fsync). If that create/ fsync failed, steps 1-4 had already mutated state but mem::replace never ran: the old segment kept a footer, the collector was wiped, and the writer kept running on that half-sealed old segment. The next append wrote frames after the footer with a collector missing the old entries -> silent recovery/compaction corruption. Codex P1. Fix: REORDER (preferred over poisoning — a cleanly-retryable failure beats a dead writer). Create + file/dir-fsync the NEW segment FIRST (while the old segment and collector are pristine), then sync the old segment + advance durable (last fallible steps, still before any footer write / collector reset); everything after is infallible/non-fatal (write footer, best-effort footer flush, reset collector, infallible mem::replace, seal, advance id, notify reader). So a create/fsync OR old-sync failure leaves the writer fully consistent — rotation simply didn't happen, the append errors cleanly, the next append retries against the unchanged old segment. SIDX-footer-before-seal preserved; the round-6 SegmentRotation injection point still fires last, unchanged. New gated fault point InjectionPoint::SegmentRotationCreate (fault.rs) fires before the new-segment create; test segment_rotation_new_segment_create_fault_leaves_writer _consistent (atomic_batch.rs, dangerous-test-hooks) asserts the rotating append fails cleanly, the next in-process append succeeds + rotates, and a clean reopen recovers exactly the committed events in order (no footer-as-frames, no lost SIDX coverage). Existing single_append_fault_at_segment_rotation_recovers unchanged. store::write 27, store::segment 93, atomic_batch (hooks) 25, cold_start 3, clippy + structural clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
harden(0.8.3): audit remediation — silent-corruption / durability / fail-closed hardening
Gives batpak a real event schema-evolution story so an event journaled years and several schema-versions ago still replays. (Phase 1 = Rust core; Phase 2 exposes version on the wire/manifest + TS regen; Phase 3 = durable idempotency.) - EventPayload::PAYLOAD_VERSION: u16 (= 1); derive gains #[batpak(version = N)] (0 is the legacy sentinel, rejected). EventHeader.payload_version: u16 with #[serde(default)] — rides inside the frame msgpack but OUTSIDE the hashed region, so it moves no content_hash and no signature (proven by a test asserting identical hash+signature for the same payload appended typed-v2 vs untyped-v0). Only event_header_v1.hex regenerated. - Stamping: threaded as a scalar (build_event/submit_prepared are impl Serialize, not EventPayload) via versioned funnels; the 8 typed entry points stamp T::PAYLOAD_VERSION. Untyped append/submit/batch/denial/lifecycle = common path = stamp 0 (documented). - Decode/upcast at the single DecodeTyped seam (both JSON + raw lanes): stored==current||0 -> tolerant; stored<current -> run registered Upcast chain over rmpv::Value then decode; stored>current -> hard TypedDecodeError:: FutureVersion. Upcast is in-memory only; stored bytes never rewritten. - Upcast trait + per-(KIND,from_version) inventory registration + chain runner (event/upcast.rs); v1->v2 self-test (rename + new non-defaulted field) proves the chain on both lanes; FutureVersion / legacy-0 / equal-version covered. - Guardrails: assert_frozen_decode (append-only golden fixtures) + check_event_payload_frozen_fixtures structural lint (warn-first, debt-seeded for the 21 manifest payloads). Schema-evolution contract folded into the existing EVENTS.md (no new root doc). - Adds rmpv dep (core + macros-support). Manifest stays unchanged (Phase 1 is manifest-invisible) so the TS drift test stays green. Traceability: INV-EVENT-PAYLOAD-DECODE-BACKCOMPAT + ART entries. lib 314, schema_evolution 12, wire_format 16, full suite + clippy + structural + manifest-drift all green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…TS regen)
Exposes the per-payload PAYLOAD_VERSION on the wire so TS regenerates and
forward-compat-decodes evolved events (owner's wire-field call). Phase 1 kept
this manifest-invisible; Phase 2 surfaces it.
- hbat/manifest.rs: EventDescriptor gains payload_version (serialized
`payloadVersion`); MANIFEST_VERSION 1 -> 2. hbat/descriptor.rs: the
hbat_event_descriptor! macro threads <T as EventPayload>::PAYLOAD_VERSION, so
every derived payload's version is manifest-visible.
- bpk-ts/batpak.manifest.json: regenerated via `cargo xtask export-ts-manifest`
(manifestVersion 2; all 21 events carry payloadVersion). Rust drift test
in_process_manifest_matches_checked_in_bpk_ts_file passes.
- bpk-ts codegen: SUPPORTED_MANIFEST_VERSION 1 -> 2; ManifestEvent.payloadVersion;
validateManifest rejects missing/<1 version (0 = legacy sentinel); emits
<Event>_PAYLOAD_VERSION consts. packages/generated regenerated via pnpm generate.
- bpk-ts client: isCompatiblePayloadVersion / classifyPayloadVersion — tolerant
forward-compat (exact / legacy-0 / older / NEWER all decode; the server upcasts
on read and Effect Struct ignores additive keys; only a malformed value is
rejected), mirroring the NetbatErrorCode `(string & {})` tolerant pattern.
- parity.test.ts + codegen.test.ts updated for the version bump + payloadVersion
(incl. tamper/obsolete-version rejection + forward-compat decode tests).
Verified: Rust drift test PASS; batpak lib 337; hbat full; structural +
traceability ok; xtask export 2; pnpm -w build/test (parity 126)/lint all clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ore)
Make `with_idempotency` a TRUE durable correctness primitive. Previously the
key→receipt dedup was durable only inside an event's retention window: once
`Retention` compaction evicted the event, its `by_id` entry vanished and a
re-run of the same key RE-APPENDED a duplicate. This closes that gap.
Durable sidecar `index.idemp` (magic FBATID, version 1, crc32fast, atomic
write) holding an in-memory DashMap<u128, IdempEntry> with the minimal tuple to
reconstruct the original AppendReceipt even after the event is evicted.
Restored UNCONDITIONALLY and early at cold-start (an authority the segment-scan
rebuild must not overwrite; NEVER reconstructed from segments). Flushed at
close, before snapshot copy, and at the compaction tail. A separate field from
`by_id`, so `replace_contents_from_fresh` cannot clear it — survival across
compaction is structural.
Writer no-op check is now map-first then by_id: a hit reconstructs and re-signs
the original receipt (true no-op even for an evicted event), falling through to
the live index for legacy entries.
Growth bound: the window-priority hybrid. The window is the inviolable
correctness guarantee; the cap (max_keys) may only ever evict keys ALREADY
OUTSIDE the window, and on a residual pigeonhole (within-window keys alone
exceed the cap) the window wins with a loud diagnostic — never a silent dup.
OverflowPolicy::{Warn(default),FailClosed,Backpressure} decides escalation for
genuinely-new keys at the cap. Net invariant (tested): a within-window keyed
retry is ALWAYS a no-op regardless of compaction, cold-start, or load.
IdempotencyKey::for_operation(domain, &[components]) — length-delimited blake3
operation-identity helper (documented as identity, not content-addressing).
Wire: bank.commit gains idempotency_key_hex (Option, serde default, additive
within manifest v2); manifest + bpk-ts generated artifacts regenerated; TS
parity golden updated. Traceability: INV-IDEMPOTENCY-DURABLE-WINDOW + artifacts.
Tests: durable no-op across retention eviction (the killed bug), close/reopen,
eviction+cold-start, snapshot carries the store, CRC/magic/truncation graceful
degradation + FutureVersion hard error, window-priority property (burst over the
cap within the window keeps every within-window key a no-op), for_operation
determinism + boundary-collision resistance. Plus examples/idempotent_pass.rs
and cookbook recipe 200_IDEMPOTENT_PASS.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…lers CI "TS/host profile proof" failed on a clean build: the Phase-3 wire field `idempotency_key_hex` generates as a required-but-nullable TS property, so the two example BankCommitRequest construction sites no longer type-checked (`tsc -b`). Local builds masked it via a stale .tsbuildinfo cache. Add `idempotency_key_hex: null` at both example sites and the one parity test that was incidentally throwing for the wrong reason. Generated schema and golden bytes are untouched — wire byte-parity is preserved (the field is present-as-nil on the wire exactly as Rust's serde-default Option encodes it). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ency All 14 findings (verified-valid; none pre-handled). Correctness: - Reject manual EventPayload PAYLOAD_VERSION == 0 at the typed-append seam (new StoreError::InvalidPayloadVersion + guard on all three versioned funnels) so a real event can never masquerade as a legacy (version-0) frame. - Durable idempotency receipt fidelity: track event eviction with a dedicated IdempEntry.event_evicted flag and leave disk_pos immutable, so a post- compaction keyed retry reconstructs the ORIGINAL receipt position (the old sentinel-in-disk_pos broke the "return original receipt" contract). - Batch idempotency admission: validate + admit the batch key set as a unit (reject duplicate in-batch keys; enforce current+unique_new <= max_keys atomically) instead of per-item checks against the unchanged map. - admit_new_key/admit_new_keys age out-of-window keys before fail-closing, so the soft cap only refuses when the within-window set genuinely fills it. - Durability: flush the correctness-critical index.idemp BEFORE the best-effort cold-start-artifact refresh at both compaction-tail and close, with the flush error propagated (was buried in a warn-swallowed helper after mmap work). - read_idemp_file rejects any version != IDEMP_VERSION (a corrupt version-0 header with a CRC-valid body no longer loads as v1). - Snapshot evidence identity now includes index.idemp presence (schema v1->v2) so a snapshot that loses the dedup authority can't hash-match one that keeps it. - upcast value_from_msgpack rejects trailing bytes after the decoded value. Hygiene/docs: - Pin rmpv exactly (=1.3.1) in core + macros-support to match the "pinned" note. - Drop "Workstream" planning labels from module/test docs. - Strict test setup (.expect over .ok); explicit match over expect_err per the core test guideline. Snapshot golden regenerated via the sanctioned GOLDEN_UPDATE path. All gates green: clippy --all-features --all-targets -D warnings, fmt, structural, traceability, lib + idempotency + dangerous-test-hooks + snapshot/compaction suites, hbat manifest drift. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… note CodeRabbit round-1 nit (the 14th finding): "shape and decodes" -> "shape and decode". Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ilot) First two of the oversize-harness sweep (retiring OVERSIZE_HARNESS_ALLOWLIST). - chaos_testing.rs (1017 -> 419) split by seam into chaos_testing.rs (concurrency/delivery), chaos_testing_byte_corruption.rs, and chaos_testing_batch_rotation.rs; shared iteration helpers in tests/support/chaos_testing.rs. All 12 tests preserved (6+3+3). - store_error_contract.rs (521 -> 87) split into the domain-family binary + store_error_contract_operational.rs (retryable + fail-closed), with the shared Case/HandlingClass/classify/assert_case_contract table in tests/support/store_error_contract.rs. All 27 contract cases preserved; both binaries route through one contract_table() filtered by family so the shared builders are consumed in every binary (no dead-code surface). Each split binary carries its own PROVES/CATCHES/SEEDED header. Ledger locations + cargo-test commands updated; OVERSIZE entries for both files and the stale chaos_testing header-debt entry removed. structural-check, traceability-check, and all split test binaries green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ESS_ALLOWLIST Completes the harness-cleanup sweep (pilot was 61dcc3e). Splits the 10 remaining ledger-governed oversize integration harnesses along their named seams into cohesive per-shape binaries (each <=500 lines) plus shared subdir support modules, then retires the entire OVERSIZE_HARNESS_ALLOWLIST (now `&[]`). Harnesses split (original -> kept stem + new siblings; every #[test] preserved, verified by diffing test-name sets): - control_plane_surface (1055): + _ticket, _fence, _pressure - cursor_durability (578): + _progress - durable_frontier_semantics (1044): + _frontier, _close_regression - durable_frontier_waits (620): + _append_gate - fuzz_chaos_feedback (757): machinery -> support/, single test stays - perf_gates (1350): + _throughput_latency, _cold_start, _correctness - projection_cache (1213): + _freshness, _corruption - raw_projection_mode (923): + _flow_matrix, _incremental - segment_scan_hardening (1406): + _frame_bounds, _untrusted_offset, _untrusted_tail - store_advanced: already a stub from a prior split; stale 1675 entry removed Shared helpers live in tests/support/<harness>.rs (subdir, not a binary) and hold only items used by EVERY split binary; per-family helpers stay inline so no binary carries dead code (pub does not suppress dead_code in a binary crate). Each split binary carries its own PROVES/CATCHES/SEEDED header. Traceability kept in lockstep (the coupling, not the split, was the work): - testing_ledger.yaml: 17 per-fn commands repointed to the binaries tests moved to; new binaries added to locations/commands. - artifacts.yaml: split-sibling paths added so invariant->test citations resolve. - observations.yaml: 4 evidence refs repointed to the files tests moved to. - harness_lints.rs: OVERSIZE_HARNESS_ALLOWLIST emptied; 6 stale header-debt entries removed. - xtask ci perf_gates: runs all four perf binaries (#[ignore]d gates only fire via the --run-ignored job). Re-added a dropped INV-PERFORMANCE-GATES-ENFORCED citation and fixed an invalid LAW-001 justifies anchor to a real INV. All gates green: clippy --all-features --all-targets -D warnings, fmt, structural-check, traceability-check, and the full --all-features test suite (every split binary passes, 0 failed). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Test-quality fixes to the split harnesses (no production code change): - Fix 41 malformed repro hints across the store_* behavior suites where the cargo-test binary and the test-fn filter were concatenated without a space (`--test store_append_behaviorappend_...` -> `--test store_append_behavior append_...`); broad scan confirmed no other split file had the pattern. - Guard the store_error_contract family tests against vacuous success: collect the per-family filtered cases, assert the set is non-empty, then validate (the contract_table()+filter refactor could otherwise pass on an empty family). - Strengthen the torn-last-frame untrusted-offset recovery assertion from `!entries.is_empty()` to `assert_eq!(entries.len(), 6)` (N derived from seed_store(.., 6); the torn frame is the trailing close, so all 6 committed user events survive prefix recovery). - Match the forged victim SYSTEM_CLOSE_COMPLETED frame by full HLC, not wall_ms alone: also compare the HLC clock (IndexEntry::clock() == DagPosition:: sequence()), so two closes in the same millisecond can't mis-match. (The index commit global_sequence is not present in the frame, so it cannot be the tiebreak — verified by probing on-disk values.) - Retarget the perf_gates_cold_start `#[ignore]` repro hint to its own binary (`--test perf_gates_cold_start`). Deferred (pre-existing, tracked separately): contract_table() does not yet cover every classify()'d StoreError variant. All gates green: clippy --all-features --all-targets -D warnings, fmt, structural-check, traceability-check, affected suites pass. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
feat(0.8.3): event schema evolution + durable idempotency (pre-release)
chore(0.8.3): split oversize test harnesses, retire OVERSIZE_HARNESS_ALLOWLIST
…n fields A Rust `#[serde(default)] Option<T>` field is additive on the wire (absent decodes to None), but the codegen rendered every `option<…>` as `Schema.NullOr(...)` — a REQUIRED-but-nullable TS property forcing callers to write `field: null`. This makes such fields omittable on the TS input side while keeping the encoded wire form byte-identical (present-nil), so byte-parity with Rust's `to_vec_named` None→nil is preserved. Manifest: `FieldDescriptor`/`FieldRow` gain `optional: bool` (`#[serde(default)]`, additive within manifest v2); the hbat descriptor macro accepts an optional third tuple element. Audited all 23 `option<…>` manifest fields — exactly one is `#[serde(default)]` (`bank.commit.idempotency_key_hex`), marked `optional: true`; the rest are plain `Option` (absent → decode error) and stay required-nullable. Codegen: optional fields render via an `optionalNullable` helper — `NullOr(inner)` on the wire, `decodeTo` an `optionalKey(NullOr(inner))` Type with an encode-direction default that fills `null` for a missing key. Effect 4 beta has no `optionalWith`; the default/optional constructors all drop the key on encode (would break parity), so this transform is the construct that keeps the field present-nil when omitted. `: null` stays valid too (backward-compatible). Byte-parity proof: regenerated goldens are byte-identical; parity suite 126→127 with a new test asserting an OMITTED `idempotency_key_hex` encodes to the same `…idempotency_key_hex c0` golden. Dropped the `idempotency_key_hex: null` workarounds from the two examples + the parity throw-test. Gates green: hbat manifest drift test, structural-check, traceability-check, pnpm -w build + test (clean tsbuildinfo). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…guard Closes the coverage gap between classify() and contract_table(): ~24 StoreError variants were classify'd but had no Case row, so their handling-class / Display / source contract could drift silently. Adds a Case per uncovered variant (verified against error.rs Display + source() impls), routed into the matching family builder: - Domain: Coordinate(+NulByte/PathTraversal/ControlChar), CheckpointId, EventPayloadRegistry, InvalidPayloadVersion, IdempotencyRequired, IdempotencyPartialBatch, VisibilityFenceActive/NotActive/Cancelled, RangeMalformed, InvalidCausation, InvalidCommitMetadata, EntityClockOverflow - Retryable: IdempotencyOverflowFailClosed (soft-cap backpressure — eviction can free capacity, so a later retry succeeds; grouped with WaitTimeout) - FailClosed: WriterCrashed, SegmentTooManyEntries, DataDirMalformed, AncestryCorrupt, IdempotencyFutureVersion, FaultInjected (cfg-gated) classify() is now explicit-per-variant (the 5 previously-unclassified variants got real arms instead of hitting the wildcard panic). StoreError is #[non_exhaustive], so a wildcard arm is mandatory and full compile-time exhaustiveness from the test crate isn't achievable; the new `every_store_error_variant_has_a_contract_case` test builds one representative of every variant (no catch-all match, so the list can't silently drift) and asserts each discriminant is present in contract_table() — turning a forgotten Case into a test failure. classify()'s panicking wildcard remains the runtime backstop for a variant added inside core itself. Gates green: contract suites (default + dangerous-test-hooks), store::error lib tests, clippy --all-features --all-targets -D warnings, structural-check, traceability-check. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
#118) A stray `"true"`/`1` for `field.optional` would slip past the `=== true` check and silently emit the non-optional schema. Fail fast with a CodegenError when `optional` is present but not a boolean. Adds codegen guard tests for both the new invalid_optional_flag check and the pre-existing (untested) optional_non_option_field check (codegen suite 26 -> 28). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…teness test(0.8.3): complete StoreError contract table + exhaustiveness guard
feat(0.8.3): codegen emits omittable TS input for serde-default option fields
Post-merge sanity sweep (two read-only audits) found and fixes real drift; the machine-checked surface (structural/traceability/docs-contract) was already consistent. Stale counts corrected to reality: - README.md: catalog tally 71 invariants / 119 artifacts -> 73 / 124. - crates/core/README.md: 71 -> 73 named invariants. - bpk-ts/README.md: codegen 24 -> 28 direct tests; parity 120 -> 127; total 211 -> 220 tests across packages. CHANGELOG: [Unreleased] documented only the #115 integrity work; added an `### Added` section for the headline 0.8.3 features it was missing — event schema evolution (PAYLOAD_VERSION/upcast/FutureVersion/manifest v2), durable idempotency (index.idemp sidecar, window-priority IdempotencyRetention + OverflowPolicy, for_operation, bank.commit idempotency_key_hex), and the omittable-optional TS codegen. Folded the two headline guarantees into the canonical narrative docs (they are major invariants now, not ceremony): INVARIANTS.md gains "Payload Shape Evolves On Read" and "Idempotency Is Durable"; REPLAY.md notes payload decoding is versioned (older upcast in memory, newer = hard error / fail closed). Verified clean (no change needed): EVENTS.md (canonical schema-evo + idempotency coverage), cookbook/200_IDEMPOTENT_PASS.md, examples/idempotent_pass.rs (compiles), all cookbook recipe references, manifest v2, no dangling links, no references to the now-empty oversize allowlist. structural-check + traceability-check green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
docs(0.8.3): currency pass — stale counts + CHANGELOG + headline-invariant narrative
…RITY, PR template Adds a Project Direction section to CONTRIBUTING.md (maintainer-led; RFC-first for public-API/architecture/dependency/invariant changes; the engineering doctrine as the explicit contribution bar; AI-assisted-but-understood rule). Adds CODEOWNERS (maintainer on all paths, crown-jewel surfaces called out), SECURITY.md (private vulnerability reporting + pre-1.0 support policy), and a PR template carrying the doctrine checklist. Docs/config only, no code. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018kcV9VfchXfpzvFvXhszBn
Qodo reviews are paused for this user.Troubleshooting steps vary by plan Learn more → On a Teams plan? Using GitHub Enterprise Server, GitLab Self-Managed, or Bitbucket Data Center? |
|
Warning Review limit reached
More reviews will be available in 29 minutes and 55 seconds. Learn how PR review limits work. Your organization has used up its prepaid credits, and credit purchases are no longer available. Enable the review add-on in the billing tab to keep reviews running — you're only billed for reviews past your plan's rate limits ($0.25/file). ⌛ How to resolve this issue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based credits. 🚦 How do rate limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan refill rate. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, the refill rate gradually slows as usage increases. The highest same-day bursts are limited more strictly. Please see our Fair Usage Limits Policy for further information. ℹ️ Review info⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (4)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds the governance/direction layer that the repo was missing (existing CONTRIBUTING.md only covered the mechanical workflow).
Docs/config only — no code. First PR through the new
main-crown-guardruleset (PR required, CI-fast required, linear history, no force-push).