Skip to content

moonpool-engine.md

Latest commit

 

History

History
382 lines (287 loc) · 52.5 KB

moonpool-engine.md

File metadata and controls

382 lines (287 loc) · 52.5 KB
Raw

Moonpool Engine

magnetar-runtime-moonpool is the provider-generic engine used for deterministic simulation. It drives the same sans-io magnetar-proto::Connection state machine as the tokio engine while routing networking, time, task scheduling, randomness, and storage through Moonpool providers.

This document is the canonical description of the Moonpool 0.8 runtime boundary, engine surface, TLS adapter, deterministic chaos pack, and differential equivalence harness.

For the production engine and the workspace-wide architecture, see ../ARCHITECTURE.md (its Overview section is the 10-minute read).

What moonpool is

moonpool-sim 0.8 is a deterministic simulation engine with a single-threaded seeded executor. Application code talks to [moonpool_core::Providers], a bundle of:

  • NetworkProvider — TCP-shaped byte pipes.
  • TimeProvider — virtual or wall-clock time.
  • TaskProvider — Tokio or deterministic-executor task spawning.
  • RandomProvider — seeded RNG.
  • StorageProvider — file I/O.

Under simulation each provider is virtualised so a given seed replays bit-for-bit. magnetar-runtime-moonpool plugs the engine onto a Providers bundle of the caller's choosing:

Provider bundle Task execution Time and I/O Use
[moonpool_core::TokioProviders] Ambient Tokio runtime Wall clock, real network, host storage, real RNG Production-style and differential real-broker runs.
moonpool_sim::SimProviders Moonpool's seeded deterministic executor Virtual clock, scripted network/storage, seeded RNG Reproducible chaos without an ambient Tokio runtime.

The published library target depends on moonpool-core; the crate's test suite adds moonpool-sim as a development dependency and plugs SimProviders into the same engine.

Determinism boundary

ADR-0078 makes the provider boundary authoritative for code that runs under either provider bundle.

  • Runtime tasks are spawned through TaskProvider::spawn_task.
  • Sleeps and timeouts run through TimeProvider::sleep or TimeProvider::timeout.
  • Concurrent waits use moonpool_core::select!. Its fair form draws the starting branch from Moonpool's seeded source; biased; keeps explicit source order where protocol fairness or shutdown priority requires it.
  • Network connects and byte streams come from NetworkProvider.
  • tokio::sync::Notify remains the payload-free wakeup primitive, but no provider-generic path depends on a Tokio reactor merely to park or wake a future. Application-side readiness waits reuse the existing topic_list_notify wake bus so they cannot consume driver permits and the public ConnectionShared field layout remains source-compatible.

TokioProviders intentionally maps those operations to Tokio. SimProviders maps them to Moonpool's native executor, virtual clock, and simulated network, so the same (commit, seed) replays task interleavings, timer races, network faults, and fair selection order.

Simulation observability uses the production tracing vocabulary. Actors emit constant-name events with flat structured fields inside a span carrying ip; invariants read TraceEvent values through TraceQuery::since or TraceQuery::snapshot. Moonpool 0.8 therefore requires no TrailQuery, TrailQueryExt, Valuable, or Serde payload bridge.

Engine surface

MoonpoolEngine<P: Providers> exposes these entries:

Method Role
MoonpoolEngine::new(providers: P) Construct the engine over a Providers bundle.
connect_plain(addr, config) Plain TCP connect + handshake. Returns (Arc<ConnectionShared>, DriverHandle).
connect_plain_with_resolver(addr, config, resolver) Plain TCP via injected DnsResolver.
connect_tls(addr, server_name, tls_config, config) TLS via the in-crate rustls byte-pipe adapter (tls.rs).
connect_plain_supervised(addr, config, service_url_provider, reconnect) Plain TCP wrapped in the supervised reconnect loop.

The user-facing client lives at magnetar-runtime-moonpool::Client<P>, mirroring the tokio engine's Client surface: connect_plain, connect_plain_supervised, partitioned-metadata lookup, transaction coordinator helpers, is_connected, close. Client::from_parts remains the Tokio-backed convenience constructor for an externally-created (ConnectionShared, DriverHandle) pair. Deterministic simulation and custom provider users must call Client::from_parts_with_providers so consumer receive deadlines inherit P::Time instead of an ambient Tokio clock. The provider-owned sleep function lives in the private Client / Consumer runtime state rather than in the public ConnectionShared layout.

At the façade layer the engine is selected via the Engine marker trait, so PulsarClient<MoonpoolEngine<P>> is the canonical public type (ADR-0019). The higher-level façade surfaces (partitioned, multi-topics, pattern, reader, table-view, transactions, typed schemas) were lifted to be engine-generic over E: Engine, so they build on both engines; only a few narrow tokio-only specialisations remain. See ../README.md#engine-by-engine-surface-coverage for the authoritative per-feature, per-engine snapshot.

Apache Pulsar Proxy connection pool

ADR-0039 (amended 2026-06-01) lands the per-broker connection pool on the moonpool engine. The pool lives at crates/magnetar-runtime-moonpool/src/pool.rs and mirrors crates/magnetar-runtime-tokio/src/pool.rs 1:1.

The pool is populated only when the client is built via Client::connect_plain_supervised — that constructor wraps the bootstrap connect inputs (proxy address, ConnectionConfig template, Providers bundle, optional ServiceUrlProvider + DnsResolver) into a ConnectionFactory<P> and hands it to a fresh ProxyConnectionPool<P>. The Client::resolve_target hook then routes any LookupOutcome::Connect { proxy_through_service_url = true, .. } to the pool via the pool::get_or_open(Arc<Self>, logical_broker_url) async free function, which:

  1. Probes the entries map; on a hit, returns the cached Ready entry.
  2. On a miss, installs a Pending(PendingDial) slot and spawns one dial through TaskProvider::spawn_task. This keeps the single-flight ownership model identical under TokioProviders and SimProviders while racing callers await the same result.
  3. The spawned dial task runs network.connect → handshake_plain → spawn_supervised, then publishes the Arc<Result<Arc<ConnectionShared>, EngineError>> into the PendingDial::result slot and fans the result out via Notify::notify_waiters. Racing waiters all Arc::clone the same outcome.
  4. The dial task promotes the entry only when the map still contains the identical Pending generation and the pool is open; stale or post-close successes are closed and joined instead of replacing a newer entry.
  5. A normal failure evicts only its own generation so a detached older task cannot remove a newer dial.
  6. The spawned task owns the single provider-native operation_timeout around connect plus handshake, so a silent peer terminates the task and drops its socket instead of timing out only the caller.

Client::close drains every Ready pool entry's supervised driver in addition to the bootstrap. Pending entries resolve their waiters with EngineError::PeerClosed. Close signals the pending dial's cancellation notification and awaits its completion notification before returning. The latched closed state and generation check prevent any detached dial from resurrecting the entry; a late successful connection is closed and its driver is joined by the dial task.

Per-broker ConnectionConfig.proxy_to_broker_url is set on the cloned config inside build_entry_async; the bootstrap config itself stays untouched, so the bootstrap connection's CommandConnect omits the field (matching the Java client + Pulsar Proxy contract).

Producer + consumer façades

magnetar-runtime-moonpool::Producer<P> and magnetar-runtime-moonpool::Consumer<P> mirror their tokio counterparts. The two engines share the same sans-io state machine, so the public method shape (send / flush / close / stats / ack variants / nack / seek / pause / DLQ drain) is identical. The difference is which now: Instant source the engine snapshots at the call site and which byte pipe carries the wire bytes.

PIP-4 message-crypto bridge

The moonpool engine ships the PIP-4 end-to-end encryption bridge, mirroring the tokio engine exactly (ADR-0044). crypto.rs defines the engine's MessageEncryptor / MessageDecryptor traits + EncryptError, the moonpool counterparts of magnetar-runtime-tokio::crypto. The façade's MessageCryptoBridge (crates/magnetar/src/crypto_bridge.rs) implements both engines' trait pairs over magnetar-messagecrypto::MessageCrypto, so the same bridge value plugs into either engine's builders.

  • Producer (encrypt-on-send). The moonpool producer encrypts the payload, stamping pb::MessageMetadata encryption_keys / encryption_algo / encryption_param. This mirrors the tokio producer's compression → encryption ordering for the encryption step; compression itself is not yet wired on the moonpool engine — non-None CompressionKind is refused on send until the runtime codec lands (M3) — so in practice the moonpool path is encrypt-only.
  • Consumer (decrypt-on-receive). The moonpool consumer decrypts the payload — honoring the three CryptoFailureAction arms (Fail, Discard, Consume) identically to tokio — then delivers it. Because compression is refused on send, there is no decompression step to mirror: the receive path reduces to decrypt, then deliver (tokio's decrypt-first → decompress ordering, with the decompress branch a no-op on moonpool until codecs land).

The façade builders gain .encryption() / .create_with_encryption() (producer) and .encryption() / .subscribe_with_decryption() (consumer) for the moonpool engine, routing through the new Client::open_producer_with / Client::subscribe_with entries. The engine crypto API (MessageEncryptorApi / MessageDecryptorApi) is now non-stub for both engines; NoEncryption is retained only as the documented opt-out (the resolved API when no bridge is supplied), not as the moonpool default. Equivalence is asserted through the differential harness per ADR-0024 — see the differential equivalence harness section and testing.md.

Transport + vectored writes

The engine's transport adapter (crates/magnetar-runtime-moonpool/src/transport.rs) drives the moonpool_core::NetworkProvider::TcpStream directly. Moonpool 0.8's stream bounds use the futures::io::{AsyncRead, AsyncWrite} traits rather than tokio::io (ADR-0078). TokioNetworkProvider wraps its tokio::net::TcpStream in tokio_util::compat::Compat to bridge the two ecosystems. The transport adapter therefore imports the futures::io ext traits (AsyncReadExt / AsyncWriteExt) accordingly.

The read side carries a reusable heap-backed scratch (read_scratch, a Box<[u8]> of TLS_WIRE_BUFFER bytes allocated once per Transport via new_read_scratch()): read_into lands wire bytes into it / the caller's spare capacity instead of heap-allocating a fresh 16 KiB buffer on every read. The scratch lives on the heap rather than as a stack array so the returned read future stays small (a stack array tripped clippy's large_futures). Perf-only — no behaviour or wire change.

The driver dispatches the sans-io TransmitOwned descriptor (ADR-0040) as follows:

TransmitOwned arm Transport Behaviour
Vectored on the plaintext path under SimProviders futures::io::AsyncWriteExt::write_vectored over SimTcpStream Segment-granular. moonpool records each IoSlice as its own ordered delivery event, with writev-style partial-accept semantics — the chaos pack can drop / re-order individual segments.
Vectored on the plaintext path under TokioProviders futures::io::write_vectored over the Compat wrapper Single-write fallback. The Compat stream does not forward vectored writes (is_write_vectored() is false), so the slices collapse to one buffer write. Byte-identical wire output, no syscall reduction.
Contiguous (handshake, small frames) single-buffer write_all unchanged.
Vectored on the TLS path Transport::write_all_vectored coalesces, then writes ciphertext Always contiguous. The TLS arm still receives the segment list, but pushes each segment's plaintext through rustls in order and ships one ciphertext stream — rustls owns its own record buffering, so segment boundaries cannot survive encryption. See the TLS adapter section below.

This replaces the earlier placeholder that coalesced the Vectored segment list into one contiguous write_all "until moonpool-core adds vectored support" — that prerequisite is now satisfied (ADR-0040, PierreZ/moonpool#111 / PR #113).

Supervised reconnect

The moonpool driver loop mirrors the tokio supervisor exactly. See ../ARCHITECTURE.md#the-driver-loop for the shared algorithm. Specifics for the moonpool engine:

  • Backoff is driven by moonpool_core::TimeProvider::sleep — under SimProviders the deterministic executor advances the virtual clock to the next scheduled event.
  • DNS is re-resolved on every attempt through the injected DnsResolver. The crate ships StaticDnsResolver and an arc_dns_resolver helper.
  • The ServiceUrlProvider is consulted on every attempt before Transport::connect, so ControlledClusterFailover plugs straight in (see PIP-121 below).
  • After re-handshake the engine calls Connection::rebuild_producers(now) and Connection::rebuild_consumers(now) to re-issue CommandProducer / CommandSubscribe for every still-open handle.

TLS adapter

The moonpool engine cannot use tokio-rustls — tokio-rustls needs a real socket. Instead it drives a sans-io rustls::ClientConnection by hand over the byte pipe supplied by moonpool_core::NetworkProvider. The adapter lives at crates/magnetar-runtime-moonpool/src/tls.rs and follows the standard rustls "drive it yourself" pattern:

socket.read(buf)                  →  session.read_tls(buf)
                                  →  session.process_new_packets()
                                  →  session.reader().read_to_end(plaintext_in)
plaintext_out                     →  session.writer().write_all(...)
                                  →  session.write_tls(socket_out)
socket.write_all(socket_out)

The handshake therefore stays deterministic under SimProviders chaos (connection drops, partial reads, virtual-clock timeouts). The adapter never blocks on a network call inside process_new_packets — reads and writes go through the byte pipe under simulation control.

The TLS write path is always contiguous, including for producer batches the plaintext path would emit as a Vectored segment list (ADR-0040): rustls buffers and frames its own records, so per-segment boundaries cannot survive encryption. The driver still dispatches Vectored to Transport::write_all_vectored for TLS connections, but the TLS arm coalesces the segment list — pushing each segment's plaintext through rustls in order — before shipping one ciphertext stream. The segment-granular write_vectored benefit therefore applies to the plaintext arm only — see the Transport + vectored writes table.

See ADR-0006 for the binding decision.

ServiceUrlProvider plumbing (PIP-121)

The supervised reconnect path consults the configured ServiceUrlProvider on every attempt. Two implementations live in magnetar-proto (and are therefore usable by both engines):

  • StaticServiceUrlProvider — single URL, never changes.
  • ControlledClusterFailover — Arc<Mutex<String>> swappable at runtime via set_url(...). Tests or sidecars drive failover by swapping the URL between reconnects.

AutoClusterFailover<P> (PIP-121 health-probe-driven) ships on the moonpool engine as well — the probe loop runs on P::TaskProvider, so the simulator drives the schedule deterministically with no real DNS or TCP. Source: crates/magnetar-runtime-moonpool/src/auto_cluster_failover.rs.

PIP-188 TOPIC_MIGRATED

magnetar-proto::Connection::handle_bytes decodes CommandTopicMigrated and emits ConnectionEvent::TopicMigrated on the event queue. The moonpool driver consumes the event, logs the new-URL hint, and returns an error from driver_loop_inner — exactly the mechanism used by the tokio engine. The supervisor catches the error, calls Connection::reset(), and reconnects against the migrated broker. See ADR-0018.

Deterministic chaos pack

crates/magnetar-runtime-moonpool/tests/ ships a chaos test pack that exercises the supervisor + reconnect + PIP-121 + PIP-188 paths under deterministic seeds. Tests are normal cargo test integration targets — no Docker, no live broker. The sim_chaos.rs workload runs inside Moonpool's native deterministic executor and asserts invariants over named flat tracing events captured by TraceQuery. Cross-event temporal invariants compare TraceEvent::seq, the global per-seed sequence, rather than relying on the order in which per-name snapshots are queried.

Scenario Test
Mid-handshake network partition mid_handshake_partition.rs
Out-of-order frame delivery frame_reorder.rs
OAuth2 token refresh edge cases oauth_refresh_edge.rs
PIP-121 oscillation (primary → standby → primary) pip_121_oscillation.rs
PIP-188 migrate-then-migrate-again pip_188_migrate_then_migrate_again.rs
Reconnect with in-flight publishes reconnect_with_inflight.rs
Virtual-clock ack-timeout fires virtual_clock_ack_timeout.rs
Virtual-clock send-timeout fires virtual_clock_send_timeout.rs
ADR-0028 anti-thrash policy (broker ack-then-drop cascade) anti_thrash.rs
Supervised redial under a drop → accept → drop → accept cycle (anti-thrash cooldown + multi-attempt redial body) supervised_redial.rs (mirror: tokio side)
Stateful broker + invariant assertions (D2 chaos pack) sim_chaos.rs
Swarm configurations — per-seed subset of buggify labels + workload operations (ADR-0097) swarm_config.rs (mirror: tokio side)
Targeted ADR-0024 coverage closure for src/{driver,producer,consumer,lib,transport}.rs coverage_close.rs (mirror: tokio side)
Delayed-marker replicated-subscription harness (enroll-before-drain marker-accessor lost-wakeup race, ADR-0034) replicated_subscriptions_sim.rs (moonpool-only SimProviders, parity-exempt)
Bounded PIP-37 chunk reassembly — cap-eviction of the oldest incomplete buffer (ADR-0063) chunk_reassembly_bound.rs (mirror: tokio side)
TLS handshake byte-level chaos — corrupt-record rejection through RustlsByteAdapter tls_handshake_chaos.rs (mirror: tokio side)

Since the engine dispatches plaintext producer batches through real write_vectored (see Transport + vectored writes), the chaos pack now operates at segment granularity on the plaintext arm: SimTcpStream records each IoSlice as its own ordered delivery event with writev-style partial-accept semantics, so per-segment drop / re-order / short-write modelling is available where the pack previously saw only one coalesced write. The TLS arm stays contiguous, so its chaos fidelity is unchanged (rustls owns record buffering).

cargo run -p xtask -- check-runtime-test-parity (ADR-0024) skips the four PARITY_EXEMPT_FILES (sim_chaos.rs, src/pool.rs, proxy_multi_conn.rs, replicated_subscriptions_sim.rs) — moonpool-only harnesses with no tokio twin — so the strict 1:1 tokio ↔ moonpool count holds on the non-exempt set.

Every SimulationBuilder::new() test — which is every test in this chaos pack, sim_chaos.rs included — inherits moonpool-sim's default-on network chaos: bit-flip corruption, probabilistic connect failure, and buggified delay, all active from NetworkConfiguration::default(). This is a deliberate, binding decision, not an oversight: see ADR-0055, which considered and rejected both disabling bit-flip for magnetar's sim and making moonpool's bit-flip opt-in, and instead mandates hardening the client and the workloads. Every sim workload's setup path (initial connect, subscribe, open_producer) must therefore be wrapped in retry_setup / retry_supervised_connect (both defined near the top of sim_chaos.rs) so a transient chaos-induced drop during setup is retried instead of failing the whole iteration. As a verified-but-out-of-scope observation, ChaosConfiguration::disabled() exists in moonpool-sim but is unreachable from SimulationBuilder's public API — ChaosMode offers only Random and Swarm — so opting out of chaos is not available today.

Reproduce a flaky run under a specific seed:

MOONPOOL_SEED=0xdeadbeefcafebabe \
  cargo test -p magnetar-runtime-moonpool \
    --no-default-features --features crypto-aws-lc-rs \
    --locked -- --nocapture

Sweep a range of seeds locally:

for seed in $(seq 1 32); do
  MOONPOOL_SEED=$seed cargo test -p magnetar-runtime-moonpool \
    --no-default-features --features crypto-aws-lc-rs \
    --locked -- --quiet || { echo "seed $seed FAILED"; exit 1; }
done

In CI, the per-PR / per-push pipeline (.github/workflows/ci.yml) exercises the moonpool suite under the default seed via the regular test job. A dedicated moonpool-seed-sweep.yml workflow runs daily with 128 freshly-rolled random u64 seeds in parallel — see ADR-0036 for the rationale (fixed seeds in per-PR CI are wasted compute since each (commit, seed) pair is bit-for-bit reproducible; random seeds rolled daily cover the seed space far better over time). Failing seeds are echoed in the run summary — reproduce locally with MOONPOOL_SEED=<hex> cargo test -p magnetar-runtime-moonpool ….

Differential equivalence harness

magnetar-differential is a test-only crate that runs a producer/consumer Trace (a sequence of operations — connect, open producer, send, subscribe, receive, ack, seek, close) against both engines and compares the user-visible EventStreams for equivalence.

The harness components:

File Role
broker.rs Scripted in-process Pulsar broker speaking a minimal subset of the wire protocol: CONNECT/CONNECTED, PRODUCER/PRODUCER_SUCCESS, SEND/SEND_RECEIPT, SUBSCRIBE/SUCCESS, pushed MESSAGE, ACK/ACK_RESPONSE, SEEK/SUCCESS, CLOSE_PRODUCER/CLOSE_CONSUMER. Round-trips PIP-4 MessageMetadata encryption fields verbatim (mirroring a real broker's PIP-4 opacity).
trace.rs Trace (operations) and EventStream (user-visible outcomes). The …Shared op family is OpenSharedConsumer (carrying receiver_queue_size and max_redeliver_count), RecvShared, AckShared, AckLastReceivedShared, NackShared, DrainDeadLettersShared, CloseSharedConsumer and ResubscribeShared. SharedConsumerWindow groups the two per-consumer subscribe knobs into the single parameter both runners' open_shared_consumer takes, keeping that signature under clippy::too_many_arguments.
runner_tokio.rs Runs a trace against magnetar-runtime-tokio bound to 127.0.0.1.
runner_moonpool.rs Runs the same trace against magnetar-runtime-moonpool with TokioProviders.
tests/golden_traces.rs Asserts the two engines produce equivalent event streams on the shipped golden traces.
tests/crypto_roundtrip_equivalence.rs PIP-4 encrypted round-trip parity across both engines (ADR-0044).
tests/crypto_failure_action_equivalence.rs The 3-arm cryptoFailureAction matrix (Fail / Discard / Consume), pinned by golden trace tests/golden/crypto_failure_action.json.
tests/lookup_redirect_chain_equivalence.rs Redirect-target dialing across a multi-broker lookup chain (ADR-0039 amendment).
tests/message_listener_delivery_equivalence.rs MessageListener push-delivery parity for the single-topic / typed consumer (ADR-0064).
tests/wrapper_message_listener_delivery_equivalence.rs MessageListener push-delivery parity for the wrapper consumers (multi-topic / partitioned / pattern, ADR-0064).
tests/chunk_reassembly_bound_equivalence.rs Bounded PIP-37 chunk reassembly — cap-eviction parity (ADR-0063).
tests/failover_active_reflow_equivalence.rs Failover active-promotion flow rearming plus accepted incomplete PIP-37 chunk-flow replenishment parity; extended for the ConsumerEventListener active-state surface (issue #348, ADR-0081) — is_active trajectory and drained active_changes transitions across promote / redundant-promote.
tests/nack_unacked_removal_equivalence.rs Nacked ids dropped from the ack-timeout tracker (no double redelivery) parity.
tests/batch_redelivery_flow_equivalence.rs Issue #436 — ack-timeout redelivery of a partially-acked BATCHED entry on a Shared subscription: delivered positions, mark-delete advance, and per-message broker permits.
tests/dead_letter_flow_refund_equivalence.rs Issue #437 — a dead-lettered dispatch unit returns its flow permit at routing time, so the entry published behind it is still dispatched on a one-permit window (ADR-0107). Adds max_redeliver_count to Op::OpenSharedConsumer, plus Op::NackShared and Op::DrainDeadLettersShared / Event::DeadLettersDrained.
tests/broker_close_producer_reattach_equivalence.rs Issue #451 — a broker CommandCloseProducer on a connection that stays up: the suppressed ProducerClosedByBroker, the in-place CommandProducer at epoch 1, the shut send-drain gate, the deferred-then-flushed publishes, and the two refusal branches (unknown producer id, close during a pending open). Both assigned_broker_service_url shapes take the same in-place path (ADR-0106).

The differential runner intentionally uses TokioProviders rather than SimProviders because both legs talk to the same real in-process broker on the same wall-clock runtime. The separate SimProviders chaos pack exercises Moonpool's deterministic executor, virtual clock, and simulated network. Together they distinguish user-visible cross-engine equivalence from simulation-scheduler coverage.

Equivalence holds across the vectored-write change because the comparison is on wire bytes + user-visible events, not syscall shape: under TokioProviders the moonpool transport's Compat stream does not forward vectored writes (it collapses the Vectored segment list to a single buffer write — see Transport + vectored writes), so it emits byte-identical wire output to the tokio engine's write_all. The segment-granular delivery events are a SimProviders-only refinement and do not perturb the TokioProviders-backed differential trace.

The Moonpool runner awaits engine work directly on the ambient Tokio runtime. Moonpool 0.8's TokioTaskProvider uses tokio::spawn, while SimTaskProvider uses Moonpool's deterministic executor; both satisfy the same Send-bound TaskProvider contract.

What is not yet exercised under simulation

  • Property-based seed sweeps in per-PR CI: the per-PR pipeline runs the test binary on the moonpool default seed only. Multi-seed scheduling is covered by the daily 128-random-seed sweep (ADR-0036), not by per-PR CI.
  • Adversarial in-handshake byte mutation under SimProviders network chaos is not yet swept; corrupt-record rejection is covered by tls_handshake_chaos.rs on both engines (1:1), but mutating handshake bytes mid-flight as a network-chaos scenario is open work.

When one of these items moves from "known gap" to "ready to dispatch", it is added to follow-ups.md with the standard Gap / Why it stays open / /goal entry shape.

Appendix — reference patterns: FoundationDB and TigerBeetle

Audience. Engineers evaluating where magnetar's deterministic simulation infrastructure should evolve next. This appendix is a research note, not a binding spec — for binding decisions see ../specs/adr/.

Magnetar's simulation strategy is informed by two reference systems: Apple FoundationDB's simulator and TigerBeetle's VOPR. The current surface (chaos pack, differential harness, daily seed sweep) is documented above; this appendix captures the patterns that drove it and the ones that motivated ADR-0047, ADR-0048, ADR-0049, and ADR-0050.

FoundationDB simulator (the reference implementation)

The FoundationDB simulator is the canonical example of "the test strategy that made it possible to ship a production distributed database with a small team." Source: apple.github.io/foundationdb/testing.html.

Determinism architecture

  • Single-threaded Flow execution. FoundationDB is written in Flow, an actor-based language atop C++. The simulator runs the full cluster (all servers + all clients) in a single OS thread. No threading primitives, no preemption — every interleaving is a deterministic function of the seed.
  • Synchronized time stepping. The simulator advances a virtual clock and dispatches actor wake-ups in deterministic order. Real durations are compressed (~10×) so a "one-day" outage in simulation completes in a few minutes of wall time.
  • Production code IS the test target. Flow is the same language used in production binaries. There is no separate "mock" — the simulator replaces the I/O / time / random primitives only.

Fault injection — "buggify"

  • Buggify points are explicit if (BUGGIFY) { ... } blocks spread throughout the production code: rare delays, dropped messages, partial writes, restarts. Under simulation each buggify-block fires with controlled probability per seed; in production they never fire. Magnetar's equivalent landed as ADR-0048 — feature-flagged #[cfg(feature = "buggify")] blocks at four choice points in magnetar-proto.
  • Multi-layer faults: network (packet loss, reorder, partition, delay), machine (process crash, reboot, slow disk, full disk), datacenter (full-DC partition, asymmetric routing). Each layer is modelled independently and composes.
  • Swizzle-clogging: stop random subsets of nodes' network traffic, then restart them in a different random order. Exposes reconnection-ordering bugs that pure crash-restart misses. Landed as ADR-0050.

Volume + workloads

  • "Tens of thousands of simulations every night." A new commit is expected to soak through that swarm before reaching production.
  • Workload reuse: the same workload definitions drive performance tests (real cluster, real time) and simulation (virtual cluster, virtual time). One spec, two regimes.

TigerBeetle — the assertion-first philosophy

TigerStyle is the explicit set of coding rules that make deterministic simulation actually work on TigerBeetle's codebase. It is not just about the simulator — it's about how production code is written so that simulation discovers bugs cheaply.

Coding rules that make simulation effective

  • Assertion density ≥ 2 per function. Pre/postconditions, invariants, compile-time relationships. Assertions downgrade silent correctness bugs into loud liveness bugs (crashes), which the simulator catches immediately. Magnetar's equivalent landed as ADR-0049 — pair-assertions on Connection state machine entries.
  • Pair assertions (positive + negative space). Don't just assert what you expect — also assert what you don't. "Data movement across trust boundaries" gets both sides asserted.
  • Run-to-completion functions. Functions that don't suspend preserve their preconditions throughout the body — no need to re-assert after every await point. Maps directly to magnetar's sans-io Connection entries: handle_bytes(now, &[u8]) runs to completion under the caller's lock.
  • Static memory only on hot paths. No heap allocations after startup — preallocate all buffers. This rule does not transfer to magnetar: we use Vec<u8> buffers for arbitrary-sized Pulsar payloads, and Rust's allocator is fast enough that pre-allocation is not the lever it is on TigerBeetle's small fixed-size messages.
  • No shared mutable state between actors. Each actor owns its state; message-passing for coordination. Magnetar enforces the no-channels variant via ADR-0003 (Waker-slab pattern as the closest Rust analog).

VOPR — the simulator

VOPR (Viewstamped Operations Replicator) is TigerBeetle's simulator. Key properties:

  • VOPR is the final line of defence, not the first. "Assertions are a safety net, not a substitute for human understanding." Engineers reason about correctness first; VOPR catches the residual.
  • Single-threaded simulation of a full replica set. Same pattern as FoundationDB.
  • Deterministic state-machine fuzzing. Random client workloads + random network faults + assertion density = bugs found in minutes that would take days of customer traffic.
  • VOPR runs continuously on dedicated hardware. Higher throughput than nightly sweeps because the cost of one bug escaping to production is operationally catastrophic.

Status: pattern adoption in magnetar

Pattern Source Status
Buggify points in magnetar-proto FDB Landed (ADR-0048).
Assertion density in magnetar-proto TigerBeetle Landed (ADR-0049).
Swizzle-clog workload in sim_chaos FDB Landed (ADR-0050).
Per-handle invariant assertions TigerBeetle Landed (HandleResolutionInvariant).
Failing-seed registry per PR FDB Landed (ADR-0047).
Daily seed sweep 16 → 128 FDB Landed (ADR-0036 amendment).
Swarm testing (per-seed feature subsets) ISSTA 2012 Landed (ADR-0097) — labels + ProducerConsumerWorkload operations in the first cut.
Long-running soak (≥ 1 000 seeds) FDB Out of scope today — current sim runs ~50 ms per seed; 128 daily covers the seed space until a slow regression appears.
VOPR-equivalent dedicated runner TigerBeetle Out of scope — TigerBeetle runs VOPR on dedicated bare-metal because every seed costs hours; magnetar's seeds are sub-second.
Replacing moonpool with a different sim crate — Out of scope — moonpool already supplies the FDB+TB primitives (single-threaded executor, seeded RNG, virtual clock, in-process network).

References