v0.10.0-alpha.37

OpenRaft v0.10.0-alpha.37

This alpha adds openraft-rt-sim, a deterministic test runtime, and fixes three bugs it found. It also relaxes the Send and Sync bounds on custom runtimes and adds opt-in eager replication.

Breaking changes

Custom runtimes need fewer Send and Sync bounds

Without the single-threaded feature, AsyncRuntime, Mpsc and Watch no longer need Send or Sync. JoinHandle, Sleep, Oneshot::Receiver, OneshotSender and MpscReceiver no longer need Sync. block_on and run may return a value that is not Send.

Upgrade: remove T: OptionalSend from block_on in every AsyncRuntime impl. Generic code that relied on these bounds must state them itself. (#2154, #2155)

Wait::timeout is now Option<Duration>

None means no timeout. Raft::wait() keeps its signature. Code that builds a Wait by hand wraps the duration in Some. (#2147, thanks @zhixinwen)

Example crates

• dir-transfer renames DirSender to DirFrameProducer, DirReceiver to DirWriter, feed() to write_frame(), FileMeta to ManifestEntry, FORMAT_VERSION to PROTOCOL_VERSION, and MAX_NAME_LEN to MAX_FILE_NAME_BYTES. The wire format is unchanged.
• sm-rocks snapshots are RocksDB checkpoints. A snapshot is now a directory, and raft-kv-rocksdb shows how to send it. RocksStateMachine::<C>::open(dir) replaces sm_rocks::new, and the log store needs a separate directory. sm_rocks::TypeConfig and RocksNodeId are removed. Data in the old layout is not migrated.

(#2149)

New features

• openraft-rt-sim: SimRuntime runs all tasks on one thread with virtual time, and the same OPENRAFT_RT_SIM_SEED repeats the same run. openraft-memstore gains an rt-sim feature to use it. CI now runs every test in the tests crate on it. (#2099, fixes #206, thanks @pchrysostomou)
• Biased selection: AsyncRuntime::select2(), select3() and select4() pick ready futures in argument order. OpenRaft uses them instead of futures_util::select!, whose random choice a seeded runtime cannot repeat. (#2151)
• MpscReceiver::peekable() returns a PeekableReceiver, whose peek() reads the next message without removing it. (#2156)
• Config::broadcast_submitted_on_append, off by default, starts replication right after RaftLogStorage::append() returns. In the workload of #2152, it cut p99 write latency by 26% and raised p50 by 21%. On benchmarks/minimal, it lowered throughput by 13.6%. (#2152, thanks @zhixinwen)

Fixes

• A ReadIndex read is now confirmed by the heartbeat it triggers, even when both get the same clock instant. Before, the read could fail at its deadline. (#2160)
• A snapshot transfer now backs off from the 11th RPCError::Network error on. Before, it retried in a busy loop. (#2099)
• When a follower accepts none of the entries in a stream session, replication now backs off before it resends. Before, it resent at once in a busy loop. (#2099)

Performance

• Raft::wait(None) and Linearizer::await_ready() no longer arm a timer. This removes tokio timer-lock contention under read-heavy load. (#2147, thanks @zhixinwen)
• With tick disabled, the tick loop sleeps until RuntimeConfigHandle::tick(true). This saves about 100 wakeups per second for each Raft at the default heartbeat_interval. (#2150, thanks @P47Phoenix)
• RaftCore runs engine commands once per batch of notifications instead of once per notification. (#2152)
• make bench_write_latency measures write latency with simulated IO delays. (#2157)

Full changelog: v0.10.0-alpha.36...v0.10.0-alpha.37

v0.10.0-alpha.36

OpenRaft v0.10.0-alpha.36

This alpha lets a leadership transfer to a just-restarted node finish in time, adds Trigger::reset_backoff(), lets heartbeat streams finish cleanly, and cuts RaftCore task memory by about 30%.

New features

Leadership transfer resets the target's replication backoff

After AppendEntries RPCs to a follower fail, the leader waits through a backoff delay before it retries that follower. A node that has just restarted can still be inside this delay when Trigger::transfer_leader() picks it as the target. The transfer request can then time out before the node catches up.

Trigger::transfer_leader() now resets the target's backoff before the transfer starts, so the target catches up right away. The new option Config::reset_backoff_on_transfer_leader controls this: None (the default) and Some(true) enable the reset, and Some(false) keeps the backoff delay.

(#2145, fixes #2143, thanks @zhixinwen for the report)

Trigger::reset_backoff() ends replication backoff on demand

Trigger::reset_backoff() ends the active backoff for the given node ids, or for every replication target when the list is empty. Call it when the application knows that a follower is reachable again.

• A reset applies to the backoff that is active when the reset arrives.
• Snapshot transfer keeps its own retry backoff.
• stats::ExternalCommandName gains a ResetBackoff variant for runtime stats. An exhaustive match on ExternalCommandName needs an arm for it.

(#2145)

Fixes

• The heartbeat worker now drains the response stream returned by RaftNetworkV2::stream_append() after it closes the request stream, so cleanup code inside the response stream runs. With tonic bidirectional gRPC streams, heartbeat calls now end with OK. The drain time counts toward the heartbeat timeout. (#2142, thanks @jkosh44)

Performance

• The core task now keeps RaftCore in its own heap allocation, which makes the task's future smaller. In a release build of benchmarks/minimal, the task and RaftCore together use 5,496 bytes of heap instead of 7,808, and 7,288 bytes instead of 11,392 with the runtime-stats feature. (#2140)
• benchmarks/minimal gains a profile binary. It measures startup heap, allocation count, write throughput, request p99 latency, and forced snapshot build cost. (#2141)

Testing and reliability

• The Jepsen leader-removal scenario now sets up its partition within the leader lease and normalizes membership errors. (#2128, thanks @Phoenix500526)
• A Jepsen unit test now checks the exact iptables partition script. (#2139)
• The Jepsen liveness checkers now share one nemesis event query. (#2129, thanks @Phoenix500526)

Full changelog: v0.10.0-alpha.35...v0.10.0-alpha.36

v0.10.0-alpha.33

OpenRaft v0.10.0-alpha.33

This alpha introduces breaking updates to linearizable reads and snapshot metadata, alongside snapshot-safety fixes and stronger Jepsen coverage.

Breaking changes

Leadership-aware linearizable reads

Linearizable-read APIs now return ReadLogId instead of Option<LogId>:

• Raft::ensure_linearizable() returns ReadLogId.
• Raft::get_read_log_id() returns (ReadLogId, Option<LogId>).
• Linearizer::new() accepts ReadLogId.

ReadLogId is always present and identifies both the producing leadership and the inclusive log boundary the state machine must apply before serving the read.

Use ReadLogId::log_id() to access the underlying log ID, or ReadLogId::from_log_id() when reconstructing a leader-issued barrier for follower reads.

snapshot_id moved out of snapshot metadata

snapshot_id has been removed from SnapshotMeta and SnapshotSignature. It identifies a transfer session, not the state represented by a snapshot.

• State machines now construct SnapshotMeta using only last_log_id and last_membership.
• RaftNetworkV2 implementations require no changes.
• Chunked v1 implementations use openraft_legacy::network_v1::SnapshotMeta, which retains snapshot_id and the 0.9 wire layout.
• The v1 sender now generates a fresh ID for every transfer session.

No stored-data migration is required. The serialized layouts remain compatible with OpenRaft 0.9 under both named and positional formats.

See the v0.9-to-v0.10 upgrade guide.

Correctness fixes

• Reject snapshots that conflict with an entry at an already committed index, during both installation and startup recovery.
• Fix RocksDB example snapshot ordering once log indexes cross digit boundaries.
• Prevent node-only membership changes from panicking on uninitialized nodes; they now return the normal leadership error.

Testing and reliability

• Add composable Jepsen chaos combining partitions, process termination, process pauses, and membership changes.
• Add a process-pause nemesis and configurable snapshot pressure.
• Improve target selection, recovery tracking, panic detection, random-seed recording, and incomplete-run diagnostics.
• Run Jepsen linting and unit tests on pull requests and retain result artifacts for every run.

Other changes

• Replace byte-unit with parse-size, removing the vulnerable lockfile-only rkyv dependency.
• Disable unused tabled default features.
• Move the experimental ezraft implementation out of the OpenRaft repository.

Full changelog: v0.10.0-alpha.32...v0.10.0-alpha.33

v0.9.25 bug-fix

Openraft v0.9.25

A bug-fix-only release. Nine fixes, all backported from release-0.10, covering one
safety defect, one state-divergence defect, and a set of liveness problems that could
stall a leader, a follower, or a caller indefinitely.

No public API changes, no storage format changes. Upgrading is a version bump.

Upgrading from 0.9.x is strongly recommended. The commit-safety fix (#1802) and the
membership-divergence fix (#1808) can both corrupt cluster state in ways that do not heal
on their own.

Safety

Commit could be granted on a quorum that never existed (#1802, 7d8bf714)

VecProgress keeps every voter entry above granted in descending order, and the quorum
scan relies on that ordering: for each candidate value it tests whether the prefix
vector[0..=i] forms a quorum. The reordering step ran only when the updated voter had
previously been at or below granted; a voter already above it that advanced further kept
its old slot, leaving the region unsorted. The scan then counted an arbitrary prefix as
"the voters that reached this value" and could commit a log entry that fewer than a quorum
had actually accepted.

The regression test replays the five-voter sequence from the issue: before the fix it
granted index 6 with only two of five voters having reached it.

Snapshot install could leave a node on a membership config that no longer exists
(#1808, 7b66adf2)

A follower that had accepted a membership entry from an old leader, then installed a
snapshot whose last membership sits at a lower log index, kept its local effective
membership — while the same install set purge_upto to the snapshot's last log id and
deleted the entry backing that membership. The node was left running a config that neither
its log nor its snapshot could restore, permanently diverged from the rest of the cluster.

MembershipState::update_committed() now also considers the snapshot's last log index and
resets both committed and effective membership when the effective one falls inside the
purged range.

Liveness

Leader froze while rebuilding replication streams (#1810, 2447bae4)

remove_all_replication() awaited every removed replication task inline, inside the
RaftCore loop. A task parked in an AppendEntries call to an unresponsive follower only
returns after the RPC times out, so each rebuild stalled RaftCore for up to one RPC timeout
per removed target — and rebuilds happen on every membership change and every leadership
establishment. While parked, RaftCore serves neither writes, nor reads, nor metrics.

Removed tasks are now joined in the background. Progress from a detached task is discarded
by the existing session_id check, so it cannot leak into the new stream.

Callers hung forever when the state machine worker died (211b91ba)

client_write(), get_snapshot() and begin_receiving_snapshot() resolved a closed
response channel by awaiting the RaftCore task handle. That is only correct when the core
actually stopped. The state machine worker owns the responders for the commands it serves,
so if that task dies on its own the responders drop while RaftCore keeps running — and the
caller awaits a handle that never resolves.

Both response paths now wait a bounded one second for the core to stop before falling back
to Fatal::Stopped. A genuine shutdown still reports the error that caused it; a dropped
responder returns promptly. The wait observes the metrics watch channel rather than joining
the task, so it is non-destructive and safe for any number of concurrent callers.

Follower behind a fully purged log never received a snapshot (#1828, 81e304b5)

When the leader had purged its entire log (purge_upto == last_log_id) and a follower's
progress was reset, next_send() computed an empty range and returned Inflight::None on
every call. Nothing advanced, so the follower never probed, never got a snapshot, and could
not converge until a new entry happened to be proposed.

The empty range now sends a snapshot, guarded by matching.next_index() < searching_end so
that fully caught-up followers — which also reach start == end — are not shipped a
snapshot on every idle call.

Replication task panicked on an empty limited_get_log_entries() (#1601, 546ed868)

The method is documented never to return empty for a non-empty range, and replication
relied on that by unwrapping logs.first(), guarded only by a debug_assert! — which is
compiled out in release builds. A store violating the contract panicked the replication
task in exactly the builds that matter.

An empty read is now handled as a heartbeat, with a 10 ms sleep before retrying (the range
does not advance, so an immediate retry would spin) and a warning naming the offending
range. RaftCore rebuilds the stream after a task dies, so the practical 0.9 impact was a
spurious panic and a replication stall rather than an outage.

trigger().elect() on the current leader cost the group its leader (d94e232e)

Campaigning does not tear down the internal leader: state.vote moves to a new uncommitted
term while the node keeps heartbeating under the old one. Those heartbeats keep refreshing
the voters' leader leases, and an unexpired lease is exactly what makes them reject the vote
request the campaign just sent. The campaign could not win, and repeating the trigger only
repeated the loop — losing the leader and inflating the term for nothing.

The trigger is now a no-op when the node is already the leader.

Metrics

current_leader() returned None for a non-voter leader (#1693, 89629e33)

Openraft does not require a leader to be a voter — leadership follows from a committed vote.
calc_server_state() reports ServerState::Leader for a non-voter leader, but
current_leader() filtered its result through effective().is_voter(), so the same node
reported itself as Leader while current_leader was None. This is reachable whenever a
membership change removes the leader from the voter set; the node keeps operating and
committing until it steps down or a new leader is elected.

The check is removed, which also drops an is_voter() scan from a hot metrics path.

Tooling

• Publish script no longer aborts on already-published crates (c9520f1d) — crates.io
  rejects API requests without a descriptive User-Agent (403), so every existence check read
  as "not published" and re-running after a partial publish always aborted. The check now
  sends a User-Agent, treats "already exists on crates.io index" as success, polls for
  indexing instead of sleeping a flat 30 s, and logs the resolved HTTP status.

Behavior changes

These are behavior changes rather than API changes; no signature in the public surface moved.

• RaftMetrics::current_leader now reports a leader that is not a voter, where it previously
  reported None. Code that used current_leader.is_some() as an implicit "the leader is a
  voter" test needs an explicit check.
• Raft::trigger().elect() is a no-op on a node that is already the leader, instead of
  starting a campaign.

Notes

All nine fixes are backports from release-0.10. Three are hand-ports rather than
cherry-picks, because the surrounding code was restructured upstream:

• 546ed868 — 0.10 fixes this in stream_state.rs, which does not exist on 0.9.
• 81e304b5 — 0.10 splits next_send into probing and pipeline regimes; 0.9 has no
  pipeline mode, so the probing condition is added to the existing shape.
• d94e232e — the Pre-Vote half of the upstream change is dropped; 0.9 has no Pre-Vote and
  therefore no non-disruptive election trigger to fall back on.

v0.10.0-alpha.25

What's Changed

• fix: keep VecProgress sorted above quorum by @drmingdrmer in #1804
• fix: reject stale transfer-leader target election by @drmingdrmer in #1806
• chore: release: bump version to v0.10.0-alpha.25 by @drmingdrmer in #1807

Full Changelog: v0.10.0-alpha.24...v0.10.0-alpha.25

v0.9.23

Released: 2026-04-20

This is a bugfix release on the 0.9 line. The headline improvements all center on snapshot replication reliability — a tight loop in the chunk-send retry path that could burn CPU against a flaky follower, a debug-build panic triggered by a harmless race between the engine's snapshot metadata and the state machine, and an example that stopped working on newer toolchains. Upgrading is safe and recommended if you use snapshotting.

Highlights

Snapshot streaming is now bounded and drives forward on stale snapshots

Before this release, Chunked::send_snapshot silently swallowed every transient RPCError — Timeout, Unreachable, Network, even remote Fatal — and re-entered the loop without advancing the offset. A flaky target could therefore keep re-streaming the same snapshot forever, even after the leader had built a fresher one. The outer hard_ttl timeout expiring also just continued, making a 1 ms per-iteration sleep the only real brake.

v0.9.23 replaces this with an explicit retry policy:

• Timeout / Network — exponential backoff, 10 ms → 200 ms
• Unreachable — caller-supplied RaftNetwork::backoff() iterator
• PayloadTooLarge / remote Fatal — fail fast, propagate upward
• SnapshotMismatch — reset offset and retry state

After 5 consecutive transient failures, the call returns and the replication layer re-drives with the current (possibly newer) snapshot. An integration test (t91_snapshot_retry_uses_latest_snapshot) exercises this end-to-end.

Commit: 29941565 — thanks to cliff0412.

Inflight snapshot ack tolerates fresher snapshots

A debug-build panic in Inflight::ack could trip when the state machine built a newer snapshot between next_send time and the get_snapshot() call — the shipped snapshot's last_log_id was simply ahead of the one the engine had recorded. The ack check is now upto >= last_log_id instead of strict equality, which still catches real regressions (a follower's matching rolling backward) but no longer panics on the benign TOCTOU race.

Commit: cc6a2cfa.

raft-kv-rocksdb example runs on recent toolchains again

The example serves its HTTP API via tide (on async-std), but Raft methods internally await tokio::time::sleep. Under newer nightly toolchains, polling sleep outside a Tokio runtime started panicking with "there is no reactor running". The example now captures tokio::runtime::Handle::current() at startup and dispatches raft calls via tokio_handle.spawn(...), so the cluster test works again.

Commit: cffb1a99 — cherry-picked from upstream 83c8176e.

Other changes

• Toolchain: rust-toolchain updated to nightly-2025-10-01; CI lint/fmt/doc errors under the newer nightly resolved.
• Compatibility: rocksstore-v2 updated for rand 0.9 API (thread_rng() → rng()).
• Tooling: added scripts/publish.sh for release publishing.

Upgrade

[dependencies]
openraft = "0.9.23"

No API changes — drop-in upgrade from 0.9.22.

Full change log

See change-log/v0.9.23.md for per-commit detail.

Thanks to

• @cliff0412 — original author of the bounded snapshot-chunk retry policy and the raft-kv-rocksdb Tokio-runtime example fix.
• @drmingdrmer — maintenance, cherry-picks, and release.

v0.9.22

Summary:

• Fixed:
  • 6ff217da prevent redundant election when single voter is already electing.

Detail:

Fixed:

• Fixed: 6ff217da prevent redundant election when single voter is already electing; by Zhang Yanpo; 2026-04-08

  When a single-voter node calls elect() during initialize(), it
  becomes a Candidate with an uncommitted vote. Before the vote flush
  completes, an election tick can fire and trigger a second elect(),
  bumping the term unnecessarily (e.g., T1 → T2). This caused flaky
  test failures in single-term-leader mode where new_cluster()
  expected vote == <T1-N0:Q> but observed <T2-N0:Q>.

  The fix checks candidate_ref().is_some() to detect an in-progress
  election. This is precise because the Candidate struct only exists
  between an elect() call and becoming Leader — it is never present
  after a node restart, unlike vote-based checks (is_leading()) which
  would incorrectly match persisted votes from a previous run.

v0.9.20

Summary:

• Fixed:
  • 33fde29d reset progress when heartbeat response indicates conflict.

Detail:

Fixed:

• Fixed: 33fde29d reset progress when heartbeat response indicates conflict; by Zhang Yanpo; 2025-06-19

  With loosen-follower-log-revert enabled:

  Before this fix, RaftCore ignored conflict messages from Heartbeat RPCs,
  preventing leaders from discovering follower state changes. When a follower's
  state reverted and responded with a conflict message, the leader wouldn't
  retransmit necessary data to the follower.

  This commit ensures conflict responses are always processed properly and
  progress is reset to trigger data retransmission to the follower.

v0.9.19

Summary:

• Improved:
  • 9bfd517f tolerate leader state reversion upon restart.

Detail:

Improved:

• Improved: 9bfd517f tolerate leader state reversion upon restart; by Zhang Yanpo; 2025-06-10

  When a leader restarted and its log reverted, and tried to re-elect
  itself as leader:

  And when vote request is rejected and see a greater vote,
  it should only update to the non-committed version of the responded vote
  to its local state:

  This prevents a dangerous scenario when state reversion is allowed:

  1. A node was a leader but its state reverted to a previous version;
  2. The node restarts and begins election;
  3. It receives a vote response containing its own previous leader vote;
  4. Without this protection, it would update to that committed vote and
     become leader again;
  5. However, it lacks the necessary logs, causing committed entries to be
     lost or inconsistent;

  By using the non-committed version, we prevent this reverted node from
  becoming leader while still allowing proper vote updates for legitimate
  cases.

v0.9.18

Summary:

• Fixed:
  • 5ae1965b Remove serde feature detection from declare_raft_types! macro.

Detail:

Fixed:

• Fixed: 5ae1965b Remove serde feature detection from declare_raft_types! macro; by 张炎泼; 2024-11-30

  The [cfg_attr(feature = "serde")] feature detection for
  RaftTypeConfig implementation should not be evaluated in application
  crates when the declare_raft_types! macro is called. This detection
  logic should be removed from the macro and instead added to each type
  that uses RaftTypeConfig.