PoC: Raft-based cluster bus

[zuiderkwast]

This PR implements a Raft-based cluster bus protocol as an alternative to the gossip protocol. All cluster metadata operations go through Raft consensus, providing strong consistency for cluster state changes.

Based on the cluster-v2 branch (protocol separation).

The admin commands CLUSTER MEET, etc. are preserved, to make it as easy as possible for users to keep their existing control plane logic when migrating. The existing test framework for cluster tests can also be reused (except where they test gossip-specific cluster bus packet details).

Design

See design-docs/cluster-raft.md for the full design document including sequence diagrams and message formats.

This PoC uses a simple text-based protocol for the cluster bus. The focus is on checking how much of the existing testing infrastructure and test cases can be made working.

Raft log entry types

Entry	Triggered by	Description
NODE_JOIN	CLUSTER MEET	HELLO/HI/WELCOME handshake, step-down rules
NODE_FORGET	CLUSTER FORGET	Remove node from cluster
SLOT_CHANGE	ADDSLOTS/DELSLOTS/etc.	Slot assignment with key deletion on migration
SET_REPLICA_OF	CLUSTER REPLICATE	Replica assignment and promotion
FAILOVER	CLUSTER FAILOVER / auto	Manual (force, takeover) and automatic failover
NODE_INFO	CONFIG SET (address/hostname)	Address, hostname, announced ports propagation
NODE_FAIL	Leader detection	Leader detects via last_ack_time in cron

Wire protocol messages

Message	Direction	Purpose
HELLO, HI, WELCOME	Peer ↔ Peer	Greeting/handshake for CLUSTER MEET
AE, AE_ACK	Leader → Follower	AppendEntries (AE_ACK carries repl_offset)
VOTE_REQ, VOTE	Candidate ↔ Peer	Leader elections
PROPOSE	Follower → Leader	Forward proposals to leader
FAILOVER_PREPARE	Leader → Replica	Coordinated manual failover
REPL_OFFSETS	Leader → All	Replica offsets for failover ranking
PUBLISH	Leader → All	Pub/sub propagation (binary-safe)
MODULE	Leader → All	Module message propagation (binary-safe)

BLOCKED_ASYNC mechanism

Cluster mutation commands block the client asynchronously until the Raft entry is committed. This is implemented via blockClientAsync() / consumeBlockedClientAsyncHandle() in src/blocked.c. Cluster mutation commands are rejected inside MULTI/EXEC for raft.

Test results (--cluster-v2)

Example:

./runtest --cluster-v2 --single tests/unit/cluster/

(...)

Test Summary: 428 passed, 2 failed

!!! WARNING The following tests failed:

*** [err]: Failover on shutdown hands over primaryship to a fully sync'd replica - sigterm - shutdown-timeout: 10 in tests/unit/cluster/auto-failover-on-shutdown.tcl
Failover does not happened
*** [TIMEOUT]: Unable to find a replica to perform an auto failover - sigterm in tests/unit/cluster/auto-failover-on-shutdown.tcl
Cleanup: may take some time... OK

The auto-failover-on-shutdown.tcl tests are flaky and need leader transfer on shutdown to be reliable. That's future work.

All other tests that don't work with Raft yet are marked with the tag cluster-v2:skip with a comment about why each test is skipped.

• Needs raft persistence (restart/reload): announce-client-ip, availability-zone, cluster-shards, cross-version-cluster, diskless-load-swapdb, divergent-cluster-shardid-conf, failover (stress test), multidb, resharding, shardid-propagation
• Gossip-specific (uses gossip packet types, epochs, or gossip-specific mechanics): base (config epoch), cli (ADDSLOTS before MEET), cluster-multiple-meets, cluster-reliable-meet, failure-marking, faster-failover, half-migrated-slot, hostnames, human-announced-nodename, noaddr, packet, slave-selection, slave-stop-cond, slot-ownership (gossip packet filter), update-msg
• Not yet implemented for raft: info (partial), manual-failover, manual-takeover, misc, pubsub (byte stats), replica-in-sync, replica-migration, slot-migration, slot-stats
• Known flaky (needs leader transfer): auto-failover-on-shutdown (not skipped, just flaky — 2 failures)
• Other: cluster-migrateslots (partial: AOF test, shutdown test), cluster-slots (partial: gossip-specific test), failover2 (partial: epoch checks)

TODO

Completed

• Raft log replication (AE/AE_ACK)
• Leader election (VOTE_REQ/VOTE)
• CLUSTER MEET via HELLO/HI/WELCOME handshake
• CLUSTER ADDSLOTS / DELSLOTS / ADDSLOTSRANGE / DELSLOTSRANGE
• CLUSTER REPLICATE (SET_REPLICA_OF)
• CLUSTER FORGET (NODE_FORGET) - one node at a time
• CLUSTER FAILOVER (manual: normal, force, takeover)
• Automatic failover with REPL_OFFSETS and rank-based scheduling
• NODE_FAIL detection by leader
• NODE_INFO for address/hostname/port propagation
• NODE_INFO divergence detection and re-proposal
• Proposal retry on leader change (defer + retry)
• Proposal timeout (3x cluster-node-timeout)
• Pub/sub and sharded pub/sub propagation
• Module message propagation
• BLOCKED_ASYNC mechanism for async client blocking
• MULTI/EXEC rejection for cluster mutation commands
• Test framework: --cluster-v2 flag and cluster-v2:skip tag
• CLUSTER SLOTS / SHARDS / NODES compatibility
• Propagate NOFAILOVER flag from config changes

Remaining

• Persistence: store currentTerm, votedFor, and raft log in nodes.conf
• Restart / rejoin after crash (depends on persistence)
• Leader transfer on shutdown
• valkey-cli --cluster tooling compatibility (uses MULTI internally)
• Clear FAIL flag via log entry when node recovers
• Outbound link establishment timing (affects REPL_OFFSETS delivery)
• total_cluster_links_buffer_limit_exceeded stat for raft links
• slot-stats byte count accuracy (raft message overhead)
• Log compaction / snapshotting
• Pre-vote protocol (prevents term inflation in minority partitions)
• Non-voting members / learners (similar to replicas with NOFAILOVER)
• Non-voting members / learners for read replicas
• More raft-specific tests (elections, proposal retry, failover scenarios)

[murphyjacob4]

As we talked about yesterday - I think doing manual failover orchestration via REPLCONF makes sense. We can continue to align these functionalities to the replication subsystem and then standalone can use them too

[zuiderkwast]

Yes, so the replica sends a new REPLCONF FAILOVER-PAUSE or similar (corresponding to cluster message MFSTART).

It seems strait-forward. Just a compatibility issue: An old version primary doesn't expect that the primary knows the replica's capas and version via REPLCONF CAPA and REPLCONF VERSION, but the replica doesn't know the primary's capas and version. How does the replica know if the primary supports REPLCONF FAILOVER-PAUSE? (An idea: During the handshake, the replica sends a new REPLCONF WHO-ARE-YOU? (TBD) to ask for the primary's capas and/or version. An old version primary will reply -ERR Unrecognized REPLCONF option.)

Practically, how to we move forward with this? I'd like to avoid scope creep in this Raft PR. Either

1. you/we/someone implements this mechanism separately with legacy cluster and/or standalone, or
2. we merge this Raft PR to cluster-v2 first and do it in cluster-v2 afterwards.

The first option seems nicest to me. I don't know what more is required for standalone though. The primary needs to configure itself as a replica and vice versa. We may need another mechanism for that. In standalone, FAILOVER is initiated on the primary, which pauses itself and waits for the replica, then sends PSYNC FAILOVER to the replica, i.e. it's initiated by the primary and the replica doesn't need to tell the primary to pause. In cluster mode, the actual failover it's a consensus thing (failover auth voting in gossip, committed entry in raft).

[murphyjacob4]

Okay - yeah I'm okay pulling the change out of Cluster V2. In the meantime we can continue with what it is in here, and when that goes into unstable we can reconcile it?

I would like to get the cluster-v2 branch bootstrapped with a real implementation sooner rather than later, so we can start doing all the necessary testing and incremental work

[zuiderkwast]

Yeah, we can also do it in the cluster-v2 but in a separate PR. First I would like to get everything to pass. I'm focusing on failing test cases. I have to debug some things that just don't work.

[murphyjacob4]

Perhaps we can just have a general purpose message for all gosisp-style metadata? If it gets big enough, we can get smarter with fingerprinting the metadata and only requesting the content when there is a diff. But for now we can bundle this all into a periodic update

Potential other eventually consistent metadata:

1. Loaded Functions
2. ACLs
3. Node information/availability-zone/IP address/etc (may be fine in the Raft log since it is low volume?)

[zuiderkwast]

Don't we want to push functions and ACLs through the raft log to make it truly consistent? I don't see why the volume would be a problem for these.

I think we should be aware of drift already from the start. If two nodes constantly propose different ACLs and update each other and everybody else, it'd be an endless loop of updates. We should prevent that.

With that in place, I don't see why the volume would matter much. We need to think through how we store these things in a backward/forward compatible way though. Currently we store ACLs in an ACL file and functions in RDB. With raft, we'd store them in the log and cluster state (e.g. nodes.conf or something else; snapshot + append-only file).

[zuiderkwast]

Regarding general-purpose entry type for any kind of metadata or specific ones, we'll have the compatibilty problem regardless:

1. with a SET_GENERAL_PURPOSE_METADATA key="acl" value="foo -@all +hset xxx..." (intended to replace the whole ACL config) an old node can store the key/value pair but doesn't know what to do with it.
2. with a dedicated SET_ACL foo -@all +hset xxxd... an old node doesn't understand the entry at all, so it can't apply it, but it still gets stored in the raft log as it's currently implemented. The leader accepts proposals and adds the to the log without any validation so even if the leader is an old node, this can work the same.

However, in both cases, it's pretty bad if some nodes don't understand all the entries. It can lead to inconsistencies. To prevent that, each node should announce its version and/or capabilities, so that other nodes can propose only things that the other nodes can understand.

The broader topic of being able to upgrade the protocol is very important.

[murphyjacob4]

I'm okay with putting ACL/functions through the Raft log. The general-purpose gossip concept was less for compatibility and more for efficiency, since as we accumulate gossip content, we can just use a fingerprint of the gossip content rather than the full payload. But anyways I am getting ahead of the requirements. If offsets is the only case - then I think we can keep it as a dedicated message right now.

[zuiderkwast]

Oh, fingerprint gossip? Like we can use GOSSIPSHA as the command name to mirror EVALSHA? :)

We should only send changes, not push the same content over and over, rigth? If we do that, then I don't see a need for fingerprinted version.

Anyway, we don't need to commit to backward compatibility just yet. We can change anything later. We will need to keep cluster V2 experimental for some time probably.

[murphyjacob4]

I like the idea of moving fault detection to the replication subsystem since we are always sending data and ping/ponging there. We just need the Raft system to be the tie-breaker.

My vision for fault detection is:

1. Replication stream (and REPLCONF heartbeats) refresh the health timer on both primary/replica
2. When either side sees the other as unavailable, it goes to the Raft group and attempts to mark the other one as unhealthy. The primary uses a shorter timeout than the replica to always favor not failing over
3. An unhealthy node is always unsuitable for primaryship, so we can join the NODE_FAIL message with FAILOVER. I think probably calling this UPDATE_SHARD makes sense.
4. We can add a shard-level epoch as a fencing token, and now you have a clean pre-validation that we can deny dueling proposals based on first-write-wins: UPDATE_SHARD <shard-id> [PREV-EPOCH <epoch>] PRIMARY <node-id> HEALTHY-REPLICAS <count> <node-ids...> UNHEALTHY-REPLICAS <count> <node-ids...>

This would also allow us to join with SET_REPLICA_OF

What do you think? PREV-EPOCH is optional, so for operations with no fencing requirements (like adding a new node ID as a replica), we can exclude it. Although we can start with specifying it each time.

Kafka does this with: PartitionRecord(topic, partition, leader, isr, leader_epoch, partition_epoch). isr is the equivalent of "heatlhy replicas" for us. leader_epoch bumps on failover, partition_epoch bumps on new replicas or replicas going out of sync. I don't think we need more than one epoch, but we can look into the KRaft design to understand why they needed two

[zuiderkwast]

My vision for fault detection is:

1. Replication stream (and REPLCONF heartbeats) refresh the health timer on both primary/replica

Yes, this is a good idea. I'm aligned on the high level.

Here's what we already have regarding replication heartbeats:

• The replica sends REPLCONF ACK every second (once per replicationCron cycle). Hard-coded to 1 second AFAICT.
• The primary sends PING on the replication stream every 10 seconds if there's no data to replicate. Configurable with repl-ping-replica-period.
• The primary sends an newline \n to replicas waiting for the full sync (every second if I read correctly).
• If the replica didn't get any data nor PING from the primary during 60 seconds, it disconnects the replication connection. Later, it tries to reconnect and PSYNC. Configurable with repl-timeout.

Should we override these with numbers relative to the configured cluster-node-timeout?

2. When either side sees the other as unavailable, it goes to the Raft group and attempts to mark the other one as unhealthy. The primary uses a shorter timeout than the replica to always favor not failing over
3. An unhealthy node is always unsuitable for primaryship, so we can join the NODE_FAIL message with FAILOVER. I think probably calling this UPDATE_SHARD makes sense.

In the this Raft PoC, any node can propose anything, and the leader-detected FAIL can co-exist with primary and replica detecting each other as failing.

If we move the shard failure detection and failover decision entirely to the shard, how do the replicas pick the best ranked replica? Should the replicas exchange replication offset by asking each other? In legacy, this happens after marking the primary as FAIL.

I separated FAIL-flagging from FAILOVER just to leave the decision up to each replica if it wants to initiate a FAILOVER for itself. It seemed like a conservative approach. A node may not want to do a failover for some reason, though we have the NOFAILOVER node flag in the global raft state. It's bad if the best ranked replica doesn't want to initate failover for some unknown reason though, because it will just delay the failover. Maybe the leader should just pick the best replica (if it has their replication offsets, which is has in this PoC) and force it to take over by just adding FAILOVER to the log.

We should think through the different network failure modes. If the leader has contact with primary and replica, but they don't have contact with each other, failover shoudn't happen. If the primary marks the replica as FAIL first, as you say, this may be enough.

        Raft leader  (Nobody failed IMO)
           / \
          /   \
         /     \
        /       \
 Primary -- X -- Replica

(My replica         (My primary
 failed?)            failed?)

You have an idea that some nodes are not members of the raft group at all, not even as learners? Every node needs to get the topology updates so they can respond accurately to CLUSTETR SLOTS and such client commands. Then, don't we need them all to be at least as learners? Then, they'll get the raft log and heartbeats, but they don't count in the quorum for committing entries and they can't vote. When they ack heartbeats, we get leader-based failure detection for free.

4. We can add a shard-level epoch as a fencing token, and now you have a clean pre-validation that we can deny dueling proposals based on first-write-wins: UPDATE_SHARD [PREV-EPOCH ] PRIMARY HEALTHY-REPLICAS <node-ids...> UNHEALTHY-REPLICAS <node-ids...>

Shard-level epoch can be useful for preventing failover-ping-pong when multiple FAILOVERs are waiting in the Raft log. The first failover will succceed and the next one will have a mismatching shard-epoch and be ignored. Currently in this PoC, the only mechanism used is that the old primary still needs to be primary for the failover to happen. I see now that the design-doc isn't updated but the entry syntax is

FAILOVER <replica-id> <primary-id>

I don't know if we can get failover ping-pong behavior with this syntax. Maybe we can. A shard-level epoch sounds like a good idea.

This would also allow us to join with SET_REPLICA_OF

I think it's clearer if we keep them different. They have different purposes and semantics:

• SET_REPLICA_OF is for moving nodes between shards. It corresponds to CUSTER REPLICAOF <node-id> and CLUSTER REPLICAOF NO ONE. It changes the node's shard-id and doesn't affect other nodes.
• FAILOVER affects two nodes in a shard. One replica and one primary swap roles atomically.

What do you think? PREV-EPOCH is optional, so for operations with no fencing requirements (like adding a new node ID as a replica), we can exclude it. Although we can start with specifying it each time.

Yes, shard-epoch is a good idea. Probably for FAILOVER and SET_REPLICA_OF. Maybe also SLOT_CHANGE (slot migrations and addslots/delslots). It affects two shards, so should we include both shards' epochs and require both to match?

Kafka does this with: PartitionRecord(topic, partition, leader, isr, leader_epoch, partition_epoch). isr is the equivalent of "heatlhy replicas" for us. leader_epoch bumps on failover, partition_epoch bumps on new replicas or replicas going out of sync. I don't think we need more than one epoch, but we can look into the KRaft design to understand why they needed two

Is one of the epochs the global Raft leader's term? I don't know, but one shard-epoch to avoiding conflicts for concurrent changes sounds like a good idea

[zuiderkwast]

Update: I don't know if failure detection can co-exist. If a replica marks its primary as FAIL and the leader marks it as not failing (based on AppendEntriesAck), they will compete...

[murphyjacob4]

Should we override these with numbers relative to the configured cluster-node-timeout?

In the final implementation - yes. Probably MIN(1 second, node-timeout/2)? But we need to build out the rest of the health check system first. So for this PR, we can leave it at 1 second.

[murphyjacob4]

You have an idea that some nodes are not members of the raft group at all, not even as learners?

Oh yeah, they would still need to replicate. But replicating from the Raft group is not the same as being a member. It could even be chaining-style replication.

Anyways, for now we should keep them all in the same group. Adding a "learner/observer" role later should be possible.

When they ack heartbeats, we get leader-based failure detection for free.

This forces leader to be fully connected to all learners, which isn't a strict requirement. We could use chaining replication, and they also don't need to necessarily ACK the replication stream.

When I was doing the POC, I had them get a stream of AppendEntries, and if they get an AppendEntries that is too far ahead, they just request snapshot. But there is no match_index tracking

[murphyjacob4]

I don't know if we can get failover ping-pong behavior with this syntax. Maybe we can. A shard-level epoch sounds like a good idea.

Yeah the main issue here is:

1. serializing slot migrations with failovers AND
2. stale failover proposals

For more on 2, without epochs, you could have:

1. <Node A primary, Node B replica>
2. Node B -> leader: FAILOVER node-a node-b
3. Leader: Process + commit first FAILOVER node-a node-b
4. <Node B is partitioned, doesn't hear about failover>
5. <Node A comes back up, wants to be primary again>
6. Node A -> leader: FAILOVER node-b node-a
7. Leader: Process + commit FAILOVER node-b node-a
8. <Node B times out since it is partitioned, retries the failover>
9. Node B -> leader: FAILOVER node-a node-b
10. <Node B, partition resolves>
11. <Now Leader gets request for second FAILOVER node-a node-b>
12. Leader: Process + commit second FAILOVER node-a node-b
13. <Node B, back to primary again erroneously>

It's a bit of an edge case, but good to protect against

[murphyjacob4]

Update: I don't know if failure detection can co-exist. If a replica marks its primary as FAIL and the leader marks it as not failing (based on AppendEntriesAck), they will compete...

Yeah I think we should converge on one source of truth. Anyways I think we should add failure detection (as I described) incrementally. We need to start somewhere

[zuiderkwast]

We need to start somewhere

So we start with leader-based failure detection as in this PR and replace it later? It already seems to work so I'd like to get the remaining small missing pieces working (CLUSTER RESET is one of them). We should open issues or something for these follow-ups.

[murphyjacob4]

1. See later comment, but I think a per-shard fencing token would help. Then SLOT_CHANGE needs to precondition on the fencing token being unchanged. So adding a PREV-SOURCE-EPOCH parameter makes sense
2. Not sure, but we may want to also add PREV-TARGET-EPOCH since a failover on the target during migration may be data loss in async replication
3. We should bump the per-shard epoch on both the target and source when we do the final migration

With those changes, we will effectively serialize all failovers and migrations to/from a shard. So it will simplify our reasoning and safety

So I think:

SLOT_CHANGE [PREV-SOURCE-EPOCH <epoch> SOURCE <shard-id>] PREV-TARGET-EPOCH <epoch> TARGET <shard-id> RANGE <range count> <ranges...>

No SOURCE would indicate a slot claim, which would be rejected if the slot is claimed (or in the process of being claimed) by anyone else at that time. We can keep the dash syntax for un-claiming a slot, but it should still fence on source epoch.

WDYT?

[zuiderkwast]

Yeah, source epoch sounds good. I replied to the comment above before reading theis one. 😄

Shall I add shard-epoch now or shall we get this merged and do it afterwards?

[murphyjacob4]

Let's do incremental. It is already massive. But good to align!

[zuiderkwast]

OK. We can move it to an issue. Agent's suggestion about this: SLOT_CHANGE should check epoch but not bump it. SET_REPLICA_OF should bump both epochs (source and target shard). Otherwise, we can lose a whole shard if it races with ASM.

Shard-epoch

• SET_REPLICA_OF: A replica moves from shard X to shard Y (or gets promoted to
  its own shard). Both shards change membership. So yes, it should bump both the
  source and target shard epochs. A stale SET_REPLICA_OF that references an old
  epoch of either shard should be rejected.

• SLOT_CHANGE: Slots move from shard A to shard B. This changes both shards. It
  should check the epochs of both source and target shards. But should it also
  bump them?

If SLOT_CHANGE bumps the epoch, then two concurrent SLOT_CHANGE entries for
different slots in the same shard would conflict — the second one would see a
bumped epoch and become a no-op, even though it's for a different slot. That's
too aggressive.

So SLOT_CHANGE should check but not bump. Only FAILOVER and SET_REPLICA_OF bump,
because they change the shard's internal structure (who is primary, who is
replica). SLOT_CHANGE just moves slots between shards — it doesn't change the
shard's internal topology.

This means: FAILOVER in shard A invalidates pending SLOT_CHANGE entries
involving shard A (correct — the migration should be rolled back). But two
SLOT_CHANGE entries for different slots in the same shard can both apply
(correct — they're independent).

[murphyjacob4]

As I mentioned offline - I think NODE_ADD may come with addition metadata changes. Not sure how we would want to handle that? Maybe we can just add the option to claim slot ranges as part of NODE_ADD. Or we can have it reconcile post-join, with no guarantees of success?

I'm biasing towards the latter, since anyways doing ADDSLOTS -> MEET may result in collisions today, in which case we pick an arbitrary winner. Doing the same in Raft is okay to pick an arbitrary winner IMO

[zuiderkwast]

I don't know. I included the full aux metadata here as you can see. The <address> parameter is the indentification in nodes.conf format: ip:port@cport,[hostname][,aux=value]*, so that metadata does get updated.

I'm biasing towards the latter, since anyways doing ADDSLOTS -> MEET may result in collisions today, in which case we pick an arbitrary winner. Doing the same in Raft is okay to pick an arbitrary winner IMO

Yeah, as long as it's consistent.

Are you thinking about how to join two non-empty clusters?

[murphyjacob4]

Yeah I'm thinking for joining two non-empty clusters. I guess we would just cache the old cluster content, and try to apply it CRDT-style on the newly joined cluster?

This will be one of the bigger pieces of work we have to do incrementally. Either way, it seems fine for now

[zuiderkwast]

Yeah, I think it's possible to make joining two clusters work. CRDT-style sounds good, maybe using joint consensus or otherwise moving one node at a time using multiple log entries. Then, we can also make MEET and ADDSLOTS non-blocking again to behave more like in legacy.

[murphyjacob4]

We can callout the other projects (Kafka, Cockroach, TiKV). The language I saw was "domain specific operations". Etcd is the only one that does the KV style, and it makes sense since that is a more general-purpose design.

[zuiderkwast]

Please suggest some text. :)

[murphyjacob4]

I think Domain Specific Operations can still do fencing, we don't need to commit twice. It will be much simpler if we have leader pre-validate and, once replicated, always apply.

[zuiderkwast]

Much simpler?

I don't know. Leader validation adds complexity and it needs to send an reply to the propose which the proposer needs to handle. In which way does it get simpler?

Doing it at apply-time (when it's committed) using shard-epoch seems simpler. An entry with a mismatching shard-epoch will be ignored as a no-op.

[zuiderkwast]

The tradeoff:

• No validation (current): simple leader, conflicts resolved at apply time. Works well when conflicts are rare.
• Leader validation: rejects stale proposals early. More complex leader, but cleaner log. Useful if conflicts become common.
• Apply-time guard (shard-epoch): entries enter the log but are no-ops on apply. Simple, but wastes log space.

[murphyjacob4]

I think in the completeness of time they will both be useful:

1. Byzantine/buggy leaders can fail pre-validation - good to still have apply time checks
2. Pre-validation helps keep the log clean and save work
3. Pre-validation helps fast fail if we know the proposal will fail anyways, which is useful signal to the proposer

But - for starters - we only need apply time validation. So I'm okay with it

[zuiderkwast]

Yeah, after chatting yesterday, I think it will be enough that the leader sends a REJECT message back to the proposer if it rejects it. If it doesn't reject it, the ACK is when it appears in the log. We can add it later. Old nodes that don't support it will just time out some proposals (commands like manual failover will hang until timing out, 3 * node_timeout in current code IIRC).