Prevent exposure of importing keys on replicas during atomic slot migration #2635

murphyjacob4 · 2025-09-21T21:38:41Z

Problem

In the current slot migration design, replicas are completely unaware of the slot migration. Because of this, they do not know to hide importing keys, which results in exposure of these keys to commands like KEYS, SCAN, RANDOMKEY, and DBSIZE.

Design

The main part of the design is that we will now listen for and process the SYNCSLOTS ESTABLISH command on the replica. When a SYNCSLOTS ESTABLISH command is received from the primary, we begin tracking a new slot import in a special SLOT_IMPORT_OCCURRING_ON_PRIMARY state. Replicas use this state to track the import, and await for a future SYNCSLOTS FINISH message that tells them the import is successful/failed.

Success Case

     Source                                          Target                         Target Replica
       |                                                |                                 |
       |------------ SYNCSLOTS ESTABLISH -------------->|                                 |
       |                                                |----- SYNCSLOTS ESTABLISH ------>|
       |<-------------------- +OK ----------------------|                                 |
       |                                                |                                 |
       |~~~~~~~~~~~~~~ snapshot as AOF ~~~~~~~~~~~~~~~~>|                                 |
       |                                                |~~~~~~ forward snapshot ~~~~~~~~>|
       |----------- SYNCSLOTS SNAPSHOT-EOF ------------>|                                 |
       |                                                |                                 |
       |<----------- SYNCSLOTS REQUEST-PAUSE -----------|                                 |
       |                                                |                                 |
       |~~~~~~~~~~~~ incremental changes ~~~~~~~~~~~~~~>|                                 |
       |                                                |~~~~~~ forward changes ~~~~~~~~~>|
       |--------------- SYNCSLOTS PAUSED -------------->|                                 |
       |                                                |                                 |
       |<---------- SYNCSLOTS REQUEST-FAILOVER ---------|                                 |
       |                                                |                                 |
       |---------- SYNCSLOTS FAILOVER-GRANTED --------->|                                 |
       |                                                |                                 |
       |                                            (performs takeover &                  |
       |                                             propagates topology)                 |
       |                                                |                                 |
       |                                                |------- SYNCSLOTS FINISH ------->|
 (finds out about topology                              |                                 |
  change & marks migration done)                        |                                 |
       |                                                |                                 |

Failure Case

     Source                                          Target                         Target Replica
       |                                                |                                 |
       |------------ SYNCSLOTS ESTABLISH -------------->|                                 |
       |                                                |----- SYNCSLOTS ESTABLISH ------>|
       |<-------------------- +OK ----------------------|                                 |
     ...                                              ...                               ...
       |                                                |                                 |
       |                                             <FAILURE>                            |
       |                                                |                                 |
       |                                      (performs cleanup)                          |
       |                                                | ~~~~~~ UNLINK <key> ... ~~~~~~~>|
       |                                                |                                 |
       |                                                | ------ SYNCSLOTS FINISH ------->|
       |                                                |                                 |

Full Sync, Partial Sync, and RDB

In order to ensure replicas that resync during the import are still aware of the import, the slot import is serialized to a new cluster-slot-imports aux field. The encoding includes the job name, the source node name, and the slot ranges being imported. Upon loading an RDB with the cluster-slot-imports aux field, replicas will add a new migration in the SLOT_IMPORT_OCCURRING_ON_PRIMARY state.

It's important to note that a previously saved RDB file can be used as the basis for partial sync with a primary. Because of this, whenever we load an RDB file with the cluster-slot-imports aux field, even from disk, we will still add a new migration to track the import. If after loading the RDB, the Valkey node is a primary, it will cancel the slot migration. Having this tracking state loaded on primaries will ensure that replicas partial syncing to a restarted primary still get their SYNCSLOTS FINISH message in the replication stream.

AOF

Since AOF cannot be used as the basis for a partial sync, we don't necessarily need to persist the SYNCSLOTS ESTABLISH and FINISH commands to the AOF.

However, considering there is work to change this (#59 #1901) this design doesn't make any assumptions about this.

We will propagate the ESTABLISH and FINISH commands to the AOF, and ensure that they can be properly replayed on AOF load to get to the right state. Similar to RDB, if there are any pending "ESTABLISH" commands that don't have a "FINISH" afterwards upon becoming primary, we will make sure to fail those in verifyClusterConfigWithData.

Additionally, there was a bug in the existing slot migration where slot import clients were not having their commands persisted to AOF. This has been fixed by ensuring we still propagate to AOF even for slot import clients.

Promotion & Demotion

Since the primary is solely responsible for cleaning up unowned slots, primaries that are demoted will not clean up previously active slot imports. The promoted replica will be responsible for both cleaning up the slot (verifyClusterConifgWithData) and sending a SYNCSLOTS FINISH.

Other Options Considered

I also considered tracking "dirty" slots rather than using the slot import state machine. In this setup, primaries and replicas would simply mark each slot's hashtable in the kvstore as dirty when something is written to it and we do not currently own that slot.

This approach is simpler, but has a problem in that modules loaded on the replica would still not get slot migration start/end notifications. If the modules on the replica do not get such notifications, they will not be able to properly contain these dirty keys during slot migration events.

codecov · 2025-09-21T21:57:58Z

Codecov Report

❌ Patch coverage is 88.69863% with 33 lines in your changes missing coverage. Please review.
✅ Project coverage is 72.26%. Comparing base (0646749) to head (d0a1e76).
⚠️ Report is 1 commits behind head on unstable.

Files with missing lines	Patch %	Lines
src/cluster_migrateslots.c	91.08%	23 Missing ⚠️
src/valkey-check-rdb.c	0.00%	10 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##           unstable    #2635      +/-   ##
============================================
+ Coverage     72.18%   72.26%   +0.07%     
============================================
  Files           128      128              
  Lines         71037    71237     +200     
============================================
+ Hits          51277    51477     +200     
  Misses        19760    19760

Files with missing lines	Coverage Δ
src/cluster_legacy.c	`87.36% <100.00%> (-0.03%)`	⬇️
src/commands.def	`100.00% <ø> (ø)`
src/db.c	`93.24% <100.00%> (-0.01%)`	⬇️
src/rdb.c	`77.60% <100.00%> (+1.16%)`	⬆️
src/rdb.h	`100.00% <ø> (ø)`
src/replication.c	`86.14% <100.00%> (+0.08%)`	⬆️
src/server.c	`88.43% <100.00%> (+0.06%)`	⬆️
src/server.h	`100.00% <ø> (ø)`
src/valkey-check-rdb.c	`71.18% <0.00%> (-1.37%)`	⬇️
src/cluster_migrateslots.c	`91.98% <91.08%> (+0.69%)`	⬆️

... and 7 files with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

zuiderkwast

Great work!

src/cluster_migrateslots.c

zuiderkwast

LGTM, but I'm not fully familiar with the ASM code in general. I didn't review the main ASM PR carefully enough.

zuiderkwast · 2025-09-29T08:25:22Z

tests/unit/cluster/cluster-migrateslots.tcl

+            if {$owning_prim == $idx_to_restart && ! $do_save} {
+                assert_match "0" [R $owning_prim CLUSTER COUNTKEYSINSLOT 16379]
+                assert_match "0" [R $owning_prim CLUSTER COUNTKEYSINSLOT 16381]
+                assert_match "0" [R $owning_prim CLUSTER COUNTKEYSINSLOT 16383]


Why do you prefer assert_match instead of assert_equal? Match is using a glob-style pattern. No need to use it if you don't use wildcards in the pattern.

There was no reason, I can replace them now

Actually - updating the whole file makes the line count go crazy. Can I fix it in a follow-up so that we can keep this change reviewable? It is a simple find/replace, but I don't want this change to have +900 LOC

zuiderkwast · 2025-09-29T08:32:18Z

@valkey-io/valkey-committers Another pair of eyes on this wouldn't hurt. If anyone has time. It's needed for 9.0.0 to be released ASAP.

madolson

I'm mostly aligned with the high level details. I'd like us to trim down the data in the RDB. The implementation seems pretty clean otherwise.

src/cluster_migrateslots.c

tests/unit/cluster/cluster-migrateslots.tcl

src/cluster_migrateslots.c

murphyjacob4 · 2025-10-02T16:20:22Z

Hey @madolson, thanks for the review, please take another look!

madolson

I think everything else looks fine outside some of the questions around the RDB format.

Signed-off-by: Jacob Murphy <[email protected]>

…rom aux Signed-off-by: Jacob Murphy <[email protected]>

murphyjacob4 · 2025-10-06T19:27:56Z

@madolson we now don't track source node ID on the replica, and we use an RDB opcode to save the slot imports. PTAL!

madolson

LGTM, thanks!

Signed-off-by: Jacob Murphy <[email protected]>

…ration (valkey-io#2635) # Problem In the current slot migration design, replicas are completely unaware of the slot migration. Because of this, they do not know to hide importing keys, which results in exposure of these keys to commands like KEYS, SCAN, RANDOMKEY, and DBSIZE. # Design The main part of the design is that we will now listen for and process the `SYNCSLOTS ESTABLISH` command on the replica. When a `SYNCSLOTS ESTABLISH` command is received from the primary, we begin tracking a new slot import in a special `SLOT_IMPORT_OCCURRING_ON_PRIMARY` state. Replicas use this state to track the import, and await for a future `SYNCSLOTS FINISH` message that tells them the import is successful/failed. ## Success Case ``` Source Target Target Replica | | | |------------ SYNCSLOTS ESTABLISH -------------->| | | |----- SYNCSLOTS ESTABLISH ------>| |<-------------------- +OK ----------------------| | | | | |~~~~~~~~~~~~~~ snapshot as AOF ~~~~~~~~~~~~~~~~>| | | |~~~~~~ forward snapshot ~~~~~~~~>| |----------- SYNCSLOTS SNAPSHOT-EOF ------------>| | | | | |<----------- SYNCSLOTS REQUEST-PAUSE -----------| | | | | |~~~~~~~~~~~~ incremental changes ~~~~~~~~~~~~~~>| | | |~~~~~~ forward changes ~~~~~~~~~>| |--------------- SYNCSLOTS PAUSED -------------->| | | | | |<---------- SYNCSLOTS REQUEST-FAILOVER ---------| | | | | |---------- SYNCSLOTS FAILOVER-GRANTED --------->| | | | | | (performs takeover & | | propagates topology) | | | | | |------- SYNCSLOTS FINISH ------->| (finds out about topology | | change & marks migration done) | | | | | ``` ## Failure Case ``` Source Target Target Replica | | | |------------ SYNCSLOTS ESTABLISH -------------->| | | |----- SYNCSLOTS ESTABLISH ------>| |<-------------------- +OK ----------------------| | ... ... ... | | | | <FAILURE> | | | | | (performs cleanup) | | | ~~~~~~ UNLINK <key> ... ~~~~~~~>| | | | | | ------ SYNCSLOTS FINISH ------->| | | | ``` ## Full Sync, Partial Sync, and RDB In order to ensure replicas that resync during the import are still aware of the import, the slot import is serialized to a new `cluster-slot-imports` aux field. The encoding includes the job name, the source node name, and the slot ranges being imported. Upon loading an RDB with the `cluster-slot-imports` aux field, replicas will add a new migration in the `SLOT_IMPORT_OCCURRING_ON_PRIMARY` state. It's important to note that a previously saved RDB file can be used as the basis for partial sync with a primary. Because of this, whenever we load an RDB file with the `cluster-slot-imports` aux field, even from disk, we will still add a new migration to track the import. If after loading the RDB, the Valkey node is a primary, it will cancel the slot migration. Having this tracking state loaded on primaries will ensure that replicas partial syncing to a restarted primary still get their `SYNCSLOTS FINISH` message in the replication stream. ## AOF Since AOF cannot be used as the basis for a partial sync, we don't necessarily need to persist the `SYNCSLOTS ESTABLISH` and `FINISH` commands to the AOF. However, considering there is work to change this (valkey-io#59 valkey-io#1901) this design doesn't make any assumptions about this. We will propagate the `ESTABLISH` and `FINISH` commands to the AOF, and ensure that they can be properly replayed on AOF load to get to the right state. Similar to RDB, if there are any pending "ESTABLISH" commands that don't have a "FINISH" afterwards upon becoming primary, we will make sure to fail those in `verifyClusterConfigWithData`. Additionally, there was a bug in the existing slot migration where slot import clients were not having their commands persisted to AOF. This has been fixed by ensuring we still propagate to AOF even for slot import clients. ## Promotion & Demotion Since the primary is solely responsible for cleaning up unowned slots, primaries that are demoted will not clean up previously active slot imports. The promoted replica will be responsible for both cleaning up the slot (`verifyClusterConifgWithData`) and sending a `SYNCSLOTS FINISH`. # Other Options Considered I also considered tracking "dirty" slots rather than using the slot import state machine. In this setup, primaries and replicas would simply mark each slot's hashtable in the kvstore as dirty when something is written to it and we do not currently own that slot. This approach is simpler, but has a problem in that modules loaded on the replica would still not get slot migration start/end notifications. If the modules on the replica do not get such notifications, they will not be able to properly contain these dirty keys during slot migration events. --------- Signed-off-by: Jacob Murphy <[email protected]>

…ration (#2635) # Problem In the current slot migration design, replicas are completely unaware of the slot migration. Because of this, they do not know to hide importing keys, which results in exposure of these keys to commands like KEYS, SCAN, RANDOMKEY, and DBSIZE. # Design The main part of the design is that we will now listen for and process the `SYNCSLOTS ESTABLISH` command on the replica. When a `SYNCSLOTS ESTABLISH` command is received from the primary, we begin tracking a new slot import in a special `SLOT_IMPORT_OCCURRING_ON_PRIMARY` state. Replicas use this state to track the import, and await for a future `SYNCSLOTS FINISH` message that tells them the import is successful/failed. ## Success Case ``` Source Target Target Replica | | | |------------ SYNCSLOTS ESTABLISH -------------->| | | |----- SYNCSLOTS ESTABLISH ------>| |<-------------------- +OK ----------------------| | | | | |~~~~~~~~~~~~~~ snapshot as AOF ~~~~~~~~~~~~~~~~>| | | |~~~~~~ forward snapshot ~~~~~~~~>| |----------- SYNCSLOTS SNAPSHOT-EOF ------------>| | | | | |<----------- SYNCSLOTS REQUEST-PAUSE -----------| | | | | |~~~~~~~~~~~~ incremental changes ~~~~~~~~~~~~~~>| | | |~~~~~~ forward changes ~~~~~~~~~>| |--------------- SYNCSLOTS PAUSED -------------->| | | | | |<---------- SYNCSLOTS REQUEST-FAILOVER ---------| | | | | |---------- SYNCSLOTS FAILOVER-GRANTED --------->| | | | | | (performs takeover & | | propagates topology) | | | | | |------- SYNCSLOTS FINISH ------->| (finds out about topology | | change & marks migration done) | | | | | ``` ## Failure Case ``` Source Target Target Replica | | | |------------ SYNCSLOTS ESTABLISH -------------->| | | |----- SYNCSLOTS ESTABLISH ------>| |<-------------------- +OK ----------------------| | ... ... ... | | | | <FAILURE> | | | | | (performs cleanup) | | | ~~~~~~ UNLINK <key> ... ~~~~~~~>| | | | | | ------ SYNCSLOTS FINISH ------->| | | | ``` ## Full Sync, Partial Sync, and RDB In order to ensure replicas that resync during the import are still aware of the import, the slot import is serialized to a new `cluster-slot-imports` aux field. The encoding includes the job name, the source node name, and the slot ranges being imported. Upon loading an RDB with the `cluster-slot-imports` aux field, replicas will add a new migration in the `SLOT_IMPORT_OCCURRING_ON_PRIMARY` state. It's important to note that a previously saved RDB file can be used as the basis for partial sync with a primary. Because of this, whenever we load an RDB file with the `cluster-slot-imports` aux field, even from disk, we will still add a new migration to track the import. If after loading the RDB, the Valkey node is a primary, it will cancel the slot migration. Having this tracking state loaded on primaries will ensure that replicas partial syncing to a restarted primary still get their `SYNCSLOTS FINISH` message in the replication stream. ## AOF Since AOF cannot be used as the basis for a partial sync, we don't necessarily need to persist the `SYNCSLOTS ESTABLISH` and `FINISH` commands to the AOF. However, considering there is work to change this (#59 #1901) this design doesn't make any assumptions about this. We will propagate the `ESTABLISH` and `FINISH` commands to the AOF, and ensure that they can be properly replayed on AOF load to get to the right state. Similar to RDB, if there are any pending "ESTABLISH" commands that don't have a "FINISH" afterwards upon becoming primary, we will make sure to fail those in `verifyClusterConfigWithData`. Additionally, there was a bug in the existing slot migration where slot import clients were not having their commands persisted to AOF. This has been fixed by ensuring we still propagate to AOF even for slot import clients. ## Promotion & Demotion Since the primary is solely responsible for cleaning up unowned slots, primaries that are demoted will not clean up previously active slot imports. The promoted replica will be responsible for both cleaning up the slot (`verifyClusterConifgWithData`) and sending a `SYNCSLOTS FINISH`. # Other Options Considered I also considered tracking "dirty" slots rather than using the slot import state machine. In this setup, primaries and replicas would simply mark each slot's hashtable in the kvstore as dirty when something is written to it and we do not currently own that slot. This approach is simpler, but has a problem in that modules loaded on the replica would still not get slot migration start/end notifications. If the modules on the replica do not get such notifications, they will not be able to properly contain these dirty keys during slot migration events. --------- Signed-off-by: Jacob Murphy <[email protected]>

…ration (valkey-io#2635) # Problem In the current slot migration design, replicas are completely unaware of the slot migration. Because of this, they do not know to hide importing keys, which results in exposure of these keys to commands like KEYS, SCAN, RANDOMKEY, and DBSIZE. # Design The main part of the design is that we will now listen for and process the `SYNCSLOTS ESTABLISH` command on the replica. When a `SYNCSLOTS ESTABLISH` command is received from the primary, we begin tracking a new slot import in a special `SLOT_IMPORT_OCCURRING_ON_PRIMARY` state. Replicas use this state to track the import, and await for a future `SYNCSLOTS FINISH` message that tells them the import is successful/failed. ## Success Case ``` Source Target Target Replica | | | |------------ SYNCSLOTS ESTABLISH -------------->| | | |----- SYNCSLOTS ESTABLISH ------>| |<-------------------- +OK ----------------------| | | | | |~~~~~~~~~~~~~~ snapshot as AOF ~~~~~~~~~~~~~~~~>| | | |~~~~~~ forward snapshot ~~~~~~~~>| |----------- SYNCSLOTS SNAPSHOT-EOF ------------>| | | | | |<----------- SYNCSLOTS REQUEST-PAUSE -----------| | | | | |~~~~~~~~~~~~ incremental changes ~~~~~~~~~~~~~~>| | | |~~~~~~ forward changes ~~~~~~~~~>| |--------------- SYNCSLOTS PAUSED -------------->| | | | | |<---------- SYNCSLOTS REQUEST-FAILOVER ---------| | | | | |---------- SYNCSLOTS FAILOVER-GRANTED --------->| | | | | | (performs takeover & | | propagates topology) | | | | | |------- SYNCSLOTS FINISH ------->| (finds out about topology | | change & marks migration done) | | | | | ``` ## Failure Case ``` Source Target Target Replica | | | |------------ SYNCSLOTS ESTABLISH -------------->| | | |----- SYNCSLOTS ESTABLISH ------>| |<-------------------- +OK ----------------------| | ... ... ... | | | | <FAILURE> | | | | | (performs cleanup) | | | ~~~~~~ UNLINK <key> ... ~~~~~~~>| | | | | | ------ SYNCSLOTS FINISH ------->| | | | ``` ## Full Sync, Partial Sync, and RDB In order to ensure replicas that resync during the import are still aware of the import, the slot import is serialized to a new `cluster-slot-imports` aux field. The encoding includes the job name, the source node name, and the slot ranges being imported. Upon loading an RDB with the `cluster-slot-imports` aux field, replicas will add a new migration in the `SLOT_IMPORT_OCCURRING_ON_PRIMARY` state. It's important to note that a previously saved RDB file can be used as the basis for partial sync with a primary. Because of this, whenever we load an RDB file with the `cluster-slot-imports` aux field, even from disk, we will still add a new migration to track the import. If after loading the RDB, the Valkey node is a primary, it will cancel the slot migration. Having this tracking state loaded on primaries will ensure that replicas partial syncing to a restarted primary still get their `SYNCSLOTS FINISH` message in the replication stream. ## AOF Since AOF cannot be used as the basis for a partial sync, we don't necessarily need to persist the `SYNCSLOTS ESTABLISH` and `FINISH` commands to the AOF. However, considering there is work to change this (valkey-io#59 valkey-io#1901) this design doesn't make any assumptions about this. We will propagate the `ESTABLISH` and `FINISH` commands to the AOF, and ensure that they can be properly replayed on AOF load to get to the right state. Similar to RDB, if there are any pending "ESTABLISH" commands that don't have a "FINISH" afterwards upon becoming primary, we will make sure to fail those in `verifyClusterConfigWithData`. Additionally, there was a bug in the existing slot migration where slot import clients were not having their commands persisted to AOF. This has been fixed by ensuring we still propagate to AOF even for slot import clients. ## Promotion & Demotion Since the primary is solely responsible for cleaning up unowned slots, primaries that are demoted will not clean up previously active slot imports. The promoted replica will be responsible for both cleaning up the slot (`verifyClusterConifgWithData`) and sending a `SYNCSLOTS FINISH`. # Other Options Considered I also considered tracking "dirty" slots rather than using the slot import state machine. In this setup, primaries and replicas would simply mark each slot's hashtable in the kvstore as dirty when something is written to it and we do not currently own that slot. This approach is simpler, but has a problem in that modules loaded on the replica would still not get slot migration start/end notifications. If the modules on the replica do not get such notifications, they will not be able to properly contain these dirty keys during slot migration events. --------- Signed-off-by: Jacob Murphy <[email protected]>

…on (#2749) When working on #2635 I errorneously duplicated the setSlotImportingStateInAllDbs call for successful imports. This resulted in us doubling the key count in the kvstore. This results in DBSIZE reporting an incorrect sum, and also causes BIT corruption that can eventually result in a crash. The solution is: 1. Only call setSlotImportingStateInAllDbs once (in our finishSlotMigrationJob function) 2. Make setSlotImportingStateInAllDbs idempotent by checking if the delete from the kvstore importing hashtable is a no-op This also fixes a bug where the number of importing keys is not lowered after the migration, but this is less critical since it is only used when resizing the dictionary on RDB load. However, it could result in un-loadable RDBs if the importing key count gets large enough. Signed-off-by: Jacob Murphy <[email protected]>

…on (valkey-io#2749) When working on valkey-io#2635 I errorneously duplicated the setSlotImportingStateInAllDbs call for successful imports. This resulted in us doubling the key count in the kvstore. This results in DBSIZE reporting an incorrect sum, and also causes BIT corruption that can eventually result in a crash. The solution is: 1. Only call setSlotImportingStateInAllDbs once (in our finishSlotMigrationJob function) 2. Make setSlotImportingStateInAllDbs idempotent by checking if the delete from the kvstore importing hashtable is a no-op This also fixes a bug where the number of importing keys is not lowered after the migration, but this is less critical since it is only used when resizing the dictionary on RDB load. However, it could result in un-loadable RDBs if the importing key count gets large enough. Signed-off-by: Jacob Murphy <[email protected]> (cherry picked from commit 2a914aa)

…on (valkey-io#2749) When working on valkey-io#2635 I errorneously duplicated the setSlotImportingStateInAllDbs call for successful imports. This resulted in us doubling the key count in the kvstore. This results in DBSIZE reporting an incorrect sum, and also causes BIT corruption that can eventually result in a crash. The solution is: 1. Only call setSlotImportingStateInAllDbs once (in our finishSlotMigrationJob function) 2. Make setSlotImportingStateInAllDbs idempotent by checking if the delete from the kvstore importing hashtable is a no-op This also fixes a bug where the number of importing keys is not lowered after the migration, but this is less critical since it is only used when resizing the dictionary on RDB load. However, it could result in un-loadable RDBs if the importing key count gets large enough. Signed-off-by: Jacob Murphy <[email protected]> (cherry picked from commit 2a914aa) Signed-off-by: cherukum-amazon <[email protected]>

…on (valkey-io#2749) When working on valkey-io#2635 I errorneously duplicated the setSlotImportingStateInAllDbs call for successful imports. This resulted in us doubling the key count in the kvstore. This results in DBSIZE reporting an incorrect sum, and also causes BIT corruption that can eventually result in a crash. The solution is: 1. Only call setSlotImportingStateInAllDbs once (in our finishSlotMigrationJob function) 2. Make setSlotImportingStateInAllDbs idempotent by checking if the delete from the kvstore importing hashtable is a no-op This also fixes a bug where the number of importing keys is not lowered after the migration, but this is less critical since it is only used when resizing the dictionary on RDB load. However, it could result in un-loadable RDBs if the importing key count gets large enough. Signed-off-by: Jacob Murphy <[email protected]>

…on (#2749) When working on #2635 I errorneously duplicated the setSlotImportingStateInAllDbs call for successful imports. This resulted in us doubling the key count in the kvstore. This results in DBSIZE reporting an incorrect sum, and also causes BIT corruption that can eventually result in a crash. The solution is: 1. Only call setSlotImportingStateInAllDbs once (in our finishSlotMigrationJob function) 2. Make setSlotImportingStateInAllDbs idempotent by checking if the delete from the kvstore importing hashtable is a no-op This also fixes a bug where the number of importing keys is not lowered after the migration, but this is less critical since it is only used when resizing the dictionary on RDB load. However, it could result in un-loadable RDBs if the importing key count gets large enough. Signed-off-by: Jacob Murphy <[email protected]> (cherry picked from commit 2a914aa) Signed-off-by: cherukum-amazon <[email protected]>

github-actions bot assigned murphyjacob4 Sep 21, 2025

murphyjacob4 mentioned this pull request Sep 21, 2025

Atomic slot migraiton items for RC3 #2507

Closed

17 tasks

murphyjacob4 mentioned this pull request Sep 22, 2025

Add atomic slot migration test to validate DB and slot stats #2638

Draft

zuiderkwast reviewed Sep 23, 2025

View reviewed changes

murphyjacob4 requested a review from zuiderkwast September 26, 2025 00:52

zuiderkwast approved these changes Sep 29, 2025

View reviewed changes

madolson reviewed Sep 29, 2025

View reviewed changes

zuiderkwast added this to Valkey 9.0 Oct 1, 2025

zuiderkwast moved this to In Progress in Valkey 9.0 Oct 1, 2025

murphyjacob4 requested a review from madolson October 2, 2025 16:20

murphyjacob4 force-pushed the syncslots-to-repl branch from 2f5c997 to 26f9fb0 Compare October 2, 2025 17:16

madolson reviewed Oct 6, 2025

View reviewed changes

murphyjacob4 added 9 commits October 6, 2025 19:25

Prevent exposure of importing keys on replicas

fc29849

Signed-off-by: Jacob Murphy <[email protected]>

Update state machine with target replica

c46bd92

Signed-off-by: Jacob Murphy <[email protected]>

Clang format and typos

b6e9e62

Signed-off-by: Jacob Murphy <[email protected]>

Allow replicas to give full sync error message in failover testing

34a6f0b

Signed-off-by: Jacob Murphy <[email protected]>

Support slot imports in the AOF file

6537985

Signed-off-by: Jacob Murphy <[email protected]>

Address review feedback

a6de73e

Signed-off-by: Jacob Murphy <[email protected]>

Fix sending ACK when slot migration is occurring on primary

c2ad1fc

Signed-off-by: Jacob Murphy <[email protected]>

clang format

f8c9e1d

Signed-off-by: Jacob Murphy <[email protected]>

Remove source node ID from tracking on replicas, move to RDB opcode f…

7324270

…rom aux Signed-off-by: Jacob Murphy <[email protected]>

murphyjacob4 force-pushed the syncslots-to-repl branch from 6625e3a to 7324270 Compare October 6, 2025 19:26

madolson approved these changes Oct 6, 2025

View reviewed changes

murphyjacob4 added 3 commits October 6, 2025 22:53

Fix race condition during demotion causing desync

9922e89

Signed-off-by: Jacob Murphy <[email protected]>

Clang format

b16f7a7

Signed-off-by: Jacob Murphy <[email protected]>

Fix memory leak in RDB load

961614c

Signed-off-by: Jacob Murphy <[email protected]>

Clang format

d0a1e76

Signed-off-by: Jacob Murphy <[email protected]>

murphyjacob4 merged commit 383ffe4 into valkey-io:unstable Oct 7, 2025
53 checks passed

github-project-automation bot moved this from In Progress to To be backported in Valkey 9.0 Oct 7, 2025

murphyjacob4 moved this from To be backported to Done in Valkey 9.0 Oct 8, 2025

sarthakaggarwal97 mentioned this pull request Oct 9, 2025

[TEST-FAILURE] AOF maintains consistency through many migrations #2712

Closed

murphyjacob4 mentioned this pull request Oct 11, 2025

Improve atomic slot migration resilience to source and target shard failover #2724

Open

murphyjacob4 mentioned this pull request Oct 18, 2025

Fix incorrect kvstore size and BIT accounting after completed migration #2749

Merged

Prevent exposure of importing keys on replicas during atomic slot migration #2635

Prevent exposure of importing keys on replicas during atomic slot migration #2635

Uh oh!

Conversation

murphyjacob4 commented Sep 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem

Design

Success Case

Failure Case

Full Sync, Partial Sync, and RDB

AOF

Promotion & Demotion

Other Options Considered

Uh oh!

codecov bot commented Sep 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

zuiderkwast left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

zuiderkwast left a comment

Choose a reason for hiding this comment

Uh oh!

zuiderkwast Sep 29, 2025

Choose a reason for hiding this comment

Uh oh!

murphyjacob4 Oct 2, 2025

Choose a reason for hiding this comment

Uh oh!

murphyjacob4 Oct 2, 2025

Choose a reason for hiding this comment

Uh oh!

zuiderkwast commented Sep 29, 2025

Uh oh!

madolson left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

murphyjacob4 commented Oct 2, 2025

Uh oh!

madolson left a comment

Choose a reason for hiding this comment

Uh oh!

murphyjacob4 commented Oct 6, 2025

Uh oh!

madolson left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

murphyjacob4 commented Sep 21, 2025 •

edited

Loading

codecov bot commented Sep 21, 2025 •

edited

Loading