You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
fix: revert adding reclaim_memory to memory manager (#396)
This PR reverts adding `reclaim_memory` method.
Reverts several recent commits in a single change:
- `6e397bd` fix: use conservative bucket reuse that survives reload #394
- `00468e1` docs: update memory reclamation examples in the docs #392
- `b911479` docs: cleanup documentation #391
- `d1fde89` docs: use reclaim_memory() name and update docs accordingly
#388
- `a18917b` docs: add safety documentation and tests for manual bucket
release #387
- `73e96e8` feat: add manual bucket release to prevent memory waste #386
This PR restores the codebase to the state before these commits.
Done with `git revert -n 6e397bd00468e1b911479d1fde89a18917b73e96e8`
The reason for reverting this approach was that it can reclaim unused
memory in theory but provides little benefit in real-world migrations.
All due to the requirement to keep buckets in ascending order in each
VM.
Example 1: Reuse works
```
A allocates: [0, 4, 5]
B allocates: [1, 2, 3]
A frees: [0, 4, 5]
B grows: can reuse bucket 4 (since 4 > max(B) = 3)
B after grow: [1, 2, 3, 4]
```
Example 2: Reuse fails
```
A allocates: [0, 1, 2]
B allocates: [4, 5, 6]
A frees: [0, 1, 2]
B grows: cannot reuse any freed bucket (all < max(B) = 6), so allocates new bucket 7
B after grow: [4, 5, 6, 7]
```
In real life when migrating state A to state B, state B created after
state A grown, so it's first bucket ID is already higher than any free
bucket in state A virtual memory, therefore can not be reused.
Memories are abstracted with the [Memory] trait, and stable structures can work with any storage
@@ -58,108 +58,52 @@ This includes stable memory, a vector ([VectorMemory]), or even a flat file ([Fi
58
58
59
59
The example above initializes a [BTreeMap] with a [DefaultMemoryImpl], which maps to stable memory when used in a canister and to a [VectorMemory] otherwise.
Note that **stable structures cannot share memories.**
77
+
Each memory must belong to only one stable structure.
66
78
For example, this fails when run in a canister:
67
79
68
-
```rust,ignore
80
+
```no_run
69
81
use ic_stable_structures::{BTreeMap, DefaultMemoryImpl};
70
-
let mut map_a: BTreeMap<u64, u8, _> = BTreeMap::init(DefaultMemoryImpl::default());
71
-
let mut map_b: BTreeMap<u64, u8, _> = BTreeMap::init(DefaultMemoryImpl::default());
82
+
let mut map_1: BTreeMap<u64, u64, _> = BTreeMap::init(DefaultMemoryImpl::default());
83
+
let mut map_2: BTreeMap<u64, u64, _> = BTreeMap::init(DefaultMemoryImpl::default());
72
84
73
-
map_a.insert(1, b'A');
74
-
map_b.insert(1, b'B');
75
-
assert_eq!(map_a.get(&1), Some(b'A')); // ❌ FAILS: Returns b'B' due to shared memory!
76
-
assert_eq!(map_b.get(&1), Some(b'B')); // ✅ Succeeds, but corrupted map_a
85
+
map_1.insert(1, 2);
86
+
map_2.insert(1, 3);
87
+
assert_eq!(map_1.get(&1), Some(2)); // This assertion fails.
77
88
```
78
89
79
-
It fails because both `map_a` and `map_b` are using the same stable memory under the hood, and so changes in `map_b` end up changing or corrupting `map_a`.
90
+
It fails because both `map_1` and `map_2` are using the same stable memory under the hood, and so changes in `map_1` end up changing or corrupting `map_2`.
80
91
81
-
### Using MemoryManager
82
-
83
-
To address this issue, we use the [MemoryManager](memory_manager::MemoryManager), which takes a single memory and creates up to 255 virtual memories for our use.
84
-
Here's the above failing example, but fixed:
92
+
To address this issue, we make use of the [MemoryManager](memory_manager::MemoryManager), which takes a single memory and creates up to 255 virtual memories for our disposal.
93
+
Here's the above failing example, but fixed by using the [MemoryManager](memory_manager::MemoryManager):
assert_eq!(map_a.get(&1), Some(b'A')); // ✅ Succeeds: Each map has its own memory
99
-
assert_eq!(map_b.get(&1), Some(b'B')); // ✅ Succeeds: No data corruption
100
-
```
101
-
102
-
### Memory Reclamation
103
-
104
-
During migrations you often create a new structure (B) and copy data from an existing one (A). Without reclamation, this can double memory usage even after A is no longer needed.
105
-
106
-
Bucket IDs are an internal implementation detail — hidden and not user-controllable — and each virtual memory must receive bucket IDs in strictly ascending order. Because of this, reuse of freed buckets is guaranteed when allocating into a newly created (empty) structure. For existing structures, reuse may or may not work: it succeeds only if there is a free bucket with an ID greater than the structure’s current maximum; otherwise a new bucket is allocated.
107
-
108
-
Example: A = `[0, 4, 5]`, B = `[1, 2, 3]`. After releasing A, `free = [0, 4, 5]`. When B grows, it can’t take `0` (must be `> 3`) but can take `4` → `B = [1, 2, 3, 4]`, `free = [0, 5]`.
109
-
110
-
**Recommendation:** for predictable reuse migrate into a newly created structure rather than relying on reuse with a populated one.
111
-
112
-
> **⚠️ CRITICAL SAFETY REQUIREMENT:**
113
-
> -**MUST** drop the original structure object before calling `reclaim_memory`.
114
-
> -**NEVER** use the original structure after reclamation — doing so corrupts data.
115
-
116
-
The `MemoryManager` provides a `reclaim_memory` method to efficiently handle these scenarios:
-[Basic Example](https://github.com/dfinity/stable-structures/tree/main/examples/src/basic_example) (the one above)
217
161
-[Quickstart Example](https://github.com/dfinity/stable-structures/tree/main/examples/src/quick_start): Ideal as a template when developing a new canister
218
162
-[Custom Types Example](https://github.com/dfinity/stable-structures/tree/main/examples/src/custom_types_example): Showcases storing your own custom types
219
163
220
164
## Combined Persistence
221
165
222
-
If your project uses only stable structures, memory can expand in size without requiring`pre_upgrade`/`post_upgrade` hooks.
166
+
If your project exclusively relies on stable structures, the memory can expand in size without the requirement of`pre_upgrade`/`post_upgrade` hooks.
223
167
224
-
However, if you also need to serialize/deserialize heap data, you must use the memory manager to avoid conflicts. To combine both approaches effectively, refer to the [Quickstart Example](https://github.com/dfinity/stable-structures/tree/main/examples/src/quick_start) for guidance.
168
+
However, it's important to note that if you also intend to perform serialization/deserialization of the heap data, utilizing the memory manager becomes necessary. To effectively combine both approaches, refer to the [Quickstart Example](https://github.com/dfinity/stable-structures/tree/main/examples/src/quick_start) for guidance.
225
169
226
170
## Fuzzing
227
171
228
-
Stable structures require strong guarantees to work reliably and scale over millions of operations. To that extent, we use fuzzing to emulate such operations on the available data structures.
172
+
Stable structures requires strong guarantees to work reliably and scale over millions of operations. To that extent, we use fuzzing to emulate such operations on the available data structures.
0 commit comments