Repository navigation
Commit 85bb569
authored
Persist effect result types as ids through a dedicated type store (#254)
* Rename ITypeStore to IFlowTypeStore and its implementations
* Persist effect result types as ids through a dedicated type store
StoredEffect.ResultType is now a long - the first 8 bytes of the SHA-256
hash of the type's encoded form (UTF-8 of its simple qualified name) -
instead of the inline encoded type. The new ITypeStore persists the
id -> encoded-type mapping in a {prefix}_dotnet_types table in every
store; ids are content-derived, so inserts are idempotent.
The registry-wide TypeMapper computes ids without touching the store and
tracks which mappings are known-persisted. Every effect write path
awaits TypeMapper.EnsurePersisted before handing effects to the store
(EffectResults.Flush, control-panel effect writes, staged-message
children and CreateFunction's initial effects), so a mapping row is
always durable before the first effect referencing it - a crash between
the two writes can never leave effects whose results cannot be
deserialized. Resolution of an unknown id refreshes the whole (small)
mapping table once per process.
The flow-type store moves to IFunctionStore.FlowTypeStore, freeing the
TypeStore property for the new store. ISerializer no longer serializes
types at all: the SerializeType/ResolveType default methods are deleted
and the encoding lives in TypeHelper extensions, shared by the TypeMapper
and the message paths (where message types remain inline bytes).
* Track persistedness by insertion order instead of a separate marker set
_serializedTypes now only ever contains mappings that are durable in the
type store: an entry is added after its insert has completed (or from a
store refresh), never before - making the dictionary itself the persisted
marker and removing the separate _persisted set. Concurrent
EnsurePersisted calls may insert the same mapping more than once; inserts
are idempotent, so duplicates are accepted rather than queued behind a
lock.
Bytes for a minted-but-not-yet-persisted id are recovered by a reverse
lookup over the ids GetTypeId has handed out - both when persisting the
mapping and when an effect created in this process is read back before
its first flush.
* Persist message types as TypeIds and thread the id through the message model
TypeId - a readonly struct wrapping the content-derived long - replaces
raw longs and inline encoded types everywhere a type travels: an effect's
ResultType, SerializedMessage.Type, StoredMessage/StoredDlqMessage's
MessageType and the PendingMessages effect-carrier encoding all carry a
TypeId now (8 bytes in the packed forms). A null message type marks an
empty restart-poke - the poke's empty-content-and-type encoding is gone.
Message producers mint ids through the registry's TypeMapper and
consumers resolve through it; the ISerializer-independent inline type
bytes are gone from the message pipeline. EnsurePersisted becomes
parameterless - it persists every id minted by this process that is not
yet known durable - so ids buried inside already-encoded payloads
(staged-message children, delivered-message captures) are covered by the
effect flush without threading them around, and MessageSender persists
before appending rows for the same reason.
Test-side, DefaultDeserialize takes the mapper it resolves through, and
directly-constructed messages mint-and-persist their type id via the new
TypeIdTestHelper.
* Name the type-mapping table _types and the flow-type table _flowtypes
The .NET-type mapping is the primary type table, so it takes the _types
name; the flow-type table moves to _flowtypes (matching its
IFlowTypeStore rename on main).
* Track unpersisted type mappings directly instead of sweeping all minted ids
GetTypeId records a newly minted mapping in an unpersisted dictionary,
making EnsurePersisted's fast path an emptiness check instead of an
iteration over every minted type, and making unpersisted bytes
addressable by id - which replaces the reverse lookup over minted ids in
both the persist and the read-back-before-first-flush resolution paths.
Entries move to the persisted dictionary before they are drained, so a
concurrent reader always finds a mapping in at least one of the two.
* Cache resolved TypeId -> Type lookups
GetTypeId seeds the cache with the Type it already has in hand, and the
first resolution of a foreign id fills it lazily - so repeated
ResolveType calls are a single dictionary lookup instead of a
Type.GetType round-trip over the encoded name.
* Resolve type ids asynchronously instead of blocking on the store
TypeMapper.ResolveType returns a Task<Type>: the refresh a foreign id
falls back on was blocked on with GetAwaiter().GetResult(), and three of
its four call chains reached it from inside EffectResults' _sync lock -
so one first-resolution parked a thread pool thread on a store
round-trip and held every other effect operation on that flow behind it.
Nothing required the synchronous signature; every call site bottoms out
in an already-async method.
EffectResults' three deserializing paths (TryGet, CreateOrGet and the
generic InnerCapture) now snapshot the StoredEffect under the lock and
resolve outside it - safe because the pending change and the stored
effect are immutable records - so no store round-trip happens while the
effect state is locked. The new await sits only on the already-completed
return path, leaving the write paths' interleaving unchanged.
Out params cannot survive async, so EffectResults/Effect.TryGet return
(bool Success, T? Value) and Effect.Get returns a Task; both are
internal. IdempotencyKeys.Initialize follows its caller into async. The
public surface changes are ResolveType, the ResolveResultType extension
and the two test-only DefaultDeserialize methods, which now return tasks
- the test helper blocks on them rather than rippling awaits through
assertion sites, some of which sit inside LINQ predicates.
* Admit one type-store refresh at a time
Resolving an unknown id refreshes the whole mapping table, so concurrent
misses each fetched it in full. A restart reading back a batch of
foreign payloads misses on many distinct ids at once, which made the cost
scale with the number of unknown types rather than with the one fetch
that already covers all of them.
A semaphore admits one refresh, and waiters re-check the persisted
mappings before reaching the store: the refresh they queued behind
fetched every type, so their id is present and they return without
further I/O. Only an id that genuinely is not in the store - the path to
the TypeLoadException below - refreshes again, so a mapping written
between the two attempts is still picked up rather than cached away as a
permanent miss.1 parent dfc5c06 commit 85bb569
73 files changed
Lines changed: 1062 additions & 406 deletions
File tree
- Core
- Cleipnir.ResilientFunctions.Tests
- InMemoryTests
- Messaging
- InMemoryTests
- TestTemplates
- TestTemplates
- FunctionTests
- WatchDogsTests
- UtilsTests
- Utils
- Cleipnir.ResilientFunctions
- CoreRuntime
- Invocation
- Serialization
- Domain
- Helpers
- Messaging
- Queuing
- Storage
- Samples/Sample.ConsoleApp/Utils
- Stores
- MariaDB
- Cleipnir.ResilientFunctions.MariaDB.Tests
- Cleipnir.ResilientFunctions.MariaDB
- PostgreSQL
- Cleipnir.ResilientFunctions.PostgreSQL.Tests
- Cleipnir.ResilientFunctions.PostgreSQL
- SqlServer
- Cleipnir.ResilientFunctions.SqlServer.Tests
- Cleipnir.ResilientFunctions.SqlServer
- StressTests/StressTests
Some content is hidden
Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
Lines changed: 3 additions & 1 deletion
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
26 | 26 | | |
27 | 27 | | |
28 | 28 | | |
| 29 | + | |
29 | 30 | | |
30 | 31 | | |
31 | 32 | | |
| |||
133 | 134 | | |
134 | 135 | | |
135 | 136 | | |
136 | | - | |
| 137 | + | |
| 138 | + | |
137 | 139 | | |
138 | 140 | | |
139 | 141 | | |
| |||
Lines changed: 18 additions & 9 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
25 | 25 | | |
26 | 26 | | |
27 | 27 | | |
| 28 | + | |
28 | 29 | | |
29 | 30 | | |
30 | 31 | | |
| |||
38 | 39 | | |
39 | 40 | | |
40 | 41 | | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
41 | 50 | | |
42 | 51 | | |
43 | 52 | | |
| |||
55 | 64 | | |
56 | 65 | | |
57 | 66 | | |
58 | | - | |
| 67 | + | |
59 | 68 | | |
60 | 69 | | |
61 | 70 | | |
| |||
72 | 81 | | |
73 | 82 | | |
74 | 83 | | |
75 | | - | |
| 84 | + | |
76 | 85 | | |
77 | 86 | | |
78 | 87 | | |
| |||
89 | 98 | | |
90 | 99 | | |
91 | 100 | | |
92 | | - | |
| 101 | + | |
93 | 102 | | |
94 | 103 | | |
95 | 104 | | |
| |||
106 | 115 | | |
107 | 116 | | |
108 | 117 | | |
109 | | - | |
| 118 | + | |
110 | 119 | | |
111 | 120 | | |
112 | 121 | | |
| |||
133 | 142 | | |
134 | 143 | | |
135 | 144 | | |
136 | | - | |
| 145 | + | |
137 | 146 | | |
138 | 147 | | |
139 | 148 | | |
| |||
157 | 166 | | |
158 | 167 | | |
159 | 168 | | |
160 | | - | |
| 169 | + | |
161 | 170 | | |
162 | 171 | | |
163 | 172 | | |
| |||
181 | 190 | | |
182 | 191 | | |
183 | 192 | | |
184 | | - | |
| 193 | + | |
185 | 194 | | |
186 | 195 | | |
187 | 196 | | |
| |||
197 | 206 | | |
198 | 207 | | |
199 | 208 | | |
200 | | - | |
| 209 | + | |
201 | 210 | | |
202 | 211 | | |
203 | 212 | | |
| |||
221 | 230 | | |
222 | 231 | | |
223 | 232 | | |
224 | | - | |
| 233 | + | |
225 | 234 | | |
226 | 235 | | |
227 | 236 | | |
| |||
Lines changed: 2 additions & 28 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
18 | 18 | | |
19 | 19 | | |
20 | 20 | | |
21 | | - | |
22 | | - | |
| 21 | + | |
| 22 | + | |
23 | 23 | | |
24 | 24 | | |
25 | 25 | | |
| |||
65 | 65 | | |
66 | 66 | | |
67 | 67 | | |
68 | | - | |
69 | | - | |
70 | | - | |
71 | | - | |
72 | | - | |
73 | | - | |
74 | | - | |
75 | | - | |
76 | | - | |
77 | | - | |
78 | | - | |
79 | | - | |
80 | | - | |
81 | | - | |
82 | | - | |
83 | | - | |
84 | | - | |
85 | | - | |
86 | | - | |
87 | | - | |
88 | | - | |
89 | | - | |
90 | | - | |
91 | | - | |
92 | | - | |
93 | | - | |
94 | 68 | | |
Lines changed: 4 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
142 | 142 | | |
143 | 143 | | |
144 | 144 | | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
145 | 149 | | |
146 | 150 | | |
147 | 151 | | |
| |||
Lines changed: 5 additions & 3 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1 | 1 | | |
| 2 | + | |
2 | 3 | | |
3 | 4 | | |
4 | 5 | | |
| |||
199 | 200 | | |
200 | 201 | | |
201 | 202 | | |
202 | | - | |
| 203 | + | |
203 | 204 | | |
204 | 205 | | |
205 | 206 | | |
206 | | - | |
| 207 | + | |
| 208 | + | |
207 | 209 | | |
208 | 210 | | |
209 | 211 | | |
| |||
212 | 214 | | |
213 | 215 | | |
214 | 216 | | |
215 | | - | |
| 217 | + | |
216 | 218 | | |
217 | 219 | | |
218 | 220 | | |
| |||
Lines changed: 1 addition & 1 deletion
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
137 | 137 | | |
138 | 138 | | |
139 | 139 | | |
140 | | - | |
| 140 | + | |
141 | 141 | | |
142 | 142 | | |
143 | 143 | | |
| |||
Lines changed: 22 additions & 22 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
47 | 47 | | |
48 | 48 | | |
49 | 49 | | |
50 | | - | |
51 | | - | |
52 | | - | |
53 | | - | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
54 | 54 | | |
55 | 55 | | |
56 | 56 | | |
| |||
67 | 67 | | |
68 | 68 | | |
69 | 69 | | |
70 | | - | |
| 70 | + | |
71 | 71 | | |
72 | 72 | | |
73 | 73 | | |
| |||
99 | 99 | | |
100 | 100 | | |
101 | 101 | | |
102 | | - | |
103 | | - | |
| 102 | + | |
| 103 | + | |
104 | 104 | | |
105 | 105 | | |
106 | 106 | | |
| |||
130 | 130 | | |
131 | 131 | | |
132 | 132 | | |
133 | | - | |
| 133 | + | |
134 | 134 | | |
135 | 135 | | |
136 | 136 | | |
| |||
139 | 139 | | |
140 | 140 | | |
141 | 141 | | |
142 | | - | |
| 142 | + | |
143 | 143 | | |
144 | 144 | | |
145 | 145 | | |
| |||
173 | 173 | | |
174 | 174 | | |
175 | 175 | | |
176 | | - | |
177 | | - | |
| 176 | + | |
| 177 | + | |
178 | 178 | | |
179 | 179 | | |
180 | 180 | | |
| |||
183 | 183 | | |
184 | 184 | | |
185 | 185 | | |
186 | | - | |
| 186 | + | |
187 | 187 | | |
188 | 188 | | |
189 | 189 | | |
| |||
205 | 205 | | |
206 | 206 | | |
207 | 207 | | |
208 | | - | |
| 208 | + | |
209 | 209 | | |
210 | 210 | | |
211 | 211 | | |
| |||
214 | 214 | | |
215 | 215 | | |
216 | 216 | | |
217 | | - | |
218 | | - | |
| 217 | + | |
| 218 | + | |
219 | 219 | | |
220 | 220 | | |
221 | 221 | | |
222 | 222 | | |
223 | | - | |
224 | | - | |
| 223 | + | |
| 224 | + | |
225 | 225 | | |
226 | 226 | | |
227 | 227 | | |
228 | | - | |
| 228 | + | |
229 | 229 | | |
230 | 230 | | |
231 | 231 | | |
| |||
244 | 244 | | |
245 | 245 | | |
246 | 246 | | |
247 | | - | |
| 247 | + | |
248 | 248 | | |
249 | 249 | | |
250 | 250 | | |
| |||
258 | 258 | | |
259 | 259 | | |
260 | 260 | | |
261 | | - | |
| 261 | + | |
262 | 262 | | |
263 | 263 | | |
264 | 264 | | |
265 | 265 | | |
266 | | - | |
| 266 | + | |
267 | 267 | | |
268 | 268 | | |
269 | 269 | | |
270 | | - | |
| 270 | + | |
271 | 271 | | |
272 | 272 | | |
273 | 273 | | |
| |||
0 commit comments