Skip to content

feat: SQLite persistence core#1358

Merged
kevin-dp merged 150 commits intomainfrom
kevin/persistence
Mar 16, 2026
Merged

feat: SQLite persistence core#1358
kevin-dp merged 150 commits intomainfrom
kevin/persistence

Conversation

@kevin-dp
Copy link
Contributor

@kevin-dp kevin-dp commented Mar 12, 2026

Summary

Replaces #1230 — this PR contains the core persistence packages only, split out for easier review:

  • db-sqlite-persisted-collection-core — shared SQLite adapter, persistence runtime, tx pruning, and coordinator protocol
  • db-browser-wa-sqlite-persisted-collection — browser persistence via wa-sqlite (OPFS) with multi-tab coordinator (BroadcastChannel + Web Locks)
  • db-react-native-sqlite-persisted-collection — React Native/Expo persistence via expo-sqlite
  • db — index lifecycle events, removeIndex API, and hydration support for persistence

Runtime-specific persistence layers are in follow-up PRs:

Implements #865 (comment) from #865

Test plan

  • Unit and contract tests pass for db-sqlite-persisted-collection-core
  • Browser wa-sqlite persistence tests pass (single-tab + multi-tab coordinator)
  • React Native/Expo persistence and lifecycle tests pass
  • db package index lifecycle tests pass
  • pnpm test:pr passes

🤖 Generated with Claude Code

@changeset-bot
Copy link

changeset-bot bot commented Mar 12, 2026

🦋 Changeset detected

Latest commit: 8dc3daa

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 15 packages
Name Type
@tanstack/db Patch
@tanstack/db-sqlite-persisted-collection-core Patch
@tanstack/db-browser-wa-sqlite-persisted-collection Patch
@tanstack/db-react-native-sqlite-persisted-collection Patch
@tanstack/angular-db Patch
@tanstack/electric-db-collection Patch
@tanstack/offline-transactions Patch
@tanstack/powersync-db-collection Patch
@tanstack/query-db-collection Patch
@tanstack/react-db Patch
@tanstack/rxdb-db-collection Patch
@tanstack/solid-db Patch
@tanstack/svelte-db Patch
@tanstack/trailbase-db-collection Patch
@tanstack/vue-db Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@pkg-pr-new
Copy link

pkg-pr-new bot commented Mar 12, 2026

More templates

@tanstack/angular-db

npm i https://pkg.pr.new/TanStack/db/@tanstack/angular-db@1358

@tanstack/db

npm i https://pkg.pr.new/TanStack/db/@tanstack/db@1358

@tanstack/db-browser-wa-sqlite-persisted-collection

npm i https://pkg.pr.new/TanStack/db/@tanstack/db-browser-wa-sqlite-persisted-collection@1358

@tanstack/db-ivm

npm i https://pkg.pr.new/TanStack/db/@tanstack/db-ivm@1358

@tanstack/db-react-native-sqlite-persisted-collection

npm i https://pkg.pr.new/TanStack/db/@tanstack/db-react-native-sqlite-persisted-collection@1358

@tanstack/db-sqlite-persisted-collection-core

npm i https://pkg.pr.new/TanStack/db/@tanstack/db-sqlite-persisted-collection-core@1358

@tanstack/electric-db-collection

npm i https://pkg.pr.new/TanStack/db/@tanstack/electric-db-collection@1358

@tanstack/offline-transactions

npm i https://pkg.pr.new/TanStack/db/@tanstack/offline-transactions@1358

@tanstack/powersync-db-collection

npm i https://pkg.pr.new/TanStack/db/@tanstack/powersync-db-collection@1358

@tanstack/query-db-collection

npm i https://pkg.pr.new/TanStack/db/@tanstack/query-db-collection@1358

@tanstack/react-db

npm i https://pkg.pr.new/TanStack/db/@tanstack/react-db@1358

@tanstack/rxdb-db-collection

npm i https://pkg.pr.new/TanStack/db/@tanstack/rxdb-db-collection@1358

@tanstack/solid-db

npm i https://pkg.pr.new/TanStack/db/@tanstack/solid-db@1358

@tanstack/svelte-db

npm i https://pkg.pr.new/TanStack/db/@tanstack/svelte-db@1358

@tanstack/trailbase-db-collection

npm i https://pkg.pr.new/TanStack/db/@tanstack/trailbase-db-collection@1358

@tanstack/vue-db

npm i https://pkg.pr.new/TanStack/db/@tanstack/vue-db@1358

commit: 8dc3daa

@kevin-dp kevin-dp mentioned this pull request Mar 12, 2026
4 tasks
@github-actions
Copy link
Contributor

github-actions bot commented Mar 12, 2026

Size Change: +1.93 kB (+1.8%)

Total Size: 109 kB

Filename Size Change
./packages/db/dist/esm/collection/events.js 434 B +46 B (+11.86%) ⚠️
./packages/db/dist/esm/collection/index.js 3.69 kB +241 B (+6.99%) 🔍
./packages/db/dist/esm/collection/indexes.js 2.35 kB +1.25 kB (+113.57%) 🆘
./packages/db/dist/esm/index.js 2.85 kB +71 B (+2.55%)
./packages/db/dist/esm/indexes/lazy-index.js 1.24 kB +135 B (+12.24%) ⚠️
./packages/db/dist/esm/query/compiler/evaluators.js 1.62 kB +189 B (+13.23%) ⚠️
ℹ️ View Unchanged
Filename Size
./packages/db/dist/esm/collection/change-events.js 1.39 kB
./packages/db/dist/esm/collection/changes.js 1.38 kB
./packages/db/dist/esm/collection/lifecycle.js 1.75 kB
./packages/db/dist/esm/collection/mutations.js 2.47 kB
./packages/db/dist/esm/collection/state.js 5.2 kB
./packages/db/dist/esm/collection/subscription.js 3.71 kB
./packages/db/dist/esm/collection/sync.js 2.43 kB
./packages/db/dist/esm/collection/transaction-metadata.js 144 B
./packages/db/dist/esm/deferred.js 207 B
./packages/db/dist/esm/errors.js 4.83 kB
./packages/db/dist/esm/event-emitter.js 748 B
./packages/db/dist/esm/indexes/auto-index.js 777 B
./packages/db/dist/esm/indexes/base-index.js 766 B
./packages/db/dist/esm/indexes/btree-index.js 2.17 kB
./packages/db/dist/esm/indexes/reverse-index.js 538 B
./packages/db/dist/esm/local-only.js 890 B
./packages/db/dist/esm/local-storage.js 2.1 kB
./packages/db/dist/esm/optimistic-action.js 359 B
./packages/db/dist/esm/paced-mutations.js 496 B
./packages/db/dist/esm/proxy.js 3.75 kB
./packages/db/dist/esm/query/builder/functions.js 792 B
./packages/db/dist/esm/query/builder/index.js 5.15 kB
./packages/db/dist/esm/query/builder/ref-proxy.js 1.05 kB
./packages/db/dist/esm/query/compiler/expressions.js 430 B
./packages/db/dist/esm/query/compiler/group-by.js 2.69 kB
./packages/db/dist/esm/query/compiler/index.js 3.62 kB
./packages/db/dist/esm/query/compiler/joins.js 2.11 kB
./packages/db/dist/esm/query/compiler/order-by.js 1.5 kB
./packages/db/dist/esm/query/compiler/select.js 1.11 kB
./packages/db/dist/esm/query/effect.js 4.78 kB
./packages/db/dist/esm/query/expression-helpers.js 1.43 kB
./packages/db/dist/esm/query/ir.js 784 B
./packages/db/dist/esm/query/live-query-collection.js 360 B
./packages/db/dist/esm/query/live/collection-config-builder.js 7.63 kB
./packages/db/dist/esm/query/live/collection-registry.js 264 B
./packages/db/dist/esm/query/live/collection-subscriber.js 1.94 kB
./packages/db/dist/esm/query/live/internal.js 145 B
./packages/db/dist/esm/query/live/utils.js 1.57 kB
./packages/db/dist/esm/query/optimizer.js 2.62 kB
./packages/db/dist/esm/query/predicate-utils.js 2.97 kB
./packages/db/dist/esm/query/query-once.js 359 B
./packages/db/dist/esm/query/subset-dedupe.js 960 B
./packages/db/dist/esm/scheduler.js 1.3 kB
./packages/db/dist/esm/SortedMap.js 1.3 kB
./packages/db/dist/esm/strategies/debounceStrategy.js 247 B
./packages/db/dist/esm/strategies/queueStrategy.js 428 B
./packages/db/dist/esm/strategies/throttleStrategy.js 246 B
./packages/db/dist/esm/transactions.js 2.9 kB
./packages/db/dist/esm/utils.js 927 B
./packages/db/dist/esm/utils/browser-polyfills.js 304 B
./packages/db/dist/esm/utils/btree.js 5.61 kB
./packages/db/dist/esm/utils/comparison.js 1.05 kB
./packages/db/dist/esm/utils/cursor.js 457 B
./packages/db/dist/esm/utils/index-optimization.js 1.54 kB
./packages/db/dist/esm/utils/type-guards.js 157 B
./packages/db/dist/esm/virtual-props.js 360 B

compressed-size-action::db-package-size

@github-actions
Copy link
Contributor

github-actions bot commented Mar 12, 2026

Size Change: 0 B

Total Size: 4.23 kB

ℹ️ View Unchanged
Filename Size
./packages/react-db/dist/esm/index.js 249 B
./packages/react-db/dist/esm/useLiveInfiniteQuery.js 1.32 kB
./packages/react-db/dist/esm/useLiveQuery.js 1.34 kB
./packages/react-db/dist/esm/useLiveQueryEffect.js 355 B
./packages/react-db/dist/esm/useLiveSuspenseQuery.js 559 B
./packages/react-db/dist/esm/usePacedMutations.js 401 B

compressed-size-action::react-db-package-size

@kevin-dp kevin-dp force-pushed the kevin/persistence branch 2 times, most recently from 38ea25f to 9ce6ae0 Compare March 12, 2026 10:59
Copy link
Collaborator

@samwillis samwillis left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's get this one in immediately after merging the includes.

Two things:

  • Need to remove the PERSISTNCE-PLAN-SQLITE-ONLY.md doc and the /persistance-plan dir.
  • add a changeset (patch only)

Note: On release we will need @KyleAMathews to do the initial push of the new packages to npm locally as CI will not do that.

@kevin-dp kevin-dp force-pushed the kevin/persistence branch from fed2e62 to 37c3bf5 Compare March 16, 2026 11:16
cursoragent and others added 23 commits March 16, 2026 12:24
…cate tx skipping

The PersistedCollectionRuntime never restored its stream position from
the database on startup, always beginning at localTerm=1, localSeq=0.
After a page reload, the first new mutation would collide with a
previously applied transaction (term=1, seq=1), causing the SQLite
adapter's applyCommittedTx to silently skip it as a duplicate.

Add getStreamPosition to PersistenceAdapter (optional) and implement it
in SQLiteCorePersistenceAdapter. Call it from startInternal() so that
observeStreamPosition seeds the local counters before any mutations.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Include full row values in the tx:committed message so receiving tabs
can apply changes directly without a SQLite round-trip via loadRowsByKeys.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…ti-tab support

Web Locks for per-collection leadership election, BroadcastChannel for
cross-tab RPC transport, DB writer lock for SQLite write serialization,
envelope dedup for exactly-once mutations, and leader heartbeats.
Includes 15 unit tests with Web Locks/BroadcastChannel mocks.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Previously, `state.isLeader = true` was set before the setup code that
calls `getStreamPosition()`. If `getStreamPosition` threw (e.g. due to
a UNIQUE constraint violation from React StrictMode double-mounting),
`isLeader` remained permanently stuck at `true` because the `finally`
block that resets it was inside an inner try/finally that was never
entered.

Fix: Wrap the entire lock callback body in a single try/finally. Set
`state.isLeader = true` only after successful setup (stream position
restore and term increment). The finally block always runs and resets
`isLeader = false` + cleans up the heartbeat timer.

Also refactors the coordinator to support lazy adapter wiring via
`setAdapter()`, allowing `createBrowserWASQLitePersistence` to inject
the adapter after construction. This enables the demo to construct the
coordinator without requiring the adapter upfront.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…nt seq collisions

The leader tab had two mutation paths: a "direct" path (write to SQLite
and broadcast) and an RPC path (through the coordinator). Previously,
only follower tabs used the RPC path — the leader bypassed the
coordinator and wrote directly.

This caused a seq collision: the leader's direct writes incremented the
runtime's `localSeq` but left the coordinator's `state.latestSeq` at 0.
When a follower later sent an RPC, the coordinator assigned seq starting
from 1 again, producing duplicate seq numbers. The leader then skipped
these "already-seen" tx:committed messages, causing follower mutations
to silently disappear.

Fix: Always route through `requestApplyLocalMutations` when available,
regardless of leader/follower status. This keeps the coordinator's seq
counter in sync with all writes.

Also removes `requestApplyLocalMutations` from `SingleProcessCoordinator`
— it was a stub that returned success without persisting, which would
break now that the leader uses this path. Single-process mode correctly
falls back to the direct path since it has no multi-tab coordination.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…mmitted messages

The leader tab's `onCoordinatorMessage` handler skipped ALL messages
where `senderId` matched the coordinator's own node ID. But when the
coordinator processes a follower's RPC in `handleApplyLocalMutations`,
it delivers the resulting `tx:committed` to local subscribers using the
coordinator's own `senderId`. This caused the leader's runtime to
silently ignore follower mutations — they were written to SQLite but
never applied to the leader's in-memory collection.

Fix: Allow `tx:committed` messages from self to pass through the filter.
The seq dedup logic in `processCommittedTxUnsafe` already prevents
double-processing: when the leader's own mutations go through the
coordinator, `observeStreamPosition` is called with the response's
term/seq before the local `tx:committed` delivery runs under the mutex,
so the duplicate is detected via `txCommitted.seq <= this.latestSeq`.
Other message types (heartbeats, resets) from self are still skipped.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…riteMessage

When queryCollectionOptions detects a server-side deletion, it sends
{ type: 'delete', value: oldItem } through the sync. The persistence
layer only checked for 'key' in message to detect deletes, causing
value-based deletes to be misclassified as updates. Also use optional
chaining for process.versions in React Native where process exists but
versions may not.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…ss-window sync

Add ElectronCollectionCoordinator using BroadcastChannel + Web Locks for
leader election and cross-window coordination in Electron renderer windows.
Wire coordinator into renderer persistence via setAdapter(), add
getStreamPosition to the IPC protocol, and export from package index.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
… follow-up branches

Remove db-electron-sqlite-persisted-collection, db-node-sqlite-persisted-collection,
and db-cloudflare-do-sqlite-persisted-collection from this branch so it contains only
the core persistence packages (db, db-sqlite-persisted-collection-core,
db-browser-wa-sqlite-persisted-collection, db-react-native-sqlite-persisted-collection).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@kevin-dp kevin-dp force-pushed the kevin/persistence branch from 37c3bf5 to ca46819 Compare March 16, 2026 11:30
@kevin-dp kevin-dp merged commit d351c67 into main Mar 16, 2026
7 checks passed
@kevin-dp kevin-dp deleted the kevin/persistence branch March 16, 2026 12:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants