Skip to content

Commit 994afb5

Browse files
authored
Add Postgres bucket storage option to self-hosting config (#80)
* Add Postgres bucket storage option * Simplify skeleton config * Improve wording * Improve wording * Update references to MongoDB storage * Fix some missing MySQL references * Update self-hosting/installation/powersync-service-setup.mdx * Wording improvements * Beta label and postgres server
1 parent c285c73 commit 994afb5

File tree

9 files changed

+69
-23
lines changed

9 files changed

+69
-23
lines changed

installation/app-backend-setup.mdx

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,7 @@ When using PowerSync, your app backend is responsible for the following:
77

88
1. Authenticating app users
99
2. [Generating JWTs](/installation/authentication-setup) that allows the PowerSync Client SDK to authenticate users against the server-side [PowerSync Service](/architecture/powersync-service).
10-
3. [Writing client-side changes](/installation/app-backend-setup/writing-client-changes) to the backend database (Postgres or MongoDB)
10+
3. [Writing client-side changes](/installation/app-backend-setup/writing-client-changes) to the backend database (Postgres, MongoDB or MySQL)
1111

1212
<Frame caption="An overview of how PowerSync interacts with your backend application.">
1313
<img src="/images/powersync-docs-app-backend-setup-diagram (2).png"/>

installation/app-backend-setup/writing-client-changes.mdx

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -20,7 +20,7 @@ In other words, don't place writes into something like a queue for processing la
2020
</Warning>
2121

2222
<Accordion title="Why must my write endpoint be synchronous?">
23-
Client applications advance to a write checkpoint after uploads have been processed, so if the client believes that the server has written changes into your backend database (Postgres or MongoDB), but the next checkpoint does not contain your uploaded changes, those changes will be removed from the client. This could manifest as UI glitches for your end-users, where the changes disappear from the device for a few seconds and then re-appear.
23+
Client applications advance to a write checkpoint after uploads have been processed, so if the client believes that the server has written changes into your backend database (Postgres, MongoDB or MySQL), but the next checkpoint does not contain your uploaded changes, those changes will be removed from the client. This could manifest as UI glitches for your end-users, where the changes disappear from the device for a few seconds and then re-appear.
2424
</Accordion>
2525

2626
### Changes recorded on the client

installation/client-side-setup/integrating-with-your-backend.mdx

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,7 @@ description: "The PowerSync 'backend connector' provides the connection between
77
It is used to:
88

99
1. Retrieve a JWT token which can be used by the PowerSync Client SDK to authenticate against your [PowerSync Service](/architecture/powersync-service) instance.
10-
2. Upload writes to your backend: Writes that are made to the local SQLite database are sent to your backend application, where you control how they're applied to your backend database (Postgres or MongoDB)
10+
2. Upload writes to your backend: Writes that are made to the local SQLite database are sent to your backend application, where you control how they're applied to your backend database (Postgres, MongoDB or MySQL)
1111

1212
Accordingly, the connector must implement two methods:
1313

resources/demo-apps-example-projects.mdx

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -94,10 +94,11 @@ Example projects are listed under backend they use, but you can easily wire up y
9494

9595
<Accordion title="Self-Hosting" icon="desktop">
9696
* [To-Do List App with Docker Compose](https://github.com/powersync-ja/self-host-demo)
97-
* [Postgres + Node.js Backend](https://github.com/powersync-ja/self-host-demo/tree/main/demos/nodejs)
97+
* [Postgres + Node.js Backend](https://github.com/powersync-ja/self-host-demo/tree/main/demos/nodejs-postgres-bucket-storage)
98+
* [Postgres + Postgres sync bucket storage + Node.js Backend](https://github.com/powersync-ja/self-host-demo/tree/main/demos/nodejs)
9899
* [MongoDB + Node.js Backend](https://github.com/powersync-ja/self-host-demo/tree/main/demos/nodejs-mongodb)
99100
* [MySQL + Node.js Backend](https://github.com/powersync-ja/self-host-demo/tree/main/demos/nodejs-mysql)
100-
* [Supabase Backend + Local Development Example](https://github.com/powersync-ja/self-host-demo/tree/main/demos/supabase)
101+
* [Supabase Backend (Postgres) + Local Development Example](https://github.com/powersync-ja/self-host-demo/tree/main/demos/supabase)
101102
* [Django Backend](https://github.com/powersync-ja/self-host-demo/tree/main/demos/django)
102103
</Accordion>
103104

self-hosting/installation.mdx

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -29,7 +29,7 @@ In order to run the PowerSync Service, the following activities are required:
2929
<CardGroup>
3030
<Card title="Configure source database" icon="database" href="/installation/database-setup"horizontal />
3131
<Card title="Configure PowerSync Service" icon="gears" href="/self-hosting/installation/powersync-service-setup"horizontal />
32-
<Card title="Configure MongoDB" icon="database" href="/self-hosting/installation/powersync-service-setup#configure-mongodb" horizontal/>
32+
<Card title="Configure sync bucket storage" icon="database" href="/self-hosting/installation/powersync-service-setup#configure-sync-bucket-storage" horizontal/>
3333
<Card title="Connect PowerSync to database" icon="plug" href="/self-hosting/installation/powersync-service-setup#database-connection" horizontal/>
3434
<Card title="Define sync rules" icon="list-check" href="/self-hosting/installation/powersync-service-setup#sync-rules" horizontal/>
3535
<Card title="Import PowerSync client SDK" icon="file-import" href="/installation/client-side-setup"horizontal />

self-hosting/installation/powersync-service-setup.mdx

Lines changed: 55 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -3,15 +3,15 @@ title: "PowerSync Service Setup"
33
description: "Configuration details for connecting the PowerSync Service to your database"
44
---
55

6-
After configuring your Postgres database for PowerSync, you'll setup your [PowerSync Service](/architecture/powersync-service).
6+
After configuring your source database for PowerSync, you'll need to setup your [PowerSync Service](/architecture/powersync-service).
77

88
This entails:
99

10-
1. Configuring MongoDB (if required)
10+
1. Configuring sync bucket storage
1111

1212
2. Defining your PowerSync config
1313

14-
1. Defining connections to Postgres and MongoDB
14+
1. Defining connections to your source database, and sync bucket storage database
1515

1616
2. Defining your [Sync Rules](/usage/sync-rules)
1717

@@ -23,9 +23,13 @@ Examples of the above can be found in our demo application [here](https://github
2323
**Deploy PowerSync on Coolify:** See our [integration guide](/integration-guides/coolify) for deploying the PowerSync Service on Coolify. This can simplify the setup and management of the deployment.
2424
</Tip>
2525

26-
## Configure MongoDB
26+
## Configure Sync Bucket Storage
2727

28-
The PowerSync Service uses [MongoDB](https://www.mongodb.com/) under the hood and requires at least one replica set node. A single node is fine for development/staging environments, but a 3-node replicas set is recommended [for production](/self-hosting/lifecycle-maintenance/server-specs).
28+
The PowerSync Service requires a storage backend for sync buckets. You can use either MongoDB or Postgres for this purpose. The storage backend is separate from your source database.
29+
30+
### MongoDB Storage
31+
32+
MongoDB requires at least one replica set node. A single node is fine for development/staging environments, but a 3-node replica set is recommended [for production](/self-hosting/lifecycle-maintenance/server-specs).
2933

3034
[MongoDB Atlas](https://www.mongodb.com/products/platform/atlas-database) enables replica sets by default for new clusters.
3135

@@ -50,6 +54,36 @@ If you are rolling your own Docker environment, you can include this init script
5054
- 'sleep 10 && mongosh --host mongo:27017 --eval ''try{rs.status().ok && quit(0)} catch {} rs.initiate({_id: "rs0", version: 1, members: [{ _id: 0, host : "mongo:27017" }]})'''
5155
```
5256
57+
### Postgres Storage (Beta)
58+
59+
Available since version 1.3.8 of [`journeyapps/powersync-service`](https://hub.docker.com/r/journeyapps/powersync-service), you can use Postgres as an alternative storage backend for sync buckets. This feature is currently in [beta](/resources/feature-status).
60+
61+
#### Database Setup
62+
63+
You'll need to create a dedicated user and schema for PowerSync storage. You can either:
64+
65+
1. Let PowerSync create the schema (recommended):
66+
67+
```sql
68+
CREATE USER powersync_storage_user WITH PASSWORD 'secure_password';
69+
-- The user should only have access to the schema it created
70+
GRANT CREATE ON DATABASE postgres TO powersync_storage_user;
71+
```
72+
73+
2. Or manually create the schema:
74+
75+
```sql
76+
CREATE USER powersync_storage_user WITH PASSWORD 'secure_password';
77+
CREATE SCHEMA IF NOT EXISTS powersync AUTHORIZATION powersync_storage_user;
78+
GRANT CONNECT ON DATABASE postgres TO powersync_storage_user;
79+
GRANT USAGE ON SCHEMA powersync TO powersync_storage_user;
80+
GRANT ALL PRIVILEGES ON ALL TABLES IN SCHEMA powersync TO powersync_storage_user;
81+
```
82+
83+
<Tip>
84+
**Demo app:** A demo app with Postgres bucket storage is available [here](https://github.com/powersync-ja/self-host-demo/tree/main/demos/nodejs-postgres-bucket-storage).
85+
</Tip>
86+
5387
## PowerSync Configuration
5488

5589
The PowerSync Service is configured using key/value pairs in a config file, and supports the following configuration methods:
@@ -70,7 +104,7 @@ The config file schema is also available here:
70104

71105
<Card title="self-host-demo/schema/schema.json at main · powersync-ja/self-host-demoGitHub" icon="github" href="https://github.com/powersync-ja/self-host-demo/blob/main/schema/schema.json" horizontal />
72106

73-
Below is a skeleton config file that you can copy/paste and edit locally:
107+
Below is a skeleton config file you can copy and paste to edit locally:
74108

75109
```yaml
76110
# config.yaml
@@ -90,16 +124,23 @@ replication:
90124
91125
# SSL settings
92126
sslmode: disable # 'verify-full' (default) or 'verify-ca' or 'disable'
93-
# 'disable' is OK for local/private networks, not for public networks
127+
# Note: 'disable' is only suitable for local/private networks, not for public networks
94128
95-
# Connection settings for sync bucket storage
129+
# Connection settings for sync bucket storage (MongoDB and Postgres are supported)
96130
storage:
131+
# Option 1: MongoDB Storage
97132
type: mongodb
98133
uri: mongodb://mongo:27017/powersync_demo
99-
# use these if authentication is required
134+
# Use these if authentication is required. The user should have `readWrite` and `dbAdmin` roles
100135
# username: myuser
101136
# password: mypassword
102137

138+
# Option 2: Postgres Storage
139+
# type: postgresql
140+
# This accepts the same parameters as a Postgres replication source connection
141+
# uri: postgresql://powersync_storage_user:secure_password@storage-db:5432/postgres
142+
# sslmode: disable
143+
103144
# The port which the PowerSync API server will listen on
104145
port: 80
105146

@@ -132,6 +173,10 @@ client_auth:
132173

133174
```
134175

176+
<Info>
177+
**Important:** When using Postgres for sync bucket storage, a separate server is currently required for replication connections (if using Postgres for replication) and storage. Using the same server might cause unexpected results.
178+
</Info>
179+
135180
Specify the connection to Postgres in the `replication` section. Retrieving your database connection string / individual parameters differs by database hosting provider. See [Database Connection](/self-hosting/appendix/database-connection) for further details.
136181

137182
<Info>
@@ -142,7 +187,7 @@ Specify the connection to Postgres in the `replication` section. Retrieving your
142187
This is because Supabase only allows direct database connections over IPv6 — PowerSync cannot connect using the connection pooler.
143188
</Info>
144189

145-
Specify the connection to MongoDB in the `storage` section.
190+
Specify the connection to your sync bucket storage provider (Postgres or MongoDB) in the `storage` section.
146191

147192
### Environment Variables
148193

self-hosting/lifecycle-maintenance.mdx

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -12,14 +12,14 @@ Migrations run automatically by default.
1212
# powersync.yaml
1313

1414
migrations:
15-
# Setting this to true will skip automatic migrations.
16-
# Migrations can be triggered externally by altering the container `command`.
15+
# Setting this to false (default) enables automatic migrations on startup.
16+
# When set to true, migrations must be triggered manually by modifying the container `command`.
1717
disable_auto_migration: true
1818
```
19-
MongoDB locks are used to ensure that migrations are only executed once in the case where multiple containers can be started simultaneously.
19+
MongoDB locks ensure migrations are executed exactly once, even when multiple containers start simultaneously.
2020
2121
## Backups
2222
23-
We recommend using git to backup your configuration files.
23+
We recommend using Git to backup your configuration files.
2424
25-
Your MongoDB database (for sync bucket storage) does not need to be backed up since it's easily recreated from Postgres.
25+
The sync bucket storage database doesn't require backups as it can be easily reconstructed.

usage/sync-rules.mdx

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -15,7 +15,7 @@ The remainder of these docs dive further into the details.
1515

1616
Each [PowerSync Service](/architecture/powersync-service) instance has a Sync Rules configuration where Sync Rules are defined using SQL-like queries (limitations and more info [here](/usage/sync-rules/operators-and-functions)) combined together in a YAML file.
1717

18-
This SQL-like syntax is used when connecting to either Postgres or MongoDB as the backend source database.
18+
This SQL-like syntax is used when connecting to either Postgres, MongoDB or MySQL as the backend source database.
1919

2020
The [PowerSync Service](/architecture/powersync-service) uses these SQL-like queries to group data into "sync buckets" when replicating data to client devices.
2121

usage/sync-rules/advanced-topics/sharded-databases.mdx

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,7 @@ In the case of Postgres, PowerSync cannot replicate Postgres [foreign tables](ht
88
However, PowerSync does have options available to support sharded databases in general.
99

1010
<Info>
11-
When using MongoDB as the backend source database, PowerSync does not currently support connecting to sharded clusters.
11+
When using MongoDB or MySQL as the backend source database, PowerSync does not currently support connecting to sharded clusters.
1212
</Info>
1313

1414
The primary options are:

0 commit comments

Comments
 (0)