Skip to content

Commit 68659ba

Browse files
rcrowleyrcrowley
authored
Merge branch 'main' into update-readmes
2 parents 3c715f3 + 0ea0e0f commit 68659ba

File tree

10 files changed

+1819
-38
lines changed

10 files changed

+1819
-38
lines changed

‎README.md‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,10 @@ Use these scripts to migrate a Postgres database to PlanetScale for Postgres or
66

77
This direct migration uses logical replication and, optionally, a proxy which can manage connections and sequences for a zero-downtime migration.
88

9+
## [Heroku Postgres to PlanetScale for Postgres](./heroku-planetscale)
10+
11+
Heroku notably does not support logical replication. This strategy uses Bucardo to manage trigger-based asynchronous replication from Heroku into PlanetScale for Postgres.
12+
913
## [Postgres to PlanetScale for Postgres or Vitess via AWS DMS](./postgres-planetscale)
1014

1115
This has some speed limitations and is only recommended for databases 100GB or less.

‎heroku-planetscale/README.md‎

Lines changed: 180 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,180 @@
1+
Heroku Postgres to PlanetScale for Postgres via Bucardo asynchronous replication
2+
================================================================================
3+
4+
Heroku notably does not support logical replication, which has left many of its customers on outdated Postgres and without a convenient migration path to another provider. PlanetScale have tested a wide variety of plausible strategies and [Bucardo](https://bucardo.org/Bucardo/)'s trigger-based asynchronous replication has proven to be the most reliable option with the least downtime.
5+
6+
There may be a variation of this strategy that uses the [`ff-seq.sh`](../postgres-direct/ff-seq.sh) tool from our logical replication strategy that can provide a true zero-downtime exit strategy from Heroku. [Get in touch](mailto:[email protected]) if this is a requirement for you.
7+
8+
Setup
9+
-----
10+
11+
1. Launch an EC2 instance where you'll run Bucardo. It must run Linux and have network connectivity to both Heroku and PlanetScale.
12+
13+
2. Install and configure Bucardo there:
14+
15+
```sh
16+
sh install.sh
17+
```
18+
19+
3. Export two environment variables there:
20+
* `HEROKU`: URL-formatted Heroku Postgres connection information for the source database.
21+
* `PLANETSCALE`: Space-delimited PlanetScale for Postgres connection information for the `postgres` role (as shown on the Connect page for your database) for the target database.
22+
23+
Bulk copy and replication
24+
-------------------------
25+
26+
1. Sync table definitions outside of Bucardo (just like we'd do for logical replication and _before adding the databases to Bucardo_):
27+
28+
```sh
29+
pg_dump --no-owner --no-privileges --no-publications --no-subscriptions --schema-only "$HEROKU" | psql "$PLANETSCALE" -a
30+
```
31+
32+
2. Connect the source Heroku Postgres database:
33+
34+
```sh
35+
sudo -H -u "bucardo" bucardo add database "heroku" host="$(echo "$HEROKU" | cut -d "@" -f 2 | cut -d ":" -f 1)" user="$(echo "$HEROKU" | cut -d "/" -f 3 | cut -d ":" -f 1)" password="$(echo "$HEROKU" | cut -d ":" -f 3 | cut -d "@" -f 1)" dbname="$(echo "$HEROKU" | cut -d "/" -f 4 | cut -d "?" -f 1)"
36+
```
37+
38+
3. Connect the target PlanetScale for Postgres database:
39+
40+
```sh
41+
sudo -H -u "bucardo" bucardo add database "planetscale" ${PLANETSCALE%%" ssl"*}
42+
```
43+
44+
4. Add all sequences:
45+
46+
```sh
47+
sudo -H -u "bucardo" bucardo add all sequences --relgroup "planetscale_import"
48+
```
49+
50+
5. Add all tables:
51+
52+
```sh
53+
sudo -H -u "bucardo" bucardo add all tables --relgroup "planetscale_import"
54+
```
55+
56+
6. Create a sync:
57+
58+
```sh
59+
sudo -H -u "bucardo" bucardo add sync "planetscale_import" dbs="heroku,planetscale" onetimecopy=1 relgroup="planetscale_import"
60+
```
61+
62+
7. Start Bucardo replicating:
63+
64+
```sh
65+
sudo -H -u "bucardo" bucardo reload
66+
```
67+
68+
Monitor progress
69+
----------------
70+
71+
Bucardo status:
72+
73+
```sh
74+
sudo -H -u "bucardo" bucardo status
75+
```
76+
77+
Bucardo's state will bounce between several descriptive values. It's not possible to confirm that your replication is caught up and keeping up based on these state values alone. Instead, you need to confirm that the complete data is present (e.g. by using the `count(*)` aggregation) before moving on.
78+
79+
Count rows to gauge how caught-up the asynchronous replication is (where `example` is one of your table names):
80+
81+
```sh
82+
psql "$HEROKU" -c "SELECT count(*) FROM example;"; psql "$PLANETSCALE" -c "SELECT count(*) FROM example;"
83+
```
84+
85+
Run ad-hoc queries against the Bucardo metadata:
86+
87+
```sh
88+
sudo -H -u "bucardo" psql
89+
```
90+
91+
Tail the Bucardo logs:
92+
93+
```sh
94+
tail -F "/var/log/bucardo/log.bucardo"
95+
```
96+
97+
Switch traffic
98+
--------------
99+
100+
Because Bucardo is replicating both table and sequence data, it's critical to stop write traffic at the source completely. Most likely, this can best be accomplished at the application level. However, it is possible to enforce at the database level, too:
101+
102+
```sh
103+
psql "$HEROKU" -c "REVOKE INSERT, UPDATE, DELETE ON ALL TABLES IN SCHEMA public FROM $(echo "$HEROKU" | cut -d "/" -f 3 | cut -d ":" -f 1);"
104+
```
105+
106+
Once writes have stopped reaching Heroku, it's safe to begin writes to PlanetScale via an application deploy or reconfiguration.
107+
108+
If you issued the `REVOKE` statement above and need to abort before switching traffic and return to service on Heroku, revert as follows:
109+
110+
```sh
111+
psql "$HEROKU" -c "GRANT INSERT, UPDATE, DELETE ON ALL TABLES IN SCHEMA public TO $(echo "$HEROKU" | cut -d "/" -f 3 | cut -d ":" -f 1);"
112+
```
113+
114+
Cleanup
115+
-------
116+
117+
1. Stop and remove the Bucardo sync:
118+
119+
```sh
120+
sudo -H -u "bucardo" bucardo remove sync "planetscale_import"
121+
```
122+
123+
2. Remove every table from Bucardo's management:
124+
125+
```sh
126+
sudo -H -u "bucardo" bucardo list tables | cut -d " " -f 3 | xargs sudo -H -u "bucardo" bucardo remove table
127+
```
128+
129+
3. Remove every sequence from Bucardo's management:
130+
131+
```sh
132+
sudo -H -u "bucardo" bucardo list sequences | cut -d " " -f 2 | xargs sudo -H -u "bucardo" bucardo remove sequence
133+
```
134+
135+
4. Remove intermediate Bucardo grouping objects:
136+
137+
```sh
138+
sudo -H -u "bucardo" bucardo remove relgroup "planetscale_import" && sudo -H -u "bucardo" bucardo remove dbgroup "planetscale_import"
139+
```
140+
141+
5. Remove the PlanetScale database from Bucardo's management:
142+
143+
```sh
144+
sudo -H -u "bucardo" bucardo remove database "planetscale"
145+
```
146+
147+
6. Remove the Heroku database from Bucardo's management:
148+
149+
```sh
150+
sudo -H -u "bucardo" bucardo remove database "heroku"
151+
```
152+
153+
7. Stop Bucardo:
154+
155+
```sh
156+
sudo -H -i -u "bucardo" bucardo stop
157+
```
158+
159+
8. Remove Bucardo metadata from the Heroku database:
160+
161+
```sh
162+
psql "$HEROKU" -c "DROP SCHEMA bucardo CASCADE;"
163+
```
164+
165+
8. Optionally, terminate the EC2 instance that was hosting Bucardo.
166+
167+
9. When the migration is complete and validated, delete the source Heroku Postgres database.
168+
169+
See also
170+
--------
171+
172+
* <https://bucardo.org/Bucardo/pgbench_example>
173+
* <https://gist.github.com/Leen15/da42bd23b363867e14a378d824f2064e>
174+
* <https://smartcar.com/blog/zero-downtime-migration>
175+
* <https://medium.com/@logeshmohan/postgresql-replication-using-bucardo-5-4-1-6e78541ceb5e>
176+
* <https://justatheory.com/2013/02/bootstrap-bucardo-mulitmaster/>
177+
* <https://www.porter.run/blog/migrating-postgres-from-heroku-to-rds>
178+
* <https://medium.com/hellogetsafe/pulling-off-zero-downtime-postgresql-migrations-with-bucardo-and-terraform-1527cca5f989>
179+
* <https://github.com/nxt-insurance/bucardo-terraform-archive>
180+
* <https://bucardo-general.bucardo.narkive.com/hznUofas/replication-of-tables-without-primary-keys>

‎heroku-planetscale/install.sh‎

Lines changed: 52 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,52 @@
1+
set -e -x
2+
3+
BUCARDO_VERSION="5.6.0"
4+
5+
echo "deb http://apt.postgresql.org/pub/repos/apt $(lsb_release -c -s)-pgdg main" |
6+
sudo tee "/etc/apt/sources.list.d/pgdg.list"
7+
curl -L -S -f -s "https://www.postgresql.org/media/keys/ACCC4CF8.asc" |
8+
sudo gpg --dearmor -o "/etc/apt/trusted.gpg.d/postgresql.gpg" --yes
9+
sudo apt-get update
10+
11+
sudo apt-get -y install "libdbd-pg-perl" "libdbix-safe-perl" "libpod-parser-perl" "postgresql-17" "postgresql-plperl-17"
12+
13+
if ! which "bucardo"
14+
then
15+
if [ ! -f "$TMP/bucardo-$BUCARDO_VERSION.tar.gz" ]
16+
then curl -L -o "$TMP/bucardo-$BUCARDO_VERSION.tar.gz" "https://github.com/bucardo/bucardo/archive/$BUCARDO_VERSION.tar.gz"
17+
fi
18+
if [ ! -d "$TMP/bucardo-$BUCARDO_VERSION" ]
19+
then tar -C "$TMP" -f "$TMP/bucardo-$BUCARDO_VERSION.tar.gz" -x
20+
fi
21+
(
22+
cd "$TMP/bucardo-$BUCARDO_VERSION"
23+
perl "Makefile.PL"
24+
make
25+
sudo make install
26+
)
27+
fi
28+
29+
sudo useradd -M -U -d "/var/run/bucardo" -s "/bin/sh" "bucardo" ||
30+
sudo usermod -d "/var/run/bucardo" -s "/bin/sh" "bucardo"
31+
sudo mkdir -p "/var/log/bucardo" "/var/run/bucardo"
32+
sudo chown "bucardo:bucardo" "/var/log/bucardo" "/var/run/bucardo"
33+
34+
sudo -H -u "postgres" psql -c "CREATE ROLE bucardo WITH CREATEDB LOGIN SUPERUSER;" ||
35+
sudo -H -u "postgres" psql -c "ALTER ROLE bucardo WITH CREATEDB LOGIN SUPERUSER;"
36+
sudo -H -u "postgres" psql -c "CREATE DATABASE bucardo WITH OWNER bucardo;" ||
37+
sudo -H -u "bucardo" psql -c '\d'
38+
39+
if ! sudo -H -u "bucardo" bucardo status 2>"/dev/null"
40+
then
41+
echo "p" | sudo -H -u "bucardo" bucardo install \
42+
--db-name "bucardo" \
43+
--db-user "bucardo" \
44+
--db-host "/var/run/postgresql" \
45+
--db-port 5432 \
46+
--verbose
47+
sudo -H -i -u "bucardo" bucardo start
48+
else
49+
sudo -H -u "bucardo" bucardo upgrade --verbose || :
50+
sudo -H -i -u "bucardo" bucardo restart
51+
fi
52+
sudo -H -u "bucardo" bucardo status

‎postgres-direct/README.md‎

Lines changed: 8 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,14 @@ All of these tools provide a usage message when run without arguments.
88
Migrating data via logical replication
99
--------------------------------------
1010

11-
First, enable logical replication in your source Postgres database. In Amazon Aurora this is the `rds.logical_replication` parameter in the database cluster parameter group (and takes a while to apply, even if you apply immediately). In Neon this is a database-level setting. Alas, in Heroku this is not supported at all. Then, use the tools as follows:
11+
First, enable logical replication in your source Postgres database by setting `wal_level = logical`. Some providers expose this setting under a different name:
12+
13+
* In Amazon Aurora this is the `rds.logical_replication` parameter in the database cluster parameter group (and takes a while to apply, even if you apply immediately).
14+
* In Google CloudSQL this is the `cloudsql.enable_pglogical` setting (and note that this does _not_ require `CREATE EXTENSION pglogical;`).
15+
* In Neon this is a database-level setting.
16+
* Alas, in Heroku this is not supported at all.
17+
18+
Second, ensure there is network connectivity from the Internet to your database so that PlanetScale can reach it. In most hosts this is trivially the case. In AWS, you will need to ensure your Aurora or RDS _instance_ (not your Aurora _cluster_) allows public connectivity, its security group allows public connectivity, and its subnets' routing table(s) have a route from the Internet via an Internet Gateway.
1219

1320
`mk-logical-repl.sh` sets up logical replication between a primary (presumably elsewhere) and a replica (presumably PlanetScale for Postgres), including importing the schema.
1421

‎postgres-direct/mk-logical-repl.sh‎

100644100755
Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -42,11 +42,11 @@ then
4242
echo "primary wal_level != logical" >&2
4343
exit 1
4444
fi
45-
psql "$PRIMARY" -c "CREATE PUBLICATION _planetscale_import;"
45+
psql "$PRIMARY" -c "CREATE PUBLICATION _planetscale_import;" || :
4646
psql "$PRIMARY" -A -c '\dt' -t |
4747
cut -d "|" -f "2" |
4848
while read TABLE
49-
do psql "$PRIMARY" -c "ALTER PUBLICATION _planetscale_import ADD TABLE $TABLE;"
49+
do psql "$PRIMARY" -c "ALTER PUBLICATION _planetscale_import ADD TABLE \"$TABLE\";"
5050
done
5151

5252
# Import the primary's schema.

‎postgres-direct/mv-proxy.sh‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -58,6 +58,7 @@ psql "$PROXY" -c "
5858
BEGIN;
5959
ALTER SERVER _planetscale_import OPTIONS (SET host '$SERVER_HOSTNAME', SET port '$SERVER_PORT', SET dbname '$SERVER_DATABASE');
6060
ALTER USER MAPPING FOR $PG_USERNAME SERVER _planetscale_import OPTIONS (SET user '$SERVER_USERNAME', SET password '$SERVER_PASSWORD');
61+
SELECT postgres_fdw_disconnect('_planetscale_import');
6162
COMMIT;
6263
"
6364

‎postgres-direct/stat-logical-repl.sh‎

Lines changed: 65 additions & 28 deletions
Original file line numberDiff line numberDiff line change
@@ -30,38 +30,75 @@ fi
3030

3131
export PSQL_PAGER=""
3232

33-
set -x
34-
3533
# Inspect the schema on the primary and replica.
36-
psql "$PRIMARY" -c '\d'
37-
psql "$REPLICA" -c '\d'
38-
39-
# Inspect all Postgres' internal replication status information.
40-
psql "$PRIMARY" -c "SELECT * FROM pg_stat_replication;" -x
41-
psql "$PRIMARY" -c "SELECT * FROM pg_replication_slots;" -x
42-
psql "$PRIMARY" -c "SELECT slot_name, slot_type, active, pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn) FROM pg_replication_slots;"
43-
psql "$REPLICA" -c "SELECT * FROM pg_stat_wal_receiver;" -x || : # not supported in Aurora 13
44-
psql "$REPLICA" -c "SELECT * FROM pg_catalog.pg_stat_subscription;" -x
34+
echo >&2
35+
echo "##############################" >&2
36+
echo "# PRIMARY AND REPLICA SCHEMA #" >&2
37+
echo "##############################" >&2
38+
echo >&2
39+
(
40+
set -x
41+
psql "$PRIMARY" -c '\d'
42+
psql "$REPLICA" -c '\d'
43+
)
44+
echo >&2
4545

46-
# Do a sloppy distributed transaction to figure out how far behind we are.
47-
psql "$PRIMARY" -c "SELECT pg_current_wal_lsn();"
48-
psql "$REPLICA" -c "SELECT received_lsn FROM pg_stat_subscription WHERE subname = '_planetscale_import';"
49-
PRIMARY_LSN="$(psql "$PRIMARY" -A -c "SELECT pg_wal_lsn_diff(pg_current_wal_lsn(), '0/0');" -t)"
50-
REPLICA_LSN="$(psql "$REPLICA" -A -c "SELECT pg_wal_lsn_diff(received_lsn, '0/0') FROM pg_stat_subscription WHERE subname = '_planetscale_import';" -t)"
51-
LAG="$((PRIMARY_LSN - REPLICA_LSN))" # bytes behind
52-
set +x
53-
if [ "$LAG" -lt "1024" ]
54-
then printf "replication is caught up"
55-
else printf "replication is behind"
56-
fi
57-
echo "; lag: $LAG, primary LSN: $PRIMARY_LSN, replica LSN: $REPLICA_LSN"
58-
set -x
46+
# Inspect Postgres' internal logical replication status information.
47+
echo >&2
48+
echo "######################" >&2
49+
echo "# REPLICATION STATUS #" >&2
50+
echo "######################" >&2
51+
echo >&2
52+
(
53+
set -x
54+
psql "$PRIMARY" -a -x <<EOF
55+
SELECT * FROM pg_stat_replication WHERE application_name = '_planetscale_import';
56+
SELECT
57+
active, inactive_since, invalidation_reason, wal_status,
58+
confirmed_flush_lsn, pg_wal_lsn_diff(pg_current_wal_lsn(), confirmed_flush_lsn)
59+
FROM pg_replication_slots WHERE slot_name = '_planetscale_import';
60+
EOF
61+
psql "$REPLICA" -a -x <<EOF
62+
SELECT * FROM pg_catalog.pg_stat_subscription WHERE subname = '_planetscale_import';
63+
\x
64+
SELECT
65+
n.nspname,
66+
c.relname,
67+
sr.srsubstate,
68+
CASE
69+
WHEN sr.srsubstate = 'i' THEN 'initializing'
70+
WHEN sr.srsubstate = 'd' THEN 'data is being copied'
71+
WHEN sr.srsubstate = 'f' THEN 'finished table copy'
72+
WHEN sr.srsubstate = 's' THEN 'synchronized'
73+
WHEN sr.srsubstate = 'r' THEN 'ready (normal replication)'
74+
ELSE ''
75+
END AS srsubstate_explain,
76+
sr.srsublsn
77+
FROM pg_subscription s
78+
JOIN pg_subscription_rel sr ON s.oid = sr.srsubid
79+
JOIN pg_class c ON c.oid = sr.srrelid
80+
JOIN pg_namespace n ON c.relnamespace = n.oid
81+
WHERE s.subname = '_planetscale_import';
82+
EOF
83+
)
84+
echo >&2
5985

6086
# Send a sentinel write through the logical replication stream.
61-
TS="$(date +"%s")"
62-
psql "$PRIMARY" -c "INSERT INTO _planetscale_import VALUES ($TS, 'testing');"
63-
sleep 1
64-
psql "$REPLICA" -c "SELECT * FROM _planetscale_import WHERE ts >= $TS;"
87+
echo >&2
88+
echo "###############" >&2
89+
echo "# TEST RECORD #" >&2
90+
echo "###############" >&2
91+
echo >&2
92+
(
93+
set -x
94+
TS="$(date +"%s")"
95+
psql "$PRIMARY" -c "INSERT INTO _planetscale_import VALUES ($TS, 'testing');"
96+
sleep 1
97+
psql "$REPLICA" -c "SELECT * FROM _planetscale_import WHERE ts >= $TS;"
98+
)
99+
echo >&2
65100

101+
# Uncomment for ad-hoc psql shells.
102+
echo >&2
66103
#psql "$PRIMARY"
67104
#psql "$REPLICA"

0 commit comments

Comments
 (0)