[POC] Compression for sync streams

[rkistner]

Background on compression options we're investigating in PowerSync: #330

This enables gzip and zstd as transport compression options for sync streams. Since data sync is often performed on metered mobile connections or slow WiFi connections, this could reduce data costs and increase performance for those cases.

This uses the standard Accept-Encoding request header, and Content-Encoding response header. As an example, Chrome would send accept-encoding: gzip, deflate, br, zstd, then we'd response with content-encoding: zstd.

zstd is the best option, and is supported by default in Chrome 123+ and Firefox 126+. gzip is provided as a fallback option, and is supported in Safari and many native clients.

Since browsers send the accept-encoding header by default, this will have an immediate effect for the Web SDK. For other SDKs we'd have to test individually, and see whether we need more client-side changes to enable compression.

Performance

These are not scientific benchmarks, but just gives an initial idea of expected performance. Since decompression is performed by platform-provided native code in each SDK, this adds minimal overhead and minimal client-side changes (we're not manually decompressing using JavaScript, for example).

Synthetic test dataset: 1.6 million rows of data, 384MiB uncompressed download size, 117MiB data size (the rest is metadata).

zstd compression gives 72.4MiB download size (19% of the original size); gzip gives 76.2MiB (20% of the original size).

In all cases (uncompressed, zstd, gzip), it took around 13s to download the data with curl, and 52s with the diagnostics app. So on a desktop machine, the primary difference is in bandwidth usage, rather than performance.

When downloading the data with curl, the nodejs process averaged around 90% CPU usage for uncompressed data, and 120% when using zstd or gzip. Note that it uses more than 1 core since compression uses a separate thread.

Native clients

OkHttp supports gzip by default - likely affects both React-Native and Kotlin.

NodeJS SDK:

• Will use gzip out of the box.
• We can make minor SDK modifications to use zstd on Node versions that support it

TODO

• Test for issues and performance on each SDK
• Proper benchmarks
• Test permessage-deflate for websockets

[changeset-bot]

⚠️ No Changeset found

Latest commit: da3da38

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR