Integrate with sqlite-zstd

[franchb]

Is it possible to add sqlite-zstd extension to ncruces/go-sqlite3 WASM driver?

[ncruces]

Maybe, but it really depends on what you mean.

Any extension you can compile and statically link into your own Wasm, is probably fair game.

I can't give specific instructions for sqlite-zstd, but you can check how sqlite3.wasm is built, and how various extensions are linked and initialized. Then it's just a matter of using your own Wasm.

If, however, want you really want is readonly compressed databases, my suggestion is that you compress your entire database with Zstandard Seekable Format, and use zstd-seekable-format-go with the reader VFS.

[franchb]

Thank you! I'll give zstd-seekable-format-go a shot.

[ncruces]

Please report back!

[muety]

[...] compress your entire database with Zstandard Seekable Format, and use zstd-seekable-format-go with the reader VFS.

Just for my understanding, this solution would only be suitable for reading an already compressed database, but wouldn't support to read and write to the database from Go as usual, with compression happening transparently in the background?

Did you give it a try nevertheless, @franchb?

Would love to have a compression VFS implementation!

Apart from that, thanks a lot for your efforts, @ncruces!

[ncruces]

Yes, zstd-seekable-format-go would be strictly read-only. It's hard to envision something else TBH.

The architecture for the official ZIPVFS depends on non public APIs.

As detailed on that page, the SQLite pager depends on writing fixed size power-of-two pages to power-of-two offsets. I was able to circumvent the issue for my encryption extensions, by using encryption schemes that do not change the size of pages/sectors.

Changing the size of pages is kinda the point of compression. More importantly, a page can be really small when first written to (bunch of zeros for an half-filled page) then updated to be incompressible. We can't just make something in the middle of a file grow.

The way the ZIPVFS works is kinda like there's a database “in memory” that saves compressed pages as BLOBs to a second database on disk, just more optimized. But we have no special APIs (other than SQL) to make this fast. I wonder, at that point why not use some KV store.

Anyway, this is more work than I can take in. But hopefully making this a discussion gives others ideas to work on.