Blue sky/future: investigate alternative, high perf deflate implementions #9
Description
I presume our wrapper around zlib is as fast as possible, and faster than the C++ wrapper boost provides (refs #7).
But in the past I've seen zlib compression be a meaningful % of the time taken to work with vector tiles. Often the usecase is:
- Get a tile over a network or a db
- Decompress it
- Do an operation on it (query, shave, composite, etc)
- Recompress
- Send back over a network or put into db
When the "do an operation" is fairly speedy, the decompress and recompress times are often meaningful (at least 5-10% of the time taken on the CPU). In applications handling a lot of concurrent requests where the CPU may be working hard, we can increase the amount of concurrency possible on a single machine by reducing the amount of work on the CPU.
So, this is a long way of saying:
- gzip is fairly cheap and likely already as fast as it can be
- except in rare cases of high load with fairly optimized code (meaning where the vector tile operation is already pretty fast so the gzip part stands out more)...
- In that case we might want to revisit trying to speed up gzip coding operations
When/if we do, then we should look into benchmarking https://github.com/ebiggers/libdeflate, which claims to be faster than zlib.
//cc @GretaCB @flippmoke