Skip to content

Add zstd file compression support #93

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 9 commits into
base: main
Choose a base branch
from
Open

Conversation

draguve
Copy link
Contributor

@draguve draguve commented Jul 8, 2025

This pull request adds support for compressing files before sending them using Zstandard. Compression is enabled with the -z flag, and the compression quality can be configured via the -q flag.

The feature is gated behind the zstd feature flag. Removing this feature disables compression support entirely.

This change introduces a new dependency on the async-compression crate.

If a user attempts to send compressed files to a build that does not support zstd decompression, the system will add a .zst extension to each file and display the following warning:

Warning: This build does not support zstd decompression. Files will be saved with a `.zst` extension. 
You can manually decompress them using `unzstd <filename>`.

@n0bot n0bot bot added this to iroh Jul 8, 2025
@github-project-automation github-project-automation bot moved this to 🏗 In progress in iroh Jul 8, 2025
@rklaehn
Copy link
Collaborator

rklaehn commented Jul 14, 2025

Thanks for the PR. So this compresses each file individually and still uses the iroh-blobs protocol as is. For lots of small files you would get better results by compressing the entire transfer stream, but that would have to be a different ALPN and would require changes in blobs itself I suppose.

Zstd is also a very good choice. I love both zstd and lz4 from Yann Collet.

One thing that is a bit unfortunate is that zstd adds a C dependency. LZ4 would obviously have not nearly as good compression.

I will take a look some time this week.

@draguve
Copy link
Contributor Author

draguve commented Jul 14, 2025

I was also considering the case of sending many small files. One option is to tar the directory before sending, but that approach doesn’t allow for parallelization. Ideally, we’d handle it the way you described, though that would require changes to the blob's logic.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: 🏗 In progress
Development

Successfully merging this pull request may close these issues.

2 participants