Skip to content

Conversation

@gazi-yestemirova
Copy link
Contributor

What changed?

  • Update Shard-distributor Store config to accept Compression field, to set the compression type. Defaults to none
  • Add Compress() method to encode data with snappy compression
  • Compress executor state, reported shards and assigned state before writing to etcd
  • Extend unit-tests with compression logic check

Why?
To reducing storage footprint

How did you test it?
unit-tests and local testing

Potential risks
N/A

Release notes

Documentation Changes

@gazi-yestemirova gazi-yestemirova changed the title [shard-distributor]Compress data before writing to ETCD feat: [shard-distributor]Compress data before writing to ETCD Nov 4, 2025

// Compress encodes data using the compression library identified by compressionType
// When compressionType is empty or "none", the data is returned as-is
func Compress(data []byte, compressionType string) ([]byte, error) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This way, with explicit compressionType it is required to specify it every time.
I think it's better to not focus on which algorithm we use (for the exported function) since it's not that we're building a generic compressor - we're building serializer of data for etcd. We can decide internally which algorithm we use, the only thing we need to know is to whatever we need to compress the data.

I would suggest something like RecordWriter which initializes with compress or not (from the config?) and then does that on .Write() or something. But .Write internally already knows it uses "snappy" or whatever.
Otherwise you need to do tricks like

switch strings.ToLower(compressionType) {
	case "", "none":
```.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants