would dwarfs be suitable for storing and loading ML model files?

[unphased]

I collect a lot of different LLM model files and image generation diffusion model files (they are usually in safetensors or gguf type formats) and I happen to know that a lot of these are just model merges. basically if you have different diffusion models that are independently finetuned, if they are compatible architecture you could do a weighted average between them and arrive at an infinite number of intermediate models that incorporate both. It's a nice poor man's way of having fine tuning done by completely independent teams and gain some or most of the benefits of that training.

The idea is i might have 50 model checkpoints but if a large amount of the information contained in them is shared by a few common ancestor (as well as the fact that every single one of these 50 checkpoints share the same model architecture underneath), compression may be able to extract some or a lot of that information. Yes a compression system (probably based on loras or some similar concept) should be much more space efficient but the beauty of something like dwarfs is that there is often little overhead to having the data compressed in an archive at rest and loaded on the fly.

Has anyone done this with good results? my main concern would be that these files are not going to contain repetition of the type that the compression approach can "pick up".

[mhx]

my main concern would be that these files are not going to contain repetition of the type that the compression approach can "pick up".

As a rule of thumb, if there's redundancy on the byte-level, it can probably be picked up by DwarFS in one way or another. If the redundancy only manifests with internal knowledge of the format, then very likely not. DwarFS already contains "specialized" algorithms for two non-generic use cases: PCM audio compression (using FLAC) and RAW image compression (using a custom algorithm). In both cases, DwarFS' categorizer determines to which data a particular algorithm is applicable and extracts the necessary metadata to drive the algorithm.

If there is byte-level redundancy, there are two stages upstream of the actual block compression that can help improve compression performance:

• similarity ordering: in the case of many (small-ish) files, these are ordered such that "similar" files (mostly) end up near each other in the stream that enters the downstream stages
• segmentation: this looks for long (typically a few KiB) sequences that have been seen previously and stores them as a reference instead of passing them downstream; this has the advantage of speeding up compression and requiring (potentially much) less memory when decompressing data

Has anyone done this with good results?

Have you tried it? :)