Skip to content

Commit 5a44d11

Browse files
klauspostklauspost
authored
Add faster compression mode for speed extremely sensitive applications (#30)
1 parent de1ccac commit 5a44d11

18 files changed

Lines changed: 10677 additions & 4101 deletions

‎README.md‎

Lines changed: 67 additions & 33 deletions
Original file line numberDiff line numberDiff line change
@@ -7,14 +7,15 @@ where encoding and/or decoding speed is the primary concern.
77

88
MinLZ is designed to operate *faster than IO* for both compression and decompression and be a viable "always on"
99
option even if some content already is compressed.
10-
If slow compression is acceptable, MinLZ can be configured to produce high compression ratio,
11-
but retain high decompression speed.
10+
If slow compression is acceptable, MinLZ can be configured to produce a high compression ratio,
11+
while retaining high decompression speed.
1212

1313
* Best in class compression
1414
* Block or Streaming interfaces
1515
* Very fast decompression, even as pure Go
1616
* AMD64 encoder+decoder assembly
17-
* Adjustable Compression (3 levels)
17+
* ARM64 decoder assembly
18+
* Adjustable Compression (4 levels)
1819
* Concurrent stream Compression
1920
* Concurrent stream Decompression
2021
* Skip forward in compressed stream via independent blocks
@@ -32,6 +33,11 @@ This package implements the MinLZ specification v1.0 in Go.
3233
For format specification see the included [SPEC.md](SPEC.md).
3334

3435
# Changelog
36+
* [v1.1.0](https://github.com/minio/minlz/releases/tag/v1.1.0)
37+
* Added SuperFast compression mode. See LevelSuperFast below.
38+
* Added ARM64 decompression assembly.
39+
* Added `-follow` to mz command to allow reading files while they are being written.
40+
* Improve default compression.
3541

3642
* [v1.0.0](https://github.com/minio/minlz/releases/tag/v1.0.0)
3743
* [Initial Release Blog Post](https://blog.min.io/minlz-compression-algorithm/).
@@ -49,8 +55,9 @@ Blocks are mainly useful for small data sizes.
4955
Streams are a collection of independent blocks, which each have checksums and EOF checks,
5056
which ensures against corruption and truncation.
5157

52-
3 compression levels are provided:
58+
4 compression levels are provided:
5359

60+
* Level -1, "SuperFast": Provides the fastest compression, but at a very reduced compression ratio.
5461
* Level 1, "Fastest": Provides the fastest compression with reasonable compression.
5562
* Level 2, "Balanced": Provides a good balance between compression and speed. ~50% the speed of the fastest level.
5663
* Level 3, "Smallest": Provides the smallest output possible. Not tuned for speed.
@@ -157,6 +164,24 @@ Setting level 0 will disable compression and write the data as an uncompressed s
157164

158165
The default level is `LevelBalanced`.
159166

167+
Typical speeds are `LevelFastest` is 2x the speed of `LevelBalanced`
168+
and `LevelSmallest` is at least an order of magnitude slower.
169+
170+
### LevelSuperFast
171+
172+
Furthermore `LevelSuperFast` is provided. This compression mode is aimed purely at reducing
173+
slowdown when compressing hard-to-compress data. In practice that is short matches.
174+
175+
The compression ratio can greatly suffer in this mode, but in these cases it can be faster.
176+
So if you have a very high throughput (> 1GB/core/s) and a time-sensitive use case,
177+
this can be used to ensure the compression doesn't take too much longer on this content.
178+
179+
For these cases the speed difference can be *up to* 2x over `LevelFastest`, but also with a
180+
much worse compression ratio.
181+
However, most often it will be around 15% faster with a similar drop in the compression ratio.
182+
183+
Typically, the performance difference will also be the same for decompression.
184+
160185
#### Writer Block Size
161186

162187
The `WriterBlockSize` allows to set the maximum size of each block on the stream encoder.
@@ -344,20 +369,22 @@ Click below to see some sample benchmarks compared to Snappy and LZ4:
344369
### Protobuf Sample
345370

346371

347-
| Compressor | Size | Comp MB/s | Decomp MB/s | Reduction % |
348-
|--------------|--------|----------:|-------------|-------------|
349-
| MinLZ 1 | 17,613 | 27,837 | 116,762 | 85.15% |
350-
| MinLZ 1 (Go) | 17,479 | 22,036 | 61,652 | 85.26% |
351-
| MinLZ 2 | 16,345 | 12,797 | 103,100 | 86.22% |
352-
| MinLZ 2 (Go) | 16,345 | 9,732 | 52,964 | 86.22% |
353-
| MinLZ 3 | 14,766 | 210 | 126,385 | 87.55% |
354-
| MinLZ 3 (Go) | 14,766 | | 68,411 | 87.55% |
355-
| Snappy | 23,335 | 24,052 | 61,002 | 80.32% |
356-
| Snappy (Go) | 23,335 | 10,055 | 35,699 | 80.32% |
357-
| LZ4 0 | 18,766 | 12,649 | 137,553 | 84.18% |
358-
| LZ4 0 (Go) | 18,766 | | 64,092 | 84.18% |
359-
| LZ4 9 | 15,844 | 12,649 | 139,801 | 86.64% |
360-
| LZ4 9 (Go) | 15,844 | | 66,904 | 86.64% |
372+
| Compressor | Size | Comp MB/s | Decomp MB/s | Reduction % |
373+
|---------------|---------|----------:|-------------|-------------|
374+
| MinLZ 1 | 17,613 | 27,837 | 116,762 | 85.15% |
375+
| MinLZ 1 (Go) | 17,479 | 22,036 | 61,652 | 85.26% |
376+
| MinLZ 2 | 16,345 | 12,797 | 103,100 | 86.22% |
377+
| MinLZ 2 (Go) | 16,345 | 9,732 | 52,964 | 86.22% |
378+
| MinLZ 3 | 14,766 | 210 | 126,385 | 87.55% |
379+
| MinLZ 3 (Go) | 14,766 | | 68,411 | 87.55% |
380+
| Snappy | 23,335 | 24,052 | 61,002 | 80.32% |
381+
| Snappy (Go) | 23,335 | 10,055 | 35,699 | 80.32% |
382+
| LZ4 0 | 18,766 | 12,649 | 137,553 | 84.18% |
383+
| LZ4 0 (Go) | 18,766 | | 64,092 | 84.18% |
384+
| LZ4 9 | 15,844 | 12,649 | 139,801 | 86.64% |
385+
| LZ4 9 (Go) | 15,844 | | 66,904 | 86.64% |
386+
| MinLZ -1 | 19,889 | ------ | ------- | 83.23% |
387+
| MinLZ -1 (Go) | 19,218 | ------ | ------- | 83.74% |
361388

362389
![Compression vs Size](img/pb-block.png)
363390

@@ -369,20 +396,22 @@ Source file: https://github.com/google/snappy/blob/main/testdata/geo.protodata
369396
<details>
370397
<summary>Click To See Data + Charts (102,400 bytes input)</summary>
371398

372-
| Compressor | Size | Comp MB/s | Decomp MB/s | Reduction % |
373-
|--------------|--------|----------:|-------------|-------------|
374-
| MinLZ 1 | 20,184 | 17,558 | 82,292 | 80.29% |
375-
| MinLZ 1 (Go) | 19,849 | 15,035 | 32,327 | 80.62% |
376-
| MinLZ 2 | 17,831 | 9,260 | 58,432 | 82.59% |
377-
| MinLZ 2 (Go) | 17,831 | 7,524 | 25,728 | 82.59% |
378-
| MinLZ 3 | 16,025 | 180 | 80,445 | 84.35% |
379-
| MinLZ 3 (Go) | 16,025 | | 33,382 | 84.35% |
380-
| Snappy | 22,843 | 17,469 | 44,765 | 77.69% |
381-
| Snappy (Go) | 22,843 | 8,161 | 21,082 | 77.69% |
382-
| LZ4 0 | 21,216 | 9,452 | 101,490 | 79.28% |
383-
| LZ4 0 (Go) | 21,216 | | 40,674 | 79.28% |
384-
| LZ4 9 | 17,139 | 1,407 | 95,706 | 83.26% |
385-
| LZ4 9 (Go) | 17,139 | | 39,709 | 83.26% |
399+
| Compressor | Size | Comp MB/s | Decomp MB/s | Reduction % |
400+
|---------------|--------|----------:|-------------|-------------|
401+
| MinLZ 1 | 20,184 | 17,558 | 82,292 | 80.29% |
402+
| MinLZ 1 (Go) | 19,849 | 15,035 | 32,327 | 80.62% |
403+
| MinLZ 2 | 17,831 | 9,260 | 58,432 | 82.59% |
404+
| MinLZ 2 (Go) | 17,831 | 7,524 | 25,728 | 82.59% |
405+
| MinLZ 3 | 16,025 | 180 | 80,445 | 84.35% |
406+
| MinLZ 3 (Go) | 16,025 | | 33,382 | 84.35% |
407+
| Snappy | 22,843 | 17,469 | 44,765 | 77.69% |
408+
| Snappy (Go) | 22,843 | 8,161 | 21,082 | 77.69% |
409+
| LZ4 0 | 21,216 | 9,452 | 101,490 | 79.28% |
410+
| LZ4 0 (Go) | 21,216 | | 40,674 | 79.28% |
411+
| LZ4 9 | 17,139 | 1,407 | 95,706 | 83.26% |
412+
| LZ4 9 (Go) | 17,139 | | 39,709 | 83.26% |
413+
| MinLZ -1 | 23,487 | ------ | ------- | 77.06% |
414+
| MinLZ -1 (Go) | 22,911 | ------ | ------- | 77.63% |
386415

387416
![Compression vs Size](img/html-block.png)
388417

@@ -666,6 +695,7 @@ File names beginning with 'http://' and 'https://' will be downloaded and compre
666695
Only http response code 200 is accepted.
667696
668697
Options:
698+
-0 Perform no compression
669699
-1 Compress faster, but with a minor compression loss
670700
-2 Default compression speed (default true)
671701
-3 Compress more, but a lot slower
@@ -695,6 +725,8 @@ Options:
695725
Do not overwrite output files
696726
-verify
697727
Verify files, but do not write output
728+
-xfast
729+
Compress fastest, with a major compression loss
698730
699731
Example:
700732
@@ -731,10 +763,12 @@ Options:
731763
-c Write all output to stdout. Multiple input files will be concatenated
732764
-cpu int
733765
Maximum number of threads to use (default 32)
766+
-follow
767+
Follow file like tail -f, reopening when EOF is reached
734768
-help
735769
Display help
736770
-limit string
737-
Return at most this much data. Examples: 92, 64K, 256K, 1M, 4M
771+
Return at most this much data. Examples: 92, 64K, 256K, 1M, 4M
738772
-o string
739773
Write output to another file. Single input file only
740774
-offset string

0 commit comments

Comments
 (0)