Add Zstandard compression support to the compress plugin #12211
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
JakeChampion
force-pushed
the
jake/9.2.10-zstd
branch
19 times, most recently
from
5f12f8e to
85ec4ab
Compare
|
@JakeChampion : Thank you for working on this. It looks like a few autests have output expectation issues. The Also, the |
|
We don't add new features to patch releases and we are not planning on having another minor release of 9.x. We are happy for you to use CI to verity the change and we can have someone look over the code for you. 9.2.x is currently our LTS release and we are doing new development on ATS 10.x. For example on out release versioning: major.minor.patch |
JakeChampion
force-pushed
the
jake/9.2.10-zstd
branch
2 times, most recently
from
36cb81a to
0547b04
Compare
JakeChampion
force-pushed
the
jake/9.2.10-zstd
branch
13 times, most recently
from
9dd2e2c to
90bc687
Compare
This patch adds full support for the zstd (Zstandard) compression algorithm throughout Apache Traffic Server, including build system integration, compression plugin support, Accept-Encoding header normalization, and comprehensive test coverage. Build system and dependencies: - Add CMake support for finding zstd library with new Findzstd.cmake - Update Docker build files to include libzstd-dev package - Add TS_HAS_ZSTD feature flag for conditional compilation Core compression support: - Extend compress plugin to support zstd compression alongside gzip and brotli - Add zstd stream handling structures and functions - Update compression configuration to include zstd in supported algorithms list - Add zstd compression type constant and related infrastructure Accept-Encoding header normalization: - Extend proxy.config.http.normalize_ae configuration to support values 4 and 5 for zstd normalization - Add zstd support to header normalization logic with proper priority handling (zstd > br > gzip) - Update HTTP transaction cache matching to handle zstd encoding - Add zstd token to header parsing infrastructure API and infrastructure: - Add TS_HTTP_VALUE_ZSTD and TS_HTTP_LEN_ZSTD constants - Update MIME field handling to recognize zstd encoding - Add zstd support to traffic_layout feature detection Test coverage: - Expand compress plugin tests to cover zstd compression scenarios - Add zstd test cases to Accept-Encoding normalization tests - Update golden files to include zstd compression test results - Add new compress3.config for zstd-specific plugin configuration - Test all combinations of zstd, br, and gzip in various scenarios The implementation follows RFC 8878 standards for zstd compression and maintains backward compatibility with existing gzip and brotli compression functionality. All tests pass and the feature is properly integrated with the existing caching and content negotiation mechanisms. # Conflicts: # plugins/compress/compress.cc # tests/gold_tests/pluginTest/compress/compress.gold # tests/gold_tests/pluginTest/compress/compress.test.py
This patch fixes the Accept-Encoding quality calculation logic in the HTTP transaction cache and updates test files to reflect the corrected behavior, removing workarounds for previously broken cache matching. Cache quality calculation improvements: - Replace multiplicative quality combining with minimum quality selection for multiple content encodings - Simplify wildcard matching to return quality directly - Remove gzip-specific fallback logic that bypassed normal quality calculation and caused inconsistent cache behavior Test updates for corrected behavior: - Remove NOTICE comments describing broken cache matching behavior - Update test expectations to match correct cache responses instead of workaround responses - Fix cache hit/miss expectations for multi-encoding scenarios - Add missing Content-Encoding headers in server responses - Update response identifiers to match the actual cached alternates Specific test corrections: - "br, compress, gzip" now correctly matches "Br-Gzip-Accept-Encoding" instead of falling back to "Gzip-Accept-Encoding" - "zstd, compress, gzip" now correctly matches "Zstd-Gzip-Accept-Encoding" instead of falling back to "Gzip-Accept-Encoding" - "zstd, gzip;q=0.8" now correctly matches "Zstd-Gzip-Accept-Encoding" instead of falling back to "Gzip-Accept-Encoding" - Individual encoding requests now create proper cache alternates instead of incorrectly matching empty encoding alternates The previous multiplicative quality calculation (q_a * q_b) was causing unexpectedly low quality scores and incorrect cache alternate selection. The new minimum quality approach ensures that multi-encoding responses are properly cached and matched according to HTTP content negotiation standards. All test cases now pass without workarounds and demonstrate correct cache behavior for all compression algorithms (gzip, brotli, zstd) across various Accept-Encoding
… for compression encoding # Conflicts: # src/proxy/hdrs/HdrToken.cc
- Add support for configurable compression levels per host: * gzip-compression-level (1-9, default 6) * brotli-compression-level (0-11, default 6) * brotli-lgwin (10-24, default 16) * zstd-compression-level (1-22, default 12) This enables fine-tuning compression performance vs. speed trade-offs on a per-host basis
JakeChampion
force-pushed
the
jake/9.2.10-zstd
branch
from
304beaa to
ff96b24
Compare
JakeChampion
force-pushed
the
jake/9.2.10-zstd
branch
from
ff96b24 to
cee7bbd
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is a backport of #12201