Add Zstandard compression support and update tests

[JakeChampion]

This pull request adds full support for the zstd (Zstandard) compression algorithm throughout Apache Traffic Server, including build system integration, compression plugin support, Accept-Encoding header normalization, and test coverage.

Key Features

Build system and dependencies:

• Add CMake support for finding zstd library with new Findzstd.cmake
• Update Docker build files to include libzstd-dev package
• Add TS_HAS_ZSTD feature flag for conditional compilation

Core compression support:

• Extend compress plugin to support zstd compression alongside gzip and brotli
• Add zstd stream handling structures and functions
• Update compression configuration to include zstd in supported algorithms list

Accept-Encoding header normalization:

• Extend proxy.config.http.normalize_ae configuration to support values 4 and 5 for zstd normalization
• Update HTTP transaction cache matching to handle zstd encoding

API and infrastructure:

• Add TS_HTTP_VALUE_ZSTD and TS_HTTP_LEN_ZSTD constants
• Update MIME field handling to recognize zstd encoding
• Add zstd support to traffic_layout feature detection

Improved cache matching:

• Fix Accept-Encoding quality calculation logic in HTTP transaction cache
• Replace multiplicative quality combining with minimum quality selection
• Remove problematic gzip-specific fallback logic that caused cache inconsistencies
• Ensure proper cache behavior for all compression algorithms

Test Coverage

• Comprehensive compress plugin tests covering zstd compression scenarios
• Accept-Encoding normalization tests for all zstd combinations
• Updated golden files to include zstd compression test results
• New compress3.config for zstd-specific plugin configuration
• Test all combinations of zstd, br, and gzip in various scenarios
• Cache matching tests demonstrate correct behavior without workarounds

Standards Compliance

The implementation follows RFC 8878 standards for zstd compression and maintains backward compatibility with existing gzip and brotli compression functionality. All tests pass and the feature is properly integrated with existing caching and content negotiation mechanisms.

Configuration

New normalization modes:

• proxy.config.http.normalize_ae = 4: Prioritize zstd, fallback to br then gzip
• proxy.config.http.normalize_ae = 5: Support all combinations of zstd, br, and gzip

Benefits

• Improved compression ratios compared to gzip
• Better performance than brotli for compression speed
• Reduced bandwidth usage and faster page loads
• Seamless integration with existing ATS infrastructure

[masaori335]

[approve ci]

[JakeChampion]

@masaori335 do you know which files I need to modify to install zstd on the osx used in the Jenkins job?

[masaori335]

We need to ask @ezelkow1 to install the package in the osx env, I guess.

However, this PR has #if HAVE_ZSTD_F check, why it's failing?

[JakeChampion]

We need to ask @ezelkow1 to install the package in the osx env, I guess.

However, this PR has #if HAVE_ZSTD_F check, why it's failing?

thank you, _F was a mistake, it was meant to be _H

[JakeChampion]

Looks like all the FreeBSD machines for CI are offline

[shukitchan]

Really appreciate it if you can please add some documentation to the changes.

Thanks.

[cmcfarlen]

Hi! This doesn't compile for me. I added a line to include/tscore/ink_config.cmake.in so HAVE_ZSTD_H would be truthy for the CPP macros. This lit up that code, but then there were several compiler errors:

/Users/cmcfarlen/projects/oss/trafficserver/plugins/compress/compress.cc:203:9: error: no member named 'zstd_ctx' in 'Data'; did you mean 'zstd_cctx'?
  203 |   data->zstd_ctx = nullptr;
      |         ^~~~~~~~
      |         zstd_cctx
/Users/cmcfarlen/projects/oss/trafficserver/plugins/compress/misc.h:93:14: note: 'zstd_cctx' declared here
   93 |   ZSTD_CCtx *zstd_cctx;
      |              ^
/Users/cmcfarlen/projects/oss/trafficserver/plugins/compress/compress.cc:206:11: error: no member named 'zstd_ctx' in 'Data'; did you mean 'zstd_cctx'?
  206 |     data->zstd_ctx = ZSTD_createCCtx();
      |           ^~~~~~~~
      |           zstd_cctx
/Users/cmcfarlen/projects/oss/trafficserver/plugins/compress/misc.h:93:14: note: 'zstd_cctx' declared here
   93 |   ZSTD_CCtx *zstd_cctx;
      |              ^
/Users/cmcfarlen/projects/oss/trafficserver/plugins/compress/compress.cc:207:16: error: no member named 'zstd_ctx' in 'Data'; did you mean 'zstd_cctx'?
  207 |     if (!data->zstd_ctx) {
      |                ^~~~~~~~
      |                zstd_cctx
/Users/cmcfarlen/projects/oss/trafficserver/plugins/compress/misc.h:93:14: note: 'zstd_cctx' declared here
   93 |   ZSTD_CCtx *zstd_cctx;
      |              ^

Please include this patch in your PR to enable the ZSTD code. Thanks!

diff --git a/include/tscore/ink_config.h.cmake.in b/include/tscore/ink_config.h.cmake.in
index 634ed94c3..7da0771fd 100644
--- a/include/tscore/ink_config.h.cmake.in
+++ b/include/tscore/ink_config.h.cmake.in
@@ -184,3 +184,5 @@ const int DEFAULT_STACKSIZE = @DEFAULT_STACK_SIZE@;
 #cmakedefine YAMLCPP_LIB_VERSION "@YAMLCPP_LIB_VERSION@"

 #cmakedefine01 TS_HAS_CRIPTS
+
+#cmakedefine HAVE_ZSTD_H 1

[cmcfarlen]

Please patch ink_config.h and resolve compiler issues.

[JakeChampion]

@cmcfarlen do you know if there's anything I can do to help get this reviewed? I'm happy to join a mailing list or slack, discord, etc - my work uses ATS quite a bit and I'm eager to ensure we contribute back to the project where we can

[JosiahWI]

You can find the dev mailing list (and other mailing lists) on apache.org. I'm sure we can get you on Slack as well. I don't have a lot of availability, so I likely won't be reviewing this whole change, but hopefully this is a helpful start.

Feedback has been addressed. Thank you!

Comments have been addressed.