[ResponseCaching] Correctly handle Vary: * in delimited or multi-value headers (RFC 9111)

[Sadik00789]

Fixes #69192.

Per RFC 9111 § 4.1, a response containing * anywhere as a member of the Vary header field MUST NOT be used to satisfy a subsequent request. Previously, ResponseCachingPolicyProvider performed an exact match on varyHeader.Count == 1 && string.Equals(varyHeader, "*", ...) which failed when:

1. Downstream middleware (e.g., Response Compression) appended headers, resulting in varyHeader.Count > 1.
2. The Vary header contained comma-delimited tokens (e.g., *, Accept-Encoding).

Changes

• Output Caching Audit: Verified that OutputCaching determines vary behavior via explicit cache key rules (IOutputCachePolicy, CacheVaryByRules) rather than reading context.Response.Headers.Vary. No changes required there.
• Response Caching:
  • Replaced the exact-match check with span-based token inspection using MemoryExtensions.Split(',') on rawHeader.AsSpan().
  • Iterated over StringValues directly by index to avoid string.Join allocations.
  • Trimmed each segment span and verified equality via SequenceEqual("*") without heap allocations.
• Tests:
  • Expanded unit test IsResponseCacheable_VaryHeaderByStar_NotAllowed to a [Theory] covering single strings, multi-entry StringValues, and delimited values with spaces.
  • Added unit test asserting valid headers without * remain cacheable.
  • Added an integration test in ResponseCachingMiddlewareTests with downstream appended headers to verify subsequent requests are not served from cache.

Customer Impact

Prevents incorrectly caching responses that vary by * when response compression or other middleware appends headers to Vary.

Regression?

No.

Risk

Low. Restricts caching behavior to strictly adhere to RFC 9111 without allocating heap strings.

[dotnet-policy-service]

Thanks for your PR, @Sadik00789. Someone from the team will get assigned to your PR shortly and we'll get it reviewed.

[Sadik00789]

@dotnet-policy-service agree

[Youssef1313]

Any reason to not use a span here? Something like:

var rawHeader = varyHeader[i].AsSpan();
if (rawHeader.Length == 0)
{
    continue;
}

foreach (var segment in rawHeader.Split(','))
{
    if (rawHeader[segment].Trim().SequenceEquals("*"))
    {
        context.Logger.ResponseWithVaryStarNotCacheable();
        return false;
    }
}

[Sadik00789]

Updated to use Span.Split(',') and zero-allocation span checks, and cleaned up the delimiters array. Thanks for the suggestion!

[Youssef1313]

What about OutputCaching?

[Sadik00789]

During the initial audit, I checked src/Middleware/OutputCaching/ and found that OutputCaching does not inspect context.Response.Headers.Vary when determining cache eligibility (IOutputCachePolicy / AllowCacheStorage). Cache vary behavior there is driven strictly by its own policy rules (CacheVaryByRules / OutputCacheKeyProvider).

Did you want OutputCacheMiddleware / DefaultOutputCachePolicy to also inspect downstream response headers and refuse to cache if Vary: * is present, or should CacheVaryByRules handle * explicitly? If so, I'm happy to tackle that in this PR or a follow-up issue.

[Sadik00789]

Applied the suggested changes (IsEmpty and is ['*'] pattern matching). Ready for another look!

[cincuranet]

LGTM. @Youssef1313 do you want to give it a look?