Add cache hits/misses to telemetry

[niemyjski]

Context

The caching layer already tracks _hits, _misses, and _writes internally via Interlocked.Increment, but these are never exposed as System.Diagnostics.Metrics instruments. The HybridCacheClient similarly tracks _localCacheHits and _invalidateCacheCalls without emitting metrics. This means cache behavior is invisible to OpenTelemetry, Prometheus, dotnet-counters, and any other standard metrics tooling.

Approach

Follow the exact same pattern already established in QueueBase<T, TOptions> -- use FoundatioDiagnostics.Meter.CreateCounter<long>(...) and CreateObservableGauge<long>(...). Zero overhead when no listener is attached (built into the .NET Metrics API). No new dependencies required.

Key Design: Prefix-Based Tier Separation

Add a configurable MetricsPrefix to InMemoryCacheClientOptions (mirrors SharedQueueOptions.MetricsPrefix). This solves the HybridCacheClient double-counting problem without any suppression flags:

Standalone InMemoryCacheClient (prefix: "foundatio.cache")
  -> foundatio.cache.hit
  -> foundatio.cache.miss
  -> foundatio.cache.set
  -> foundatio.cache.remove
  -> foundatio.cache.count  (observable gauge)
  -> foundatio.cache.size   (observable gauge, when MaxMemorySize configured)

HybridCacheClient
  Internal _localCache (prefix: "foundatio.cache.local")
    -> foundatio.cache.local.hit
    -> foundatio.cache.local.miss
    -> foundatio.cache.local.set
    -> foundatio.cache.local.remove
    -> foundatio.cache.local.count

  Distributed ICacheClient (keeps its own "foundatio.cache" prefix -- no changes needed)
    -> foundatio.cache.hit
    -> foundatio.cache.miss
    -> foundatio.cache.set
    -> foundatio.cache.remove

  HybridCacheClient itself:
    -> foundatio.cache.invalidation  (remote invalidation messages received)

No double-counting. No suppression flags. Tier separation falls out naturally from the metric name prefix.

Metric Instruments

Counters (every ICacheClient implementation)

Instrument	Description
{prefix}.hit	Cache read that found a value
{prefix}.miss	Cache read that did not find a value
{prefix}.set	Cache write/update operations
{prefix}.remove	Cache removal operations

Observable Gauges (InMemoryCacheClient only)

Instrument	Description
{prefix}.count	Current number of items in cache
{prefix}.size	Current memory size in bytes (only when MaxMemorySize configured)

Hybrid-specific Counter

Instrument	Description
foundatio.cache.invalidation	Remote invalidation messages received by HybridCacheClient

Future

• foundatio.cache.stampede -- when a GetOrAdd pattern lands and coalesces waiters

Tags (low cardinality)

• cache.name -- optional instance name (only emitted when configured; avoids cardinality explosion)

Implementation Steps

1. Add MetricsPrefix and Name to InMemoryCacheClientOptions

// src/Foundatio/Caching/InMemoryCacheClientOptions.cs
public string MetricsPrefix { get; set; } = "foundatio.cache";
public string? Name { get; set; } // used as cache.name tag value when set

2. Instrument InMemoryCacheClient

• Add fields: Counter<long> _hitCounter, _missCounter, _setCounter, _removeCounter
• Add fields: ObservableGauge<long> _countGauge, _sizeGauge (when MaxMemorySize configured)
• Create in constructor using FoundatioDiagnostics.Meter.CreateCounter<long>($"{prefix}.hit", ...)
• At each existing Interlocked.Increment(ref _hits), also call _hitCounter.Add(1) (same for misses/writes/removes)

3. Wire HybridCacheClient with local prefix

• When creating _localCache, pass MetricsPrefix = "foundatio.cache.local" in the options
• Add a Counter<long> _invalidationCounter for foundatio.cache.invalidation -- increment where _invalidateCacheCalls currently increments

4. Tests

Use the existing InMemoryMetrics test utility (src/Foundatio.TestHarness/Utility/InMemoryMetrics.cs) to verify:

• Standalone InMemoryCacheClient emits foundatio.cache.hit/miss/set/remove
• HybridCacheClient emits foundatio.cache.local.hit/miss and foundatio.cache.invalidation
• Observable gauges report correct count/size values

5. Sample

samples/Foundatio.HostingSample/ServiceDefaults.cs already registers .AddMeter("Foundatio") -- no changes needed, all new cache metrics flow automatically.

Design Decisions

• No new packages -- uses only System.Diagnostics.Metrics already in the project
• No EventSource -- System.Diagnostics.Metrics is the modern path; it's what Foundatio already uses for queues and locks
• No suppression logic -- prefix-based naming means tier separation is automatic
• Backward compatible -- existing Hits/Misses/Writes properties and ResetStats() remain unchanged
• Near-zero overhead -- .Add(1) on a Counter<long> with no listener is effectively a no-op in the .NET runtime
• Consistent naming -- follows the foundatio.{component}.{metric} pattern from the queue layer
• Extensible -- Redis, Azure, AWS cache client repos can add the same instruments using the same prefix convention

[niemyjski]

@ejsmith let me know what your thoughts are on this plan.