history: add RPC P99 latency check to DeepHealthCheck

[laniehei]

Summary

• Adds CheckTypePersistenceP99Latency (check 6) to DeepHealthCheck with a configurable threshold (default 500ms)
• Adds CheckTypeRPCP99Latency (check 7) to DeepHealthCheck with a configurable threshold (default 1000ms)
• Adds Percentile(p float64) method to MovingWindowAvgImpl to compute percentiles over the moving window
• Adds P99Latency() to both persistence.HealthSignalAggregator and interceptor.HealthSignalAggregator
• Emits history_rpc_p99_latency_ms gauge on each DeepHealthCheck call

Motivation

The existing DeepHealthCheck checks average latency and error ratio. These are blind to incidents where a single pod has degraded persistence connectivity: the average latency stays low because only a small fraction of operations (e.g. shard recovery) are slow, and requests hang to deadline rather than returning errors so the error ratio stays near zero.

The P99 checks catch tail latency degradation that averages hide:

• Persistence P99 fires on the affected pod during shard recovery when persistence operations are slow — detecting the root cause before downstream callers are affected
• RPC P99 fires when inbound RPCs are hanging — a backstop for latency issues that don't go through persistence

Test plan

• go test ./common/aggregate/... — Percentile method
• go test ./common/persistence/... — P99Latency() on persistence aggregator
• go test ./common/rpc/interceptor/... — P99Latency() on RPC aggregator
• go test ./service/history/... — handler + config wiring

🤖 Generated with Claude Code