[improve][pip] PIP-492: Add pulsar_subscription_storage_backlog_age_seconds metric

pip/pip-492.md

@@ -0,0 +1,339 @@
+# PIP-492: Add subscription-level storage backlog age metric and stats field
+
+# Background Knowledge
+
+Pulsar currently exposes topic-level backlog age through the Prometheus metric
+`pulsar_storage_backlog_age_seconds` and through topic stats fields such as
+`oldestBacklogMessageAgeSeconds` and
+`oldestBacklogMessageSubscriptionName`. The value represents the age, in
+seconds, of the oldest unacknowledged message for a persistent topic. This
+information is useful because it measures consumer delay by time, rather than
+only reporting the number of messages in backlog.
+
+A persistent topic can have multiple subscriptions. Each subscription has its
+own managed ledger cursor and can therefore have a different backlog position
+and a different oldest unacknowledged message. In many deployments these
+subscriptions are used by different applications and have different latency
+requirements. For example, one subscription might be expected to consume in
+near real time, while another subscription intentionally processes historical
+data more slowly.
+
+The existing topic-level metric and stats fields identify the oldest
+subscription at topic granularity, but they do not expose backlog age for every
+subscription. Operators can use subscription backlog size metrics to find
+subscriptions with queued messages, but backlog size alone does not directly
+tell how old the delayed data is.
+
+# Motivation
+
+When a topic has multiple subscriptions, the topic-level backlog age metric can
+trigger an alert without showing which subscription is responsible. This makes
+alerting noisy and troubleshooting slower:
+
+* Operators cannot alert on backlog age independently for each subscription.
+* Subscriptions with intentionally slow processing can cause false alerts for
+  the whole topic.
+* Alert rules need additional joins or manual investigation to correlate topic
+  backlog age with subscription backlog state.
+* Users cannot easily express different latency objectives for different
+  subscriptions on the same topic.
+* Admin API users can see the topic-level oldest backlog age in topic stats,
+  but cannot inspect the same value for each subscription in the same response.
+
+This proposal introduces a subscription-level backlog age value in topic stats
+and Prometheus so that users can monitor message delay at the same granularity
+where consumption actually happens.
+
+# Goals
+
+## In Scope
+
+* Add a Prometheus metric named
+  `pulsar_subscription_storage_backlog_age_seconds`.
+* Report the metric per persistent durable subscription.
+* Include the standard subscription metric labels, including `cluster`,
+  `namespace`, `topic`, and `subscription`, plus existing optional topic metric
+  labels where applicable.
+* Keep the existing topic-level `pulsar_storage_backlog_age_seconds` metric
+  unchanged.
+* Add a broker configuration flag to gate the computation, defaulting to
+  disabled.
+* Expose the best-effort per-subscription backlog age in
+  `SubscriptionStats`, including the subscription entries returned by
+  `GET /admin/v2/persistent/{tenant}/{namespace}/{topic}/stats`.
+* Avoid emitting a metric sample when the value is unknown.
+
+## Out of Scope
+
+* Changing the semantics of `pulsar_storage_backlog_age_seconds`.
+* Adding subscription backlog age for non-persistent topics.
+* Reporting backlog age for non-durable Reader subscriptions in this PIP.
+* Adding OpenTelemetry-native metrics for this value. This proposal only covers
+  the existing Prometheus metrics path.
+* Adding per-message or per-consumer backlog age.
+
+# High Level Design
+
+The broker already has best-effort logic for finding the oldest backlog
+position and timestamp for a topic. This proposal reuses the same timestamp
+estimation and read logic for an optional per-subscription computation.
+
+When `exposeSubscriptionBacklogAgeInPrometheus` is disabled, the broker skips
+the per-subscription computation entirely. This preserves the existing default
+cost of the backlog quota checker and avoids charging large clusters for a
+high-cardinality metric they did not opt into.
+
+When the configuration is enabled, each persistent topic with backlog walks its
+active durable subscriptions and computes the age of the oldest unacknowledged
+message for each subscription that has backlog. The result is cached by
+subscription name and mark-delete position. If a subscription's mark-delete
+position has not changed, the broker reuses the cached value instead of
+performing another estimate or read.
+
+The topic stats endpoint exposes the cached value in each subscription's stats
+object. The Prometheus exporter emits the same value through the existing
+subscription metric path. Values of `-1` mean the backlog age is unknown or not
+applicable, and no metric sample is emitted for those values.
+
+# Detailed Design
+
+## Design & Implementation Details
+
+The implementation updates the persistent topic backlog age cache and stats
+pipeline:
+
+* `PersistentTopic` maintains a per-subscription cache of oldest backlog
+  position information.
+* `BrokerService` refreshes the subscription-level cache from a dedicated
+  background task when `exposeSubscriptionBacklogAgeInPrometheus` is enabled.
+  The existing topic-level `PersistentTopic.updateOldPositionInfo()` path
+  remains focused on the backlog quota cache.
+* The computation iterates over `PersistentTopic.subscriptions`, not over the
+  whole managed cursor container. This limits the scope to real subscriptions
+  and avoids internal cursors such as replication, deduplication, and compaction
+  cursors.
+* Non-durable Reader subscriptions are skipped. Their cursor names and version
+  behavior differ from durable subscriptions, and including them would need
+  separate correctness and cardinality considerations.
+* If a subscription has no backlog, the cached value is cleared.
+* If a subscription has backlog and its mark-delete position has not changed,
+  the cached value is reused.
+* If a subscription has backlog and its mark-delete position changed, the
+  broker computes a new oldest backlog message timestamp:
+  * In non-precise mode, it uses the existing estimated time-based backlog quota
+    check path.
+  * In precise mode, it reads the next valid entry after the subscription's
+    mark-delete position and uses that entry timestamp.
+* `PersistentSubscription.getStatsAsync()` copies the cached best-effort value
+  into `SubscriptionStatsImpl.oldestBacklogMessageAgeSeconds`, so
+  `GET /admin/v2/persistent/{tenant}/{namespace}/{topic}/stats` includes the
+  value for each subscription.
+* `NamespaceStatsAggregator` copies the value into aggregated subscription
+  stats.
+* `TopicStats.printTopicStats()` emits
+  `pulsar_subscription_storage_backlog_age_seconds` when the configuration is
+  enabled and the value is not `-1`.
+
+The value is best effort, just like the existing topic-level backlog age. It is
+updated by the background refresh task and is not recomputed synchronously for
+every stats request.
+
+## Public-facing Changes
+
+### Public API
+
+Add a getter to `org.apache.pulsar.common.policies.data.SubscriptionStats`:
+
+```java
+default long getOldestBacklogMessageAgeSeconds() {
+    return -1;
+}
+```
+
+The default implementation preserves source and binary compatibility for
+third-party implementations of `SubscriptionStats`.
+
+The value is:
+
+* `>= 0`: the best-effort age in seconds of the oldest unacknowledged message
+  for this subscription.
+* `-1`: unknown, not applicable, no backlog, or the broker configuration is
+  disabled.
+
+The topic stats response includes this field under each subscription entry. For
+example:
+
+```json
+{
+  "subscriptions": {
+    "my-sub": {
+      "msgBacklog": 100,
+      "oldestBacklogMessageAgeSeconds": 86400
+    }
+  },
+  "oldestBacklogMessageAgeSeconds": 86400,
+  "oldestBacklogMessageSubscriptionName": "my-sub"
+}
+```
+
+### Binary protocol
+
+No Pulsar binary protocol changes are proposed.
+
+### Configuration
+
+Add a broker configuration key:
+
+```properties
+exposeSubscriptionBacklogAgeInPrometheus=false
+```
+
+When disabled, the broker does not compute per-subscription backlog age.
+`SubscriptionStats.oldestBacklogMessageAgeSeconds` remains `-1`, and
+`pulsar_subscription_storage_backlog_age_seconds` is not emitted.
+
+When enabled, the broker computes best-effort backlog age for persistent durable
+subscriptions. The computed value is available through `SubscriptionStats`.
+Prometheus emission also requires topic-level metrics to be enabled.
+
+The default is `false` because the feature adds a per-subscription walk and, on
+cache misses, a per-subscription estimate or read. This follows the precedent
+of other potentially expensive per-entity metrics such as
+`exposePreciseBacklogInPrometheus`,
+`exposeSubscriptionBacklogSizeInPrometheus`,
+`exposeConsumerLevelMetricsInPrometheus`, and
+`exposeManagedCursorMetricsInPrometheus`.
+
+### CLI
+
+No new CLI option or command is proposed. Existing commands that print the
+topic stats response, such as `pulsar-admin topics stats`, will include
+`oldestBacklogMessageAgeSeconds` in each subscription stats object once the
+admin API response includes the field.
+
+### Metrics
+
+Add the following Prometheus metric:
+
+| Name | Type | Unit | Description | Labels |
+| --- | --- | --- | --- | --- |
+| `pulsar_subscription_storage_backlog_age_seconds` | Gauge | seconds | Best-effort age of the oldest unacknowledged message for a persistent durable subscription. | Standard subscription metric labels, including `cluster`, `namespace`, `topic`, and `subscription`, plus existing optional topic metric labels where applicable. |
+
+A Prometheus sample for this metric is emitted only when:
+
+* `exposeTopicLevelMetricsInPrometheus=true`;
+* `exposeSubscriptionBacklogAgeInPrometheus=true`;
+* the topic is persistent;
+* the subscription is durable; and
+* the computed value is not `-1`.
+
+The per-subscription backlog age computation itself is gated by
+`exposeSubscriptionBacklogAgeInPrometheus`, not by
+`exposeTopicLevelMetricsInPrometheus`. Therefore, when
+`exposeSubscriptionBacklogAgeInPrometheus=true` and
+`exposeTopicLevelMetricsInPrometheus=false`, the broker can still populate the
+admin stats field, but the Prometheus topic/subscription-level sample is not
+printed.
+
+# Monitoring
+
+Operators can use the new metric to alert on subscription-specific consumer
+delay. For example, a real-time subscription can alert when
+`pulsar_subscription_storage_backlog_age_seconds` exceeds a small threshold,
+while a batch or replay subscription can have a larger threshold or no alert.
+
+The existing topic-level metric remains useful as a coarse topic-level signal.
+The subscription-level metric should be used when operators need to identify
+which subscription is responsible for the delay or when subscriptions on the
+same topic have different latency objectives.
+
+Because this metric adds one potential time series per persistent durable
+subscription, operators should enable it only on clusters that need this
+granularity and should review the resulting Prometheus cardinality.
+
+# Security Considerations
+
+This proposal does not add new endpoints, protocol commands, or authorization
+rules. The new value is exposed through existing topic stats and metrics
+surfaces, so access control follows the existing admin API and metrics endpoint
+configuration.
+
+The metric exposes subscription names in the `subscription` label, consistent
+with existing subscription-level metrics.
+
+# Backward & Forward Compatibility
+
+## Upgrade
+
+No special upgrade action is required. The feature is disabled by default.
+
+After upgrading, operators who need the new metric can enable:
+
+```properties
+exposeSubscriptionBacklogAgeInPrometheus=true
+```
+
+The existing topic-level backlog age metric remains unchanged.
+
+## Downgrade / Rollback
+
+To roll back the feature before downgrading, disable:
+
+```properties
+exposeSubscriptionBacklogAgeInPrometheus=false
+```
+
+Older brokers will ignore the new metric and will not expose
+`SubscriptionStats.oldestBacklogMessageAgeSeconds`.
+
+## Pulsar Geo-Replication Upgrade & Downgrade/Rollback Considerations
+
+No special geo-replication handling is required. The metric is computed locally
+per broker from local persistent topic subscription state. Replication
+subscriptions and other internal cursors are not included.
+
+# Performance Evaluation
+
+The default configuration has no additional per-subscription computation cost
+because the broker skips the subscription backlog age calculation entirely.
+
+When enabled, each update pass is proportional to the number of persistent
+durable subscriptions on a topic. On a cache miss, each backlogged subscription
+requires one timestamp estimate in non-precise mode or one entry read in precise
+mode. On a cache hit, the cached result is reused when the subscription
+mark-delete position has not changed.
+
+A local synthetic benchmark was run with an embedded broker, one persistent
+topic, and many durable subscriptions. The benchmark used non-precise backlog
+age calculation, wrote messages to create backlog, reset all subscription
+cursors to the earliest position, and measured one subscription backlog age
+refresh pass.
+
+| Subscriptions | Disabled | Enabled first pass | Enabled cache hit |
+| ---: | ---: | ---: | ---: |
+| 20,000 | avg 0 ms | 32 ms | avg 5 ms, p95 6 ms |
+| 50,000 | avg 0 ms | 108 ms | avg 15 ms, p95 16 ms |
+
+The same benchmark verified that all subscription stats had populated backlog
+age values:
+
+* 20,000 subscriptions: `populatedBacklogAgeCount=20000`
+* 50,000 subscriptions: `populatedBacklogAgeCount=50000`
+
+The benchmark is synthetic and should not be treated as a production latency
+guarantee. Precise mode can involve managed ledger reads and can be more
+expensive. The result supports keeping the feature opt-in and caching by
+subscription mark-delete position.
+
+# General Notes
+
+The proposed subscription-level value complements, rather than replaces, the
+existing topic-level backlog age metric and stats fields. Topic-level backlog
+age remains a useful high-level signal, while subscription-level backlog age
+identifies which subscription is delayed.
+
+# Links
+
+* Mailing List discussion thread: https://lists.apache.org/thread/hjmkwx20d408f4s3orf1b8m1qp03oshz
+* Mailing List voting thread:
+* Pull request: https://github.com/apache/pulsar/pull/26313