[improve][pip] PIP-492: Add pulsar_subscription_storage_backlog_age_seconds metric

[Technoboy-]

Motivation

Pulsar currently exposes pulsar_storage_backlog_age_seconds to report the age of the oldest unacknowledged message for a topic. This metric is useful for detecting consumer lag based on actual message age instead of only backlog size.

However, the metric is only available at the topic level. When a topic has multiple subscriptions, operators cannot tell which subscription is contributing to the backlog age. This is a common production case because different subscriptions on the same topic can have different latency expectations. Some subscriptions are expected to consume messages in real time, while others may intentionally retain backlog for a longer period.

Without a subscription-level backlog age metric, users have to either alert on the topic-level metric and tolerate false positives, or build complex Prometheus rules to exclude known slow subscriptions. This makes monitoring harder and less accurate.

This change adds pulsar_subscription_storage_backlog_age_seconds, allowing users to monitor backlog age per subscription and configure alerts based on each subscription's expected latency.

Modifications

Verifying this change

• Make sure that the change passes the CI checks.

(Please pick either of the following options)

This change is a trivial rework / code cleanup without any test coverage.

(or)

This change is already covered by existing tests, such as (please describe tests).

(or)

This change added tests and can be verified as follows:

(example:)

• Added integration tests for end-to-end deployment with large payloads (10MB)
• Extended integration test for recovery after broker failure

Does this pull request potentially affect one of the following parts:

If the box was checked, please highlight the changes

• Dependencies (add or upgrade a dependency)
• The public API
• The schema
• The default values of configurations
• The threading model
• The binary protocol
• The REST endpoints
• The admin CLI options
• The metrics
• Anything that affects deployment

[Goodkat]

I find this proposal very useful.
Could it be possible to expose this metric value also in the topic stats command, like this?

GET /admin/v2/persistent/{tenant}/{namespace}/{topic}/stats

At the topic level, it returns values as shown below:

{
  "oldestBacklogMessageAgeSeconds": 86400,
  "oldestBacklogMessageSubscriptionName": "my-sub",
  ...
}

It would be great to include such a value for every subscription in the stats command output.

[Technoboy-]

I find this proposal very useful. Could it be possible to expose this metric value also in the topic stats command, like this?

GET /admin/v2/persistent/{tenant}/{namespace}/{topic}/stats

At the topic level, it returns values as shown below:

{
  "oldestBacklogMessageAgeSeconds": 86400,
  "oldestBacklogMessageSubscriptionName": "my-sub",
  ...
}

It would be great to include such a value for every subscription in the stats command output.

sure. added