Support for optimized histogram representations (otel "ExponentialHistogram" or prometheus native histogram)

[alex]

Current histogram aggregation has a bunch of downsides, which mostly revolve around accuracy, efficiency, and annoyance from bucket management.

OpenTelemetry has introduced a new ExponentialHistogram, and prometheus has an (experimental) native histogram. My understanding is that while these are not the same, their representations are convertible.

It'd be outstanding if metrics were to make these types available. I think there's an API question on whether the public API for these is to continue to use histogram! and then on the backend you configure that all your histograms are native ones, or if it's a separate public API for them.

[tobz]

The public API for recording a histogram is always, always, going to be histogram!.

Choosing how to capture/render those histograms is purely an exporter concern. It should definitely be possible to update metrics-exporter-prometheus to support emitting histograms as native histograms instead of aggregated histograms.

[alex]

Ok! So then this bug is: support configuring metrics-exporter-prometheus to use native histograms.

[tobz]

Yep, that makes sense and is definitely possible. I'm open to reviewing a PR that adds such support to metrics-exporter-prometheus.

For anyone who decides to try taking this on, I'd say that one question to answer would be if an aggregated histogram can be switched to a native histogram transparently: if we just start emitting all aggregated histograms as native histograms instead, would Grafana dashboards break or anything?

If it can be done transparently, then we'd likely just want a builder method that enables native histograms across the board. If it can't be done transparently, then we'd likely want to have it only apply to histograms that lack a custom bucket configuration. This would provide a way to lock in existing histograms as aggregated histograms so that they don't get caught up in the change.

[alex]

"Is this backwards compat" is complex: https://grafana.com/docs/mimir/latest/send/native-histograms/#migrate-from-classic-histograms lays out what a migration looks like. I'll probably take a look at implementing this this weekend.