[observability] Supporting different OTel temporality

[dprotaso]

A user reported that metrics couldn't be uploaded to their backend provider. (knative/eventing#8668)

It was discovered that their metrics backend (Dynatrace) only accepts metrics using a delta temporality (knative/eventing#8668 (comment))

There is an env var that will let you change the temporality OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE=delta but in the otel-go sdk this only happens for certain instruments.

Looking at backend documentation it seems inconsistent which instruments should be

Datadog documentation shows several instruments should be using delta but this list differs from the otel-go sdk (datadog includes Gauges). Dynatrace expects all instruments to use delta.

So there is this massive inconsistency occurring that I brought up in slack looking for answers.

I originally made a PR (#3246) to make temporality a config map option but I'm not confident this will work with different backends.

[dprotaso]

In the slack thread the dynatrace folks realized their docs was incorrectly creating a temporality selector function and that the env var value delta should work properly.

[dprotaso]

I created a PR to export the temporality functions here - open-telemetry/opentelemetry-go#7346

[dprotaso]

I'm going to close this out - I think a viable workaround is sending metrics to an OTel collector and then forwarding to the backend.

Dynatrace specifically has instructions how to setup a collector in such a way using the cumulativetodelta processor

https://docs.dynatrace.com/docs/ingest-from/opentelemetry/collector/use-cases/prometheus

I think this approach works better as we can push configuration knobs (eg. backend auth etc) into the collector