Define guidelines for stability level of metrics

[ChrsMark]

As mentioned at #13870 (review) Collector's metrics start from alpha. However maturity levels of collector components and within OTel in general start from development: https://github.com/open-telemetry/opentelemetry-specification/blob/v1.49.0/oteps/0232-maturity-of-otel.md#maturity-levels

open-telemetry/opentelemetry-collector-contrib#42809 adds explicitly the stability level for metrics of Contrib's components. The ultimate goal is to have mdatagen to require stability (to be decided).

In this, I suggest we tune https://opentelemetry.io/docs/collector/internal-telemetry/#telemetry-maturity-levels accordingly to align with https://github.com/open-telemetry/opentelemetry-specification/blob/v1.49.0/oteps/0232-maturity-of-otel.md#maturity-levels, and define guidelines within the Collector's project regarding how stability level should be handled. We focus on metrics by now, but this would expand to attributes too.

Things to consider:

• how to move through stability levels: [chore][vcenterreceiver] Add stability level per metric opentelemetry-collector-contrib#42948 (review)
• if from should be required for levels beyond development: [chore][mysqlreceiver] Add stability level per metric opentelemetry-collector-contrib#42973 (review)

[codeboten]

Thanks for opening this issue. When the telemetry levels were originally added to opentelemetry.io, i recall that there were very few distinctions between what development and alpha would mean for emitted metrics, this is why development isn't even listed in the published document.

I do think it makes sense for a beta level to be added once a metric is adhering to any semantic conventions necessary, but I wouldn't expect a release candidate level to be present. I guess my proposal is:

• alpha: initial state, no guarantees as is documented today
• beta: once metrics have been reviewed and adhere to semconv (as necessary, i don't know how this would be enforced for all components, maybe a requirement to provide a weaver schema)
• stable: after X number of beta releases a metric can be moved to stable
• deprecated: a metric can go from any previous states to deprecated. not necessary for alpha, but for beta and stable metrics, we should have at least X releases after a metric is deprecated before it is removed (maybe 2 releases for beta and 4 for stable)

For reference:

• Collector telemetry lifecycle opentelemetry.io#5092
• [telemetry] Define lifecycle and stability of telemetry #10003

[ChrsMark]

Thank's for the input @codeboten!

My thought to use development is mostly influenced by the fact that the initial stability level in SemConv is development (example).

I would be fine skipping it however I'm not sure if this can work well in alignment with SemConv. Adding a new metric in component X with stability alpha would mean the corresponding SemConv is at least alpha too?

Based on https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/coding-guidelines.md?plain=1#L335-L352 we already suggest defining metrics in SemConv along with implementing them, so I would expect that SemConv's stability will be at least equal to the stability that the metric defines in the component. Hence setting development as the baseline to match with what SemConv project does would feel more correct to me. Otherwise we might end up having alpha metrics in components which are, still, in development in SemConv.

beta: once metrics have been reviewed and adhere to semconv (as necessary, i don't know how this would be enforced for all components, maybe a requirement to provide a weaver schema)

Agreed. There is a proposal for using weaver in Contrib: open-telemetry/opentelemetry-collector-contrib#40724
However, I'm not sure if weaver can help with that validation right now. Probably we'll need to validate this in mdatagen directly until we can switch to weaver. There is an issue for linking metrics to semconv definitions already at #13297 and a draft PR at #13920. If there is a better way to achieve this right now I'd be happy to adopt.

stable: after X number of beta releases a metric can be moved to stable

I think moving it to stable should also require explicitly having a reference to a stable SemConv, right?

note: there is a proposal for using weaver in contrib's scrapers: open-telemetry/opentelemetry-collector-contrib#40724

[codeboten]

Otherwise we might end up having alpha metrics in components which are, still, in development in SemConv.

I guess starting with development is ok. I guess for metrics that will be part of the semantic conventions, then starting at the same level is ok, and for those metrics that will not be in semconv (as i suspect many will be for very application specific components) they can be moved to alpha.

Maybe development is different from alpha in that at the development stage, the metric has not been validated against any semantic conventions, and at alpha, they're either in the semantic conventions at the alpha stage or not in the conventions at all? That seems a bit strange as well.

I wonder if coupling the stage of stability for a metric emitted by collector components to semantic conventions is more confusing. As an end user of the collector, i don't necessarily care about the stage of a metric in semconv, i just want to know that my metric and the alerting i setup wont break from one version of the collector to the other.

[ChrsMark]

I wonder if coupling the stage of stability for a metric emitted by collector components to semantic conventions is more confusing. As an end user of the collector, i don't necessarily care about the stage of a metric in semconv, i just want to know that my metric and the alerting i setup wont break from one version of the collector to the other.

From my perspective I see SemConv as the ultimate place to define a metric/attribute in OpenTelemetry. If for example we allow stable metrics in the Collector that are not defined at all in SemConv how we can be sure about their quality and that we won't diverge from other OTel projects? In this, I think SemConv stability should always be at least equal to the stability that a Collector metric implements. Some examples:

1. process.cpu.time in the Collector is in development while process.cpu.time in SemConv is stable. Collector can safely mark the metric as stable since it has gone through SemConv validation.
2. metric.a.b.c in the Collector is stable while not defined at all in SemConv. Then project X from within OTel comes to SemConv and wants to define metric.a.b.c but in order for this be aligned with SemConv guidelines, it needs be renamed to metric.a.b.c.foo. In this case we will have 2 different names/definitions for the same metric across OTel which should not happen.

As far as I've understood, that's the reason we have stability WGs in SemConv like the System and K8s ones but let me know if I miss anything here.

[codeboten]

If for example we allow stable metrics in the Collector that are not defined at all in SemConv how we can be sure about their quality and that we won't diverge from other OTel projects?

That's a great question, but I don't know that we'll ever be able to make that guarantee for all components in contrib. If we say that no metric can ever be marked as stable if they're not defined in the semantic conventions, then we are requiring all code owners to push semantic conventions that possibly no one will care about and i'm not sure if we want to have semantic conventions for every possible application defined in the otel semantic conventions. Or at least, this seems like it's covering more than what was intended by semantic conventions in my understanding.

As an example, the libhoney receiver, which currently doesn't emit any metrics, but if it did, i certainly wouldn't expect that the metric would need to be defined in the semantic conventions for them to be marked as stable by the owners of that component, so that end users would know that they can rely on them.

In this case we will have 2 different names/definitions for the same metric across OTel which should not happen.

Right this is why i was wondering if tying the stability in the collector to semantic conventions is more confusing. In the sense that it's possible for a metric in a future semantic convention, cpu.time is changed to a unit of years (its a bad example but that's fine), if that happens, do we need a way for the collector metric to say it is only aligned w/ a certain version of semantic conventions?