Add jobLabel to annotation autodiscovery

[rubensf]

It was missing.

[CLAassistant]

All committers have signed the CLA.

[petewall]

Honestly, I don't know if this makes sense as the right job label. I think a more appropriate job label might be the first non-empty value:

• k8s.grafana.com/job annotation
• app.kubernetes.io/name label
• app label
• Service name (for annotated Services)
• Container name (for annotated Pods)

[rubensf]

Sorry I missed your notification here - we do have that jobLabel as an option across a bunch of other spots, such as everything under clusterMetrics. Why not add that as an option here?

Strictly, we already have a annotation check that defaults to k8s.grafana.com/job under annotations.job value:

k8s-monitoring-helm/charts/k8s-monitoring/charts/feature-annotation-autodiscovery/values.yaml

Line 32 in 9536973

job: "k8s.grafana.com/job"

[rubensf]

Just updated the PR to make it optional :)

[rubensf]

ping @petewall

[rubensf]

ping @petewall @nevermind89x @skl . Sorry for the spam!

[petewall]

Like I mentioned in my previous comment, this change as it is now it not how I think this should be implemented.

Before this change, the job label is set:

• if the user has the k8s.grafana.com/job annotation
• Else, the default value for the prometheus.scrape component (.e.g. prometheus.scrape.annotation_autodiscovery_http or something)

I agree that the default job label is not excellent and we can improve this.

Your PR would always set the job label to .annotationAutodiscovery.jobLabel if it's set, overriding the k8s.grafana.com/job annotation. I believe that this is not what users will expect.

The correct method would be check in this order:

1. k8s.grafana.com/job annotation
2. app.kubernetes.io/name label
3. app label
4. Service name (for annotated Services)
5. Container name (for annotated Pods)

And skip the jobLabel altogether. Why this order? The Job label is quite important and is used to identify the different workloads. In our case, the workloads are not distinct based on the fact that they're scraped via annotations, but on something the user defines (k8s.grafana.com annotation, or app labels) or on our best effort (container name or service name).

Also, why container name over pod name? because Pod names are often appended with random characters and are ephemeral. Container names are much more consistent and an appropriate default.

[petewall]

So, after all that, what's the course of action? If you want to take this on, find the Alloy rule where we set job from the k8s.grafana.com/job annotation. Then make copies of that rule for the other options, but checking if job was not yet set. e.g.:

rule {
  source_labels = ["job", "__meta_kubernetes_pod_label_app_kubernetes_io_name"]
  regex = ";(.+)"
  target_label = "job"
}

This will only set job to app.kubernetes.io/name if the job label is not currently set (because of nothing before the default separator ;).

[petewall]

closing in favor of #1717