feat: add the option to disable api server registration for metrics-server

[raphapr]

Add the metricsServer.registerAPIService option to disable external metrics API registration.

This is useful when you already have an API registration (e.g. Datadog Cluster Agent) and want to deploy KEDA without impacting the current external metrics API.

Checklist

• I have verified that my change is according to the deprecations & breaking changes policy
• Commits are signed with Developer Certificate of Origin (DCO - learn more)
• README is updated with new configuration values (if applicable) learn more
• A PR is opened to update KEDA core (repo) (if applicable, ie. when deployment manifests are modified)

[zroubalik]

KEDA Metrics Server is mandatory to run KEDA, you cannot use the one that's shipped with Datadog for example. The implementation is different

[raphapr]

@zroubalik I understand that is mandatory. My rationale is there are potential scenarios where deploying KEDA without registering its metrics server becomes useful for migration purposes.

For example, in my case, I manage a cluster in which 1beta1.external.metrics.k8s.io is pointed to datadog; so deploying KEDA without deregistering the actual metrics provider is desired to facilitate the creation of the KEDA objects - I want to migrate the datadogmetric to scaledobject first. And subsequently, I can switch the API by using metricsServer.registerAPIService.

Does that make sense?

[JorTurFer]

Does that make sense?

Yes, it can make sense IMO.

I guess that we could accept that feature now because I'll give another try to the proxy for supporting multiple metric servers (there have been some poeple reluctant to the subdomain idea in control plane side), so probably we will have to integrate it sooner or later and it also allows using KEDA only for ScaledJobs

Of course, this is something that requires a good explanation in docs, but I'd say that we could continue

WDYT?

[zroubalik]

I understand your use case, it is a rare one though. I am not extremely happy with this addition, but I won't block it 😄 I let others to decide.

My only requirement is that this is properly documented, with consequencies for users (if they enable this setting).

[JorTurFer]

I've been thinking about this and I'm not totally sure. Even though I understand the use case, I'm not sure if we are in the moment for supporting this feature as the proxy is still an on-going idea.
I won't block it either but it has to be really well documented with the side effects of this. @tomkerkhove ?