Set up discovery service on community infra

[justinsb]

The e2e tests require the OIDC issuer to be resolvable from within a Pod. In kOps we previously met this requirement for testing with a shared publicly readable S3 bucket, but this is problematic in the new infrastructure.

We are prototyping a public discovery service in kOps, which will allow clients to generate an RSA key pair, and effectively authenticate (via mTLS) and register the required OIDC data. Then we will serve this (very small) data to clients publicly.

A version of the service is currently deployed on https://discovery.kubedisco.com, but it would probably be better for the e2e tests if it was deployed on community infrastructure.

The manifest for the current deployment is here: https://github.com/kubernetes/kops/blob/master/discovery/k8s/manifest.yaml

Currently the data is stored in process (and is lost every time a pod restarts). We probably want to move it to etcd and etcd-operator ("all the wood behind one arrow")

[justinsb]

cc @ameukam

[ameukam]

@upodroid Do we separate cluster if we need to run the Etcd Operator or the utility cluster can be used ?

[upodroid]

We can use the utility cluster for this purpose.

The cluster has Istio and GKE Dataplanev2 enabled so we can isolate the etcd deployment from other workloads in the cluster.

Is kops planning on offerring this widely to everyone or is it a pure kube internal service(other than red tests we won't have issues with downtime)?

[justinsb]

We can use the utility cluster for this purpose.

Awesome!

The cluster has Istio and GKE Dataplanev2 enabled so we can isolate the etcd deployment from other workloads in the cluster.

Cool. etcd-operator either has or will soon have TLS certificate management, so the risk is pretty minimal, but still nice to have NetworkPolicy and friends. And if GKE doesn't support SNI routing I can put in the feature request :-)

Is kops planning on offerring this widely to everyone or is it a pure kube internal service(other than red tests we won't have issues with downtime)?

So I think we want to make it available for tests first. I don't think we should stop people using it more broadly, but we should make clear (somewhere) that the service is provided temporarily / as-is / no guarantees etc. We

[ameukam]

@justinsb is there an OCI image for etcd-operator ? I couldn't find one

[ameukam]

/kind feature
/priority important-longterm
/area infra/gcp
/milestone v1.36