Support for multiple and arbitrary gateways

[maxgio92]

Details

Describe the solution you'd like:

Add support for having multiple pod-gateways inside a single cluster and allow the end-user, to select for each client application which pod-gateway set as the default gateway for the client application's pods.

Anything else you would like to add:

Implementation

Gateway service name as annotation

At Pod level

In the simpler form, and allowing the application owner - who, in a DevOps way, would be responsible for configuring their own application's deployment - this could be implemented through annotations at pod-level.

The mutating admission wehbook would inject the $gateway environment variable to client pods' init and sidecar containers.

You can see a draft implementation here.

At Namespace level

Assuming the following:

• the injection should happen transparently (e.g. as part of a policy enforcement) and managed by an administrator which would choose through which gateway should applications' traffic go through
• Namespaces are objects managed by administrators

an annotation that would hint which gateway to use as Kubernetes Service name, could be applied at Namespace-level.

The mutating admission wehbook would inject the $gateway environment variable to client pods' init and sidecar containers.

Other

Obviously the way the feature could be achieved is not limited to the above proposed approaches.

Additional Information:

The VXLAN CIDR blocks, for each VXLAN static network (one per gateway) might be able to overlap and this shoudn't be a problem.

This is just a screenshot that shows client pods on the left panes, and gateway pods on the right panes.
The top panes represent a tunnel separated from the bottom one.

The client of which the shells on the left, knows from the $gateway environment variable, from the headless Kubernetes Service, the pod-gateway pod-network's IP (from which the init container build the Linux bridge FDB rule):

$ kubectl get pods -n client -o=jsonpath='{.items[*].spec.initContainers[*].env[0]}' | jq                                                                                                           
{
  "name": "gateway",
  "value": "pod-gateway-a.gateway-system.svc.cluster.local"
}
{
  "name": "gateway",
  "value": "pod-gateway-b.gateway-system.svc.cluster.local"
}

pod-gateway-a and pod-gateway-b are the Kubernetes Service object names that have the pod-gateway-a and pod-gateway-b deployment with one replica, residing in the gateway-system Namespace:

$kubectl get svc -n gateway-system -o wide                                                                                                                                                         
NAME                  TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE   SELECTOR
pod-gateway-a         ClusterIP   None            <none>        4789/UDP   20h   app.kubernetes.io/instance=pod-gateway-a,app.kubernetes.io/name=pod-gateway
pod-gateway-b         ClusterIP   None            <none>        4789/UDP   20h   app.kubernetes.io/instance=pod-gateway-b,app.kubernetes.io/name=pod-gateway
pod-gateway-webhook   ClusterIP   10.96.119.198   <none>        8080/TCP   21h   app.kubernetes.io/instance=pod-gateway,app.kubernetes.io/name=pod-gateway-webhook

$ kubectl get ep -n gateway-system                                                                                                                                                                  
NAME                  ENDPOINTS             AGE
pod-gateway-a         192.168.228.11:4789   20h
pod-gateway-b         192.168.228.14:4789   20h
pod-gateway-webhook   192.168.228.9:8080    21h

$ kubectl get deploy -n gateway-system -o wide                                                                                                                                                      
NAME                  READY   UP-TO-DATE   AVAILABLE   AGE   CONTAINERS    IMAGES                                                                    SELECTOR
pod-gateway-a         1/1     1            1           20h   pod-gateway   ghcr.io/k8s-at-home/pod-gateway:v1.6.1                                    app.kubernetes.io/instance=pod-gateway-a,app.kubernetes.io/name=pod-gateway
pod-gateway-b         1/1     1            1           20h   pod-gateway   ghcr.io/k8s-at-home/pod-gateway:v1.6.1                                    app.kubernetes.io/instance=pod-gateway-b,app.kubernetes.io/name=pod-gateway
pod-gateway-webhook   1/1     1            1           21h   webhook       quay.io/maxgio92/gateway-admission-controller:v3.7.1-alpha-multigateway   app.kubernetes.io/instance=pod-gateway,app.kubernetes.io/name=pod-gateway-webhook

As you can see the gateway VXLAN IPs overlap (i.e. 172.16.0.1) but the Linux FDB together with VXLAN capabilities of the device (drivers) manage for dedicated MAC addresses.

More: multiple clients per each gateway and IP management

In the example scenario of the screenshot above, client VXLAN IP addresses have been statically assigned.
DHCP, of which server and client setup already provided by the pod-gateway feature set, can manage them.

[kalik1]

Great Job! Thank you!

[angelnu]

Thanks for the proposal @maxgio92 - very well formulated!

It is currently possible to deploy multiple gateways but each of the would be bound to a particular namespace or using different labels/annotations (see helm chart).

I would like to keep compatibility with todays annotations. This might be achieved by adding ´setGatewayAnnotationValue´ in addition to the existing setGatewayAnnotation. When setGatewayAnnotationValue is set, then the annotation value would also need to match beyond just being true. This would allow to pod-gateways to use the same setGatewayAnnotation string but different setGatewayAnnotationValue values. For simatry we should also do the same extension for labels.

[maxgio92]

Sure, let us go point by point:

• does that mean that by default setGatewayAnnotationValue would be the current default one ("setGateway"), am I right?

the default value for setGatewayAnnotationValue would be 'true'.

• It means that in order to deploy multiple gateways in a single cluster it should run one deployment of webhook - and related configuration - per specific gateway (each configured with a specific setGatewayAnnotationValue), doesn't it?

almost - it requires different setGatewayAnnotation values and that the annotation is set to true

If so, I think that it doesn't scale properly, both from a performance and an operational efficiency point of view - considering a use case where 100 gateways are needed any Pod creation API request in the cluster would require 100 admission requests, only for the pod-gateway installations.

It would require 100 admission controller containers + 100 gateways. This could be reduced to "just" 1 admision controller + 100 gateways but it could lead to have to support multiple configurations in the same admission controller. At the minimum the gateway IP would be different but excluded routed networks or the NATted ports might be different as well.

From an operational perspective, that would means to keep one Helm release (with one mutating webhook Deployment) per gateway.

Right, this is how others do it currently as they "only" have 2 or perhaps 3gateways and not 100.

• Furthermore, as you already mentioned - the number of gateways is limited to one per Namespace, as it requires one headless Service and this way its name would be kept static ("pod-gateway").

The name of the gateway deployment matches the name of the helm instance. Therefore it should be possible to deploy multiple instances in the same namespace.

---

So in summary: you are right that some resources might be saved by having a common admission controller for multiple gateways but this would come with the trade off of more complicated settings. So if you have an use case for it I would not mind considering a PR but if you are only planning a few it might not be worth the effort.

[maxgio92]

Thank you @angelnu (I think you put your answer inside my comment :/).

I think we can go this way: I can open a PR with the simple implementation you proposed which has the minor impact, as of now.
From the end-user perspective additionals admission controller's flags will be available:

• --setGatewayLabelValue
• --setGatewayAnnotationValue

in order to, not only require that the label/annotation of which the key is specified by the flag --setGatewayLabel/--setGatewayAnnotation, or the default 'setGateway', has a value of 'true', but that it matches the value specified by the flag --setGatewayLabelValue/--setGatewayAnnotationValue.

For example, in order to have multiple gateways:

// run gateway "foo" deployment
gateway-admission-controller [...] --setGatewayAnnotation=inject-gateway --setGatewayAnnotationValue=foo
kubectl run [...] --annotations="inject-gateway=foo"

// run gateway "bar" deployment
gateway-admission-controller [...] --setGatewayAnnotation=inject-gateway --setGatewayAnnotationValue=bar
kubectl run [...] --annotations="inject-gateway=bar"

WDYT?

[angelnu]

Sorry for commenting in directly in your post and not in a copy x-/

I agree with your proposal. So please go ahead. The helm chart will also need an update to expose the new parameter.

[maxgio92]

No problem at all @angelnu. Thanks. PTAL at #103 :-)

[angelnu]

Thanks for the PR! Would you be able to do a PR to Helm chart as well?

[maxgio92]

Thank you @angelnu. Sure, I'll take care of it

[maxgio92]

Hi @angelnu, here it is :-)