Added details about how to use Submariner for cross cluster communication

aswinayyolath · aswinayyolath · commit b46b29f010c8 · 2024-11-15T11:56:20.000+05:30
Added details about how to use Submariner for cross cluster communication Contributes to: strimzi#129 Signed-off-by: Aswin A <aswin6303@gmail.com>
diff --git a/083-stretch-cluster.md b/083-stretch-cluster.md
@@ -128,14 +128,178 @@ The Kafka reconciler will identify all target clusters from KafkaNodePool resour
 This will ensure that even if the central cluster experiences an outage, external clients can still connect to the stretch cluster and continue their operations without interruption.
 
 #### Cross-cluster communication
-Kafka controllers/brokers are distributed across multiple Kubernetes environments and will need to communicate with each other.
-Currently, the Strimzi Kafka operator defines Kafka listeners for internal communication (controlplane and replication) between brokers/controllers (Kubernetes services using ports 9090 and 9091).
-The user is not able to influence how these services are set up and exposed outside the cluster.
-We would remove this limitation and allow users to define how these internal listeners are configured in the Kafka resource, just like they do for Kafka client listeners.
-
-Users will also be able to override listener configurations in each KafkaNodePool resource, if the listeners need to be exposed in different ways (ingress host names, Ingress annotations etc.) for each Kubernetes cluster.
-This will be similar to how KafkaNodePools are used to override other configuration like storage etc.
-To override a listener, KafkaNodePool will define configuration with same listener name as in Kafka resource.
+
+#### Cross-Cluster Communication Using Submariner
+
+For Kafka brokers and controllers distributed across multiple Kubernetes clusters, cross-cluster communication is essential.
+Submariner is a tool that facilitates this type of communication by connecting different Kubernetes clusters through secure networking, allowing data transfer without relying solely on traditional methods such as LoadBalancers or Ingresses.
+
+#### Current Limitations
+
+The Strimzi Kafka operator currently sets up Kafka listeners for internal communication between brokers and controllers within a single Kubernetes cluster.
+These services are typically defined as headless services, accessible only within the cluster’s network (e.g., `my-cluster-broker-0.my-cluster-kafka-brokers.svc.cluster.local`).
+However, these internal addresses are not accessible outside the cluster, 
+while Kubernetes provides native solutions like Ingress to expose services externally, these solutions can be slower
+and may introduce latency due to the additional routing overhead. Submariner offers a more efficient alternative by 
+enabling direct communication between clusters through secure IP routing.
+
+
+#### How Submariner Facilitates Cross-cluster Communication
+
+When multiple Kubernetes clusters are connected using Submariner,
+services of type ClusterIP can be exported, making them accessible across participating clusters in the network.
+
+To export a service using Submariner, the following command can be used
+
+```
+subctl export service --kubeconfig <CONFIG> --namespace <NAMESPACE> my-cluster-kafka-brokers
+```
+
+This command creates a ServiceExport resource in the specified namespace,
+signaling Submariner to register the service with the Submariner Broker. 
+The Broker acts as the coordinator for cross-cluster service discovery, 
+utilizing the Lighthouse component to allow services in different clusters 
+to discover and communicate with each other. 
+Submariner sets up tunnels and routing tables, ensuring direct and secure traffic flow between clusters.
+
+Once a service is exported, it becomes accessible through a global DNS name format: `<service-name>.<namespace>.svc.clusterset.local`. 
+This DNS name enables reachability for any cluster in the Submariner deployment. 
+For instance, the `advertised.listener` configuration in a Kafka setup would be updated from `my-cluster-broker-0.my-cluster-kafka-brokers.svc.cluster.local` 
+to `my-cluster-broker-0.cluster1.my-cluster-kafka-brokers.svc.clusterset.local`, where `cluster1` represents the Submariner cluster ID.
+Similarly the `controller.quorum.voters` property also need to be updated to use Submariner exported servicename. 
+This ensures that `advertised.listener` and `controller.quorum.voters` addresses are reachable from any connected cluster.
+
+
+#### Considerations for SSL/TLS Verification
+
+For SSL hostname verification between pods, the Subject Alternative Name (SAN) entries in the certificates must include the FQDNs of the Submariner-exported services. 
+This can be achieved in the following ways:
+
+1. **Defining SANs in the Kafka CR**: Users can specify the Submariner-exported FQDNs in the alternativeNames field within the Kafka CR's listener configuration. 
+For example:
+
+```yaml
+listeners:
+  - name: tls
+    port: 9093
+    type: internal
+    tls: true
+    configuration:
+      bootstrap:
+        alternativeNames:
+          - my-cluster-broker-0.cluster2.my-cluster-kafka-brokers.strimzi.svc.clusterset.local
+          - my-cluster-broker-1.cluster2.my-cluster-kafka-brokers.strimzi.svc.clusterset.local
+          - my-cluster-broker-100.cluster2.my-cluster-kafka-brokers.strimzi.svc.clusterset.local
+          - my-cluster-broker-101.cluster2.my-cluster-kafka-brokers.strimzi.svc.clusterset.local
+```
+
+This ensures that the SANs are included in the certificates provided to each broker. 
+However, this approach may inject all broker FQDNs into every broker's certificate, which is not ideal.
+
+2. **Controller Pods**: Unlike brokers, controller pods do not inherit listener configurations from the Kafka CR and use a single control plane listener (TCP 9090). 
+This requires adding SANs for the controller’s communication manually, which is not currently supported by Strimzi and is not considered an optimal solution.
+
+#### Extending KafkaNodePool Custom Resource for Cross-Cluster Communication
+
+To integrate Submariner as a cross-cluster communication solution in Strimzi, 
+the KafkaNodePool Custom Resource (CR) should be extended to include configuration fields that support various cross-cluster technologies. 
+This section outlines the proposed changes to the KafkaNodePool CR, 
+along with an explanation of their roles in enabling communication across Kubernetes clusters.
+
+#### Proposed Changes to the KafkaNodePool Custom Resource
+To make the KafkaNodePool CR more flexible and future-proof, we propose the following additions
+
+1. **New Cross-Cluster Technology Field**: Introduce a new field called crossClusterTechnology in the KafkaNodePool CR. 
+This field will allow users to specify the technology they wish to use for cross-cluster communication. 
+Initially, Submariner will be supported, but this design accommodates future integration with other technologies.
+
+2. **Submariner-Specific Configuration**: If Submariner is chosen as the cross-cluster technology, users must provide a submarinerClusterId. 
+This identifier is crucial for Submariner, as it uniquely represents each cluster for tunnel creation and communication.
+
+#### Updated KafkaNodePool CR Example
+
+Below is an updated example of the KafkaNodePool CR with the proposed changes
+
+```yaml
+apiVersion: kafka.strimzi.io/v1beta2
+kind: KafkaNodePool
+metadata:
+  name: controller
+  labels:
+    strimzi.io/cluster: my-cluster
+spec:
+  replicas: 3
+  roles:
+    - controller
+  storage:
+    .........
+  crossClusterTechnology:
+    technology: submariner
+    configuration:
+      submarinerClusterId: cluster1
+---
+
+apiVersion: kafka.strimzi.io/v1beta2
+kind: KafkaNodePool
+metadata:
+  name: broker
+  labels:
+    strimzi.io/cluster: my-cluster
+spec:
+  replicas: 3
+  roles:
+    - broker
+  storage:
+    ......
+    crossClusterTechnology:
+      technology: submariner
+      configuration:
+        submarinerClusterId: cluster1
+```
+
+#### Explanation of Changes
+
+1. crossClusterTechnology.technology Field
+
+Purpose: Specifies the cross-cluster communication technology to be used (e.g., submariner, skupper, istio...)
+
+Example: `technology: submariner` indicates that Submariner is the chosen solution for enabling communication across clusters.
+
+2. configuration
+
+Purpose: Contains nested section for each possible technology
+         Only the relevant section is populated based on the technology specified
+
+Example: `crossClusterTechnology.configuration.submarinerClusterId` Field
+
+Defines the unique cluster ID used by Submariner to establish tunnels and identify clusters.
+
+Example: `submarinerClusterId: cluster1` assigns a unique identifier for the cluster involved in cross-cluster communication.
+
+#### Operator's Role in Utilizing the CR Fields
+
+The Strimzi Operator will parse the crossCluster section to identify the chosen technology and apply the relevant configurations
+
+- **Generating advertised.listener and controller.quorum.voter Addresses**: The Operator will use the submarinerClusterId 
+to construct addresses in the format `<broker-name>.<submarinerClusterId>.<namespace>.svc.clusterset.local`. 
+This ensures that `advertised.listener` and `controller.quorum.voters` are accessible across connected clusters, facilitating data replication and leader election.
+
+- **Certificate SANs Configuration**: During the creation of SSL/TLS certificates for brokers and controllers, 
+the Operator must include the Submariner-exposed service addresses in the Subject Alternative Name (SAN) entries. 
+This guarantees that hostname verification succeeds when traffic flows between clusters.
+
+#### Example of Updated advertised.listener and controller.quorum.voters
+
+The Operator will update advertised.listener and controller.quorum.voters configurations as follows
+
+```yaml
+
+advertised.listeners=REPLICATION-9091://my-cluster-broker-0.cluster1.my-cluster-kafka-brokers.strimzi.svc.clusterset.local:9091,PLAIN-9092://my-cluster-broker-0.cluster1.my-cluster-kafka-brokers.strimzi.svc:9092,TLS-9093://my-cluster-broker-0.cluster1.my-cluster-kafka-brokers.strimzi.svc:9093
+
+controller.quorum.voters=101@my-cluster-controller-101.cluster2.my-cluster-kafka-brokers.strimzi.svc.clusterset.local:9090,1@my-cluster-controller-1.cluster1.my-cluster-kafka-brokers.strimzi.svc.clusterset.local:9090
+```
+
+These updates ensure brokers and controllers can be discovered and communicate across clusters without relying on traditional external access methods.
 
 #### Resource cleanup on remote Kubernetes clusters
 As some of the Kubernetes resources will be created on a remote cluster, we will not be able to use standard Kubernetes approaches for deleting resources based on owner references.
