Deflaking e2e tests (#3820)

lsierant · fealebenpae · commit a34db6042381 · 2024-09-30T11:22:22.000+02:00
This patch contains a number of improvements tried in order to deflake e2e tests. Overally result is quite successful: [100% green EVG](https://spruce.mongodb.com/version/66f81d4d24506900071e3563/tasks?sorts=STATUS%3AASC%3BBASE_STATUS%3ADESC) and progress with [flakes](https://ui.honeycomb.io/mongodb-4b/environments/production/result/8p2zFkfJ2Pz) Changes: - deleting kind clusters before creating kind clusters; controllable by new DELETE_KIND_NETWORK env, which is true only in EVG run (set in evg-private-context only). This one is actually of a dubious value as we're doing docker prune in teardown already. But it's useful also for our own evg hosts as we've never cleared docker network once created. - setting the subnet for docker network explicitly to 172.18.0.0/16. We hardcode usage of this subnet locally, and docker network in evg started to assign subnets on different IPs. That's why sometimes our "interconnected" tests were passing - when docker network created in evg got lucky subnet (172.18)! - added ttl and refresh to coredns config map - cannot harm, not sure if helps - changed the order of CreateOrUpdate to always update first and if 404 then create. Previously we had get first which was susceptible to not refreshed informers and CreateOrUpdate was going directly to Create which was failing. Details in [PR in MCO](mongodb/mongodb-kubernetes-operator#1623)
diff --git a/docker/mongodb-enterprise-tests/tests/conftest.py b/docker/mongodb-enterprise-tests/tests/conftest.py
@@ -1366,6 +1366,8 @@ def coredns_config(tld: str, mappings: str):
     debug
     hosts /etc/coredns/customdomains.db   {tld} {{
 {mappings}
+       ttl 10
+       reload 1m
        fallthrough
     }}
 }}
diff --git a/docker/mongodb-enterprise-tests/tests/multicluster/multi_cluster_tls_no_mesh.py b/docker/mongodb-enterprise-tests/tests/multicluster/multi_cluster_tls_no_mesh.py
@@ -212,7 +212,7 @@ def test_create_mongodb_multi(
     member_cluster_clients: List[MultiClusterClient],
     member_cluster_names: List[str],
 ):
-    mongodb_multi.assert_reaches_phase(Phase.Running, timeout=2400)
+    mongodb_multi.assert_reaches_phase(Phase.Running, timeout=2400, ignore_errors=True)
 
 
 @mark.e2e_multi_cluster_tls_no_mesh
diff --git a/go.mod b/go.mod
@@ -11,7 +11,7 @@ require (
 	github.com/hashicorp/go-retryablehttp v0.7.7
 	github.com/hashicorp/vault/api v1.14.0
 	github.com/imdario/mergo v0.3.15
-	github.com/mongodb/mongodb-kubernetes-operator v0.11.1-0.20240821171907-8e1a37b82521
+	github.com/mongodb/mongodb-kubernetes-operator v0.11.1-0.20240930083228-a6075d415434
 	github.com/pkg/errors v0.9.1
 	github.com/prometheus/client_golang v1.19.1
 	github.com/r3labs/diff/v3 v3.0.1
diff --git a/go.sum b/go.sum
@@ -143,6 +143,10 @@ github.com/modern-go/reflect2 v1.0.2 h1:xBagoLtFs94CBntxluKeaWgTMpvLxC4ur3nMaC9G
 github.com/modern-go/reflect2 v1.0.2/go.mod h1:yWuevngMOJpCy52FWWMvUC8ws7m/LJsjYzDa0/r8luk=
 github.com/mongodb/mongodb-kubernetes-operator v0.11.1-0.20240821171907-8e1a37b82521 h1:vpqPRUa6NorJ5q4BqyPHWUK4qO6K4om4uMJUXybQEXk=
 github.com/mongodb/mongodb-kubernetes-operator v0.11.1-0.20240821171907-8e1a37b82521/go.mod h1:2w08xx8V5vHfD+nrKp+DtHxOVrTZ/G+ebm0I8Q4faXo=
+github.com/mongodb/mongodb-kubernetes-operator v0.11.1-0.20240928135849-f79c8434fc8c h1:WQdcKGXKlqHqPaQkJeP5aQPgR6KwD+yiDWQU1yYFnRo=
+github.com/mongodb/mongodb-kubernetes-operator v0.11.1-0.20240928135849-f79c8434fc8c/go.mod h1:2w08xx8V5vHfD+nrKp+DtHxOVrTZ/G+ebm0I8Q4faXo=
+github.com/mongodb/mongodb-kubernetes-operator v0.11.1-0.20240930083228-a6075d415434 h1:xCKFXj5J2VcSRREC/H6JoAl5Iv75E5H90NtBOX6Dmvk=
+github.com/mongodb/mongodb-kubernetes-operator v0.11.1-0.20240930083228-a6075d415434/go.mod h1:2w08xx8V5vHfD+nrKp+DtHxOVrTZ/G+ebm0I8Q4faXo=
 github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 h1:C3w9PqII01/Oq1c1nUAm88MOHcQC9l5mIlSMApZMrHA=
 github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822/go.mod h1:+n7T8mK8HuQTcFwEeznm/DIxMOiR9yIdICNftLE1DvQ=
 github.com/nxadm/tail v1.4.4/go.mod h1:kenIhsEOeOJmVchQTgglprH7qJGnHDVpk1VPCcaMI8A=
diff --git a/scripts/dev/contexts/evg-private-context b/scripts/dev/contexts/evg-private-context
@@ -98,3 +98,5 @@ export BUILD_VARIANT="${build_variant:-"unknown"}"
 # var used in pipeline.py to determine if we're running on EVG host
 # used to discern between local pipeline image build and the build in pipeline
 export RUNNING_IN_EVG="true"
+
+export DELETE_KIND_NETWORK="true"
diff --git a/scripts/dev/evg_host.sh b/scripts/dev/evg_host.sh
@@ -112,7 +112,6 @@ recreate-kind-cluster() {
   get-kubeconfig
 }
 
-
 tunnel() {
   shift 1
   # shellcheck disable=SC2016
diff --git a/scripts/dev/recreate_kind_cluster.sh b/scripts/dev/recreate_kind_cluster.sh
@@ -3,12 +3,17 @@
 set -Eeou pipefail
 
 source scripts/dev/set_env_context.sh
+source scripts/funcs/kubernetes
 
 cluster_name=$1
 if [[ -z ${cluster_name} ]]; then
   echo "Usage: recreate_kind_cluster.sh <cluster_name>"
   exit 1
 fi
 
+if [[ "${DELETE_KIND_NETWORK:-"false"}" == "true" ]]; then
+  delete_kind_network
+fi
+
 scripts/dev/setup_kind_cluster.sh -r -e -n "${cluster_name}" -l "172.18.255.200-172.18.255.250" -c "$CLUSTER_DOMAIN"
 CTX_CLUSTER1=${cluster_name}-kind scripts/dev/install_csi_driver.sh
diff --git a/scripts/dev/recreate_kind_clusters.sh b/scripts/dev/recreate_kind_clusters.sh
@@ -2,9 +2,14 @@
 set -Eeou pipefail
 
 source scripts/dev/set_env_context.sh
+source scripts/funcs/kubernetes
 
 kind delete clusters --all
 
+if [[ "${DELETE_KIND_NETWORK:-"false"}" == "true" ]]; then
+  delete_kind_network
+fi
+
 # first script prepares registry, so to avoid race it have to finish running before we execute subsequent ones in parallel
 # To future maintainers: whenever modifying this bit, make sure you also update coredns.yaml
 scripts/dev/setup_kind_cluster.sh -n "e2e-operator" -p "10.244.0.0/16" -s "10.96.0.0/16" -l "172.18.255.200-172.18.255.210" -c "$CLUSTER_DOMAIN"
diff --git a/scripts/dev/setup_kind_cluster.sh b/scripts/dev/setup_kind_cluster.sh
@@ -57,10 +57,12 @@ shift "$((OPTIND - 1))"
 
 kubeconfig_path="$HOME/.kube/${cluster_name}"
 
-# create the kind network early unless it already exists.
-# it would normally be created automatically by kind but we
-# need it earlier to get the IP address of our registry.
-docker network create kind || true
+# We create docker network primarily so that all kind clusters use the same network.
+# We hardcode 172.18/16 subnet in few places, so we must ensure the network uses that subnet:
+#   - we set IP ranges for MetalLB
+#   - we write into coredns config map with IP used for exposing pods externally and for no-mesh test
+# In case of any errors, verify if any other subnet is not already using that subnet.
+docker network create --subnet=172.18.0.0/16 kind || true
 
 # adapted from https://kind.sigs.k8s.io/docs/user/local-registry/
 # create registry container unless it already exists
diff --git a/scripts/evergreen/setup_kubernetes_environment.sh b/scripts/evergreen/setup_kubernetes_environment.sh
@@ -2,6 +2,7 @@
 set -Eeou pipefail
 
 source scripts/dev/set_env_context.sh
+source scripts/funcs/kubernetes
 
 # shellcheck disable=SC2154
 bindir="${workdir}/bin"
diff --git a/scripts/funcs/kubernetes b/scripts/funcs/kubernetes
@@ -173,3 +173,22 @@ reset_namespace() {
   echo "Removing CSRs"
   kubectl --context "${context}" delete "$(kubectl get csr -o name | grep "${NAMESPACE}")" &> /dev/null || true
 }
+
+delete_kind_network() {
+  if ! docker network ls | grep -q "kind"; then
+     echo "Docker network 'kind' does not exist."
+     return 0
+  fi
+  echo "Docker network 'kind' exists."
+
+  # Stop all containers in the "kind" network
+  containers=$(docker network inspect -f '{{range .Containers}}{{.Name}} {{end}}' kind)
+  if [[ -n "${containers}" ]]; then
+      echo "Stopping containers ${containers} using kind network..."
+      # shellcheck disable=SC2086
+      docker stop ${containers}
+      docker container prune -f
+  fi
+
+  docker network rm kind
+}

Original file line number	Diff line number	Diff line change
`@@ -112,7 +112,6 @@ recreate-kind-cluster() {`
`112`	`112`	`get-kubeconfig`
`113`	`113`	`}`
`114`	`114`
`115`		`-`
`116`	`115`	`tunnel() {`
`117`	`116`	`shift 1`
`118`	`117`	`# shellcheck disable=SC2016`