Hello!
I've been testing percona mongodb controller and have a psmdb installation running on AWS EKS using hostPath volumes on instance-store (NVME) based servers.
When changing a replicaset's nodeSelector, eg: node.kubernetes.io/instance-type, it's very easy to destroy a cluster, in that the controller will reschedule pods onto new hosts, and does not appear to wait for replication to complete before rescheduling other members of the replicaset. I have a podDisruption budget with maxUnavailable: 1, but all pods were moved to new hosts before replication could complete, thus destroying my cluster.
Question: What is the recommended way to make changes to replicasets when using hostPath volumes? Preferably, one could change a nodeSelector, and pods would be rescheduled one by one, join the replicaset, begin restoring data, and then only finally go "ready" when replication was complete. Alternatively, a guide on creating a new psmdb installation, joining the clusters and cutting over may be useful (if this is the only path to change nodeSelectors safely).
It certainly seems like this controller is mostly used with external disks (ie: EBS PVCs on AWS) and assumes that data is persistent to a particular replica-set member. Are that any examples of using instance-store in production?
Thank you!
Hello!
I've been testing percona mongodb controller and have a psmdb installation running on AWS EKS using hostPath volumes on instance-store (NVME) based servers.
When changing a replicaset's nodeSelector, eg:
node.kubernetes.io/instance-type, it's very easy to destroy a cluster, in that the controller will reschedule pods onto new hosts, and does not appear to wait for replication to complete before rescheduling other members of the replicaset. I have a podDisruption budget with maxUnavailable: 1, but all pods were moved to new hosts before replication could complete, thus destroying my cluster.Question: What is the recommended way to make changes to replicasets when using hostPath volumes? Preferably, one could change a nodeSelector, and pods would be rescheduled one by one, join the replicaset, begin restoring data, and then only finally go "ready" when replication was complete. Alternatively, a guide on creating a new psmdb installation, joining the clusters and cutting over may be useful (if this is the only path to change nodeSelectors safely).
It certainly seems like this controller is mostly used with external disks (ie: EBS PVCs on AWS) and assumes that data is persistent to a particular replica-set member. Are that any examples of using instance-store in production?
Thank you!