[HA] Enable leader election and make it safe for multiple replicas

[mikouaj]

Community Note

• Please vote on this issue by adding a 👍 reaction
  to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do
  not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment.

Description

The leader election is currently disabled by default in Kube Startup CPU Boost.
This should be changed to allow safe increase of replicas for HA purposes.
Operator's logic should be revised to allow operation of multiple replicas.

References

• kube-startup-cpu-boost should deploy with PDB #96

[matthenry87]

Thanks for making that other modification so quickly (starting the fixed duration when the pod is scheduled instead of when created).

Part of our use-case for a controller like this is during cluster upgrades and disaster recovery excersizes, meaning that this controller will most definitely be disrupted. I started to create my own implementation in Java which is why I am so intimately close to this at the moment.

My current plan is (will ditch if your controller meets our needs) that only the leader will be subscribed to pod events, and will hold the current state of all 'CPUBoost' resources in the cluster. If it sees a pod update that has a node assignment, that pod matches the label of a CPUBoost currently in existence, and we haven't already acted on that specific pod, then we will schedule a Job pod to sleep ~90s and then fire a curl request to drop the CPU back down. This leader holds the state of which pods it has acted on along with timestamps that will let us purge these records after x time.

Sorry to comment with details of my planned implementation ionstead of simply leaving the thumbs up, but our K8s environments are constantly plauged with CPU spikes and pinned nodes and I really need something that is HA and resilient soon. With only 3 thumbs ups and given the decent amount of effort required here, will this not be worked on anytime soon?

[mikouaj]

Thanks @matthenry87 for sharing your ideas.
The controller already supports leader election (thanks to the kube builder framework it is based on). The leader election is also enabled by default. I've also did few improvements (i.e. #112, #114) that should make failover working smoothly. This ticket was created mainly to test it and fix any bugs.
After increasing the number of controller replicas to > 1, the following should happen:

• One replica will become a leader, remaining replicas will be followers
• The followers won't be ready due to health/readiness probe to avoid any webhook calls
• On a failover, one of the followers will become new leader and will become ready
• New leader will receive events about StartupCPUBoosts and Pods that were boosted
• The boost manager component will match boosts and pods building its in-mem state
• The normal operation will continue

I will validate the above but due to vacation period it will take be some time. You are also welcome to perform your tests in a meantime and share observations.

[padoffe]

Hello @mikouaj,

I've reviewed this issue along with the related one described in #126. My team is experiencing the same situation.

When the leader election feature is enabled, we're unable to perform rollout restarts or migrate the controller to another node. Since the liveness probe fails as long as a controller instance is already running, Kubernetes never acknowledges the new pod as healthy. This is particularly problematic as it prevents any rollout restarts (we performed several during the implementation phase to adjust requests/limits). We have to manually scale the deployment down to 0 and then back to 1 for the rollout process to complete successfully.

In my opinion, the leader election process should not impact the liveness probe. Otherwise, it becomes impossible to maintain the same number of healthy pods as the requested replica count, which doesn't align with best practices.

We've disabled this feature for now since a single replica is sufficient for our current needs. However, I don't believe this is a viable approach for production environments.

[mikouaj]

@padoffe thanks for rising this up, I agree - unlike readiness probe, the liveness probe should not be affected.
I will check this as liveness probe is actually configured with controller-manager's Ping checker that returns always true.

Just note that non-leader POD won't be healthy due to readiness probe - this is intentional. The controller-manager in such POD waits for a leader lease and is not able to handle requests from mutating webhook.

[padoffe]

@mikouaj regarding the liveness probe - I apologize for the confusion. I was actually referring to the readiness probe.

I understand this is intentional. However, I'm surprised by this implementation choice, as it appears to conflict with several Kubernetes best practices. As mentioned in my original post, the current rollout configuration combined with this design prevents a simple rollout procedure from completing successfully.

We are also using GCP spot instances. In the event of a node replacement, our scheduler attempts to gracefully migrate existing pods to a new node. However, this cannot be completed for the kube-startup-cpu-boost controller, as the new pod never reaches a healthy state.

[mikouaj]

@padoffe I agree that the current approach to HA in kube-startup-cpu-boost is not optimal. Unlike many kube-builder based operators, the kube-startup-cpu-boost runs stateful mutating webhook. Because of that, having more than one replicas ready (from probe perspective) when only one replica has functional webhook service means that mutation (boosting) won't work consistently. Thats why I decided to disable readiness on non-leader replicas.

Fixing the above require bigger redesign. The webhook service part of the operator needs to became separate operator with own deployment. It will remain stateful but, since it only reads boost configuration, can work in active-active mode (no leader election). The current / core part won't run webhook service anymore, will still use leader-election and it will be safe to relax the readiness probes.

[matthenry87]

I did think the leader behavior was odd. As a K8s admin, I don't like to see any pods in a Not Ready state.

In my Java implementation, all of the state is stored in annotations on the pods I act on. The leader is the only one that has a watch on the pods in order to drop their CPU back down after the timer is up (which it sets when it first sees it get scheduled).

All 3 of my replicas are members of the mutating admission controller, because it doesn't matter which one boosts the CPU - it puts the original CPU request value and a timestamp in the annotations. The leader watches for pod scheduling and schedules the thread to drop the CPU down. If it gets killed, my leader also has a cleanup process to find orphans that never got CPU dropped back down.

[mikouaj]

@matthenry87 the webhook service needs to read the boost configuration CRDs to make decisions - thats the state I was referring too. This means that controller-manager has to watch and reconcile those CRDs. The remaining part (the real state) is implemented with labels and annotations.