KEP-4781 restarting kubelet does not change pod status

[HirazawaUi]

• One-line PR description: This KEP aims to ensure that restarting the kubelet for a short period does not affect the status of pods on the node.

• Issue link: Fix inconsistent container ready state after kubelet restart #4781

• Other comments:

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: HirazawaUi
Once this PR has been reviewed and has the lgtm label, please assign dchen1107 for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• keps/sig-node/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[HirazawaUi]

I did not refer to the implementation approach of the previous KEP. After reviewing the POC PR related to that KEP, I found the implementation process somewhat cumbersome, and it also presented some potential edge case issues.

After tracing the pod status transition process, I adopted a new implementation method to achieve the goal: consistently relying on the detection results of the probeManager. This approach simplifies the implementation and helps us avoid certain edge cases. And in this section, the behavioral differences of kubelet under several scenarios are also analyzed. Could you please take a look?

My POC PR: kubernetes/kubernetes#133676

@SergeyKanzhelev @thockin

[toVersus]

If the Kubelet restart occurs within the nodeMonitorGracePeriod

Does this mean the case where the kubelet has been down for longer than nodeMonitorGracePeriod and then restarts afterward?

So basically, the Node Lifecycle Controller notices that the Lease hasn’t been updated past nodeMonitorGracePeriod, marks the Node as NotReady, and flips the Pods’ Ready condition to False. After the kubelet restarts, it fetches the Pod info for its own Node from the API server, and the prober manager simply carries over that Ready condition value, right?

[HirazawaUi]

The scenario here indeed warrants a more detailed explanation.

Since we cannot delay waiting for the prober manager to trigger a probe before updating the container status in the syncPod process, the pod status update always occurs before the probe. This means that when we first update the container status, we do not know the actual state of the container.

• For a short kubelet restart, we can confidently assume that the container's state has not changed. Therefore, we can retain the container's state and let the prober manager trigger a probe for the container to correctly update its state in the pod.
• However, for a prolonged kubelet restart, when the node is already in a NotReady state, we can no longer assume that the container's state in the pod remains unchanged. In this case, we follow the previous behavior by setting the container's Ready field to false (as mentioned in the KEP, before applying the changes in this KEP, after a pod is initially added to the prober manager, the probe result is set to an initial value, and the initial value for the readiness probe is Failure, which sets the container's Ready field to false). Then, the probe is performed, and the container's state is correctly updated.

In summary:

• For a short kubelet restart, we choose to inherit the container's state from before the kubelet restart.
• For a prolonged kubelet restart, we follow the pre-change behavior by first setting the container's Ready field to false, waiting for the actual probe result, and eventually driving the container's state to its actual value. However, compared to the pre-change behavior, this approach still has an advantage: it avoids unnecessary state transitions for the container's Ready field (from true -> false -> true) and instead transitions directly from false -> true. This prevents meaningless state flapping, reducing unnecessary reconciliation work for various controllers that watch container status, such as the EndpointSlice controller or external controllers that depend on EndpointSlice, thereby alleviating their workload.

[toVersus]

Thanks for the clear explanation!

For a prolonged kubelet restart, we follow the pre-change behavior by first setting the container's Ready field to false, waiting for the actual probe result, and eventually driving the container's state to its actual value.

I think I finally understand the part I was a bit unclear about regarding how the UPDATE PodStatus step in your diagram determines readiness. If the kubelet is down longer than nodeMonitorGracePeriod, the container’s ready condition is set to false. In that case, in your PoC, the section below is where the ready state becomes false, right?
https://github.com/kubernetes/kubernetes/blob/d207ce94fe550ec35ff6a6b120faf759b8cb9fae/pkg/kubelet/prober/prober_manager.go#L336-L339

[HirazawaUi]

yes.

[thockin]

Can we include the history of this?

kubernetes/kubernetes#100277
kubernetes/kubernetes#100277 (comment)
kubernetes/kubernetes#102367

[thockin]

I am strongly in favor of this KEP, but I leave the specific details for people most familiar with Kubelet to iron out :)

[HirazawaUi]

Can we include the history of this?

I don’t have many ideas for now, so I’ve simply placed these links in the Motivation section. If you feel the wording needs further description or that some context should be added to the links, please let me know — I’ll be happy to make the necessary changes.