KEP-5673: add Graceful Deletion support for Custom Resources #5674

dasionov · 2025-10-29T16:29:28Z

One-line PR description:
Add support for graceful deletion of Custom Resources by honoring --grace-period and setting metadata.deletionGracePeriodSeconds.

Issue link: Support for Propagating --grace-period to metadata.deletionGracePeriodSeconds for custom resources #5673

k8s-ci-robot · 2025-10-29T16:29:38Z

Welcome @dasionov!

It looks like this is your first PR to kubernetes/enhancements 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes/enhancements has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

k8s-ci-robot · 2025-10-29T16:29:39Z

Hi @dasionov. Thanks for your PR.

I'm waiting for a github.com member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

k8s-ci-robot · 2025-10-29T16:29:40Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: dasionov
Once this PR has been reviewed and has the lgtm label, please assign deads2k for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

keps/sig-api-machinery/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

lmktfy · 2025-10-30T23:30:44Z

keps/sig-api-machinery/5673-custom-resource-graceful-deletion/README.md

+- Potential for confusion about when grace periods apply (only with finalizers)
+
+## Alternatives
+


Could we achieve the same aim without any in-tree code? If so, I recommend listing that as one of the alternatives.

What do you mean by in-tree code?
I listed several alternatives, but in my opinion, none of them are as good as the main one.
If there’s another alternative you think might be a better fit that I’ve missed, I’d be glad to hear it.

In-tree means code in the repository https://github.com/kubernetes/kubernetes

lmktfy · 2025-10-30T23:32:48Z

keps/sig-api-machinery/5673-custom-resource-graceful-deletion/README.md

+
+**Decision**: Rejected. Grace periods should only apply when there's work to do
+(i.e., finalizers present), consistent with Pod behavior.
+


How I'd implement grace period for custom resources? A CEL expression that maps a requested grace period to a decision:

nope, no grace period

the actual grace period (eg: clamped to 60 second minimum)

Does that seem feasible? The expression would be part of the CustomResourceDefinition API.

The main limitation I see with the CEL-based approach is that it only evaluates whether to apply graceful deletion and possibly determines how much grace time to allow — for example, returning 0 if there are no finalizers, or request.gracePeriodSeconds otherwise.

However, evaluation is only the first step. The second and more critical part is propagating the evaluated grace period value into the resource, so that relevant controllers can act on it. This propagation step cannot be handled by a CEL expression (and not even by a webhook), since webhooks cannot mutate resources during DELETE operations.

In our KubeVirt use case, the logic needs to:

Compute the effective grace period based on the user’s request, existing finalizers, etc.

Update the resource metadata with that value so controllers can rely on it for cleanup.

Propagate it through the resource hierarchy:

The VM receives the delete request and its grace period is updated (e.g. from 1h → 30s).

The VMI deletion then reflects the same grace period.

Finally, the virt-launcher Pod is terminated with that propagated grace period.

Only after this cascade completes should the VM, VMI, and Pod all be removed in order.

For this behavior to work reliably across all delete requests, we need native support for setting and reading the metadata.gracePeriod field — something we currently can’t achieve via a custom webhook or CEL evaluation alone.

If we are writing the code for Kubernetes, the in-tree CEL evaluation absolutely can mutate metadata during handling of a DELETE.

We would still need the metadata field; the difference is that we do not directly set it to an arbitrary, client-provided value.

lmktfy · 2025-11-10T19:38:47Z

keps/sig-api-machinery/5673-custom-resource-graceful-deletion/kep.yaml

+  - "@dasionov"
+owning-sig: sig-api-machinery
+participating-sigs:
+  - sig-apimachinery


nit: this doesn't look right. I would have an empty list here.

lmktfy · 2025-11-10T19:39:02Z

keps/sig-api-machinery/5673-custom-resource-graceful-deletion/kep.yaml

+# The target maturity stage in the current dev cycle for this KEP.
+# This KEP proposes a stable implementation without feature gate as the change
+# is backward compatible and low-risk.
+stage: stable


~~alpha~~ beta perhaps?

lmktfy · 2025-11-10T19:40:27Z

keps/sig-api-machinery/5673-custom-resource-graceful-deletion/README.md

+  for honoring the grace period)
+- Supporting grace periods for Custom Resources without finalizers (consistent
+  with Pod behavior where grace periods only apply when there's cleanup work)
+- Changing the CRD API or requiring CRD authors to explicitly opt-in to this


Might need a reword. I think we are planning to change the API for CustomResourceDefinitions.

How so ?
The only thing I tend to achieve in this KEP is a mutation of metadata.deletionGracePeriodSeconds existing field on delete, if I understand CEL implamantation correctly I don't see how CRD api change is needed

lmktfy · 2025-11-10T19:41:14Z

keps/sig-api-machinery/5673-custom-resource-graceful-deletion/README.md

+behavior.
+
+This KEP proposes implementing this as stable behavior without a feature gate,
+as the change is purely additive, backward compatible, and poses minimal risk.


There should still be a way to opt out (I suggest).

lmktfy · 2025-11-10T19:44:28Z

keps/sig-api-machinery/5673-custom-resource-graceful-deletion/README.md

+   - Verify resource still exists with grace period
+   - Remove second finalizer
+   - Verify resource is deleted
+


Is there a way for a CRD to opt out of graceful deletion (eg using CEL)? If so, let's test that the opt out works how we expect it would.

We are not fully implementing the graceful termination mechanism in Kubernetes as it works for Pods yet. Instead, we currently only consider the user’s request to shorten or extend the existing graceful termination period in valid cases — for example, when finalizers are present to handle the termination process or under similar conditions.

The responsibility for implementing graceful termination lies with the CRD controllers, so there’s no need to opt out when no graceful termination logic is implemented in the first place.

lmktfy · 2025-11-10T19:45:25Z

keps/sig-api-machinery/5673-custom-resource-graceful-deletion/README.md

+alpha/beta graduation process. This is justified because:
+
+1. **Backward Compatible**: The change only adds information (a metadata field)
+   that existing controllers can safely ignore


Have we tested all existing controllers to ensure they don't break?

If not, we should caveat "safely".

lmktfy · 2025-11-10T19:47:10Z

keps/sig-api-machinery/5673-custom-resource-graceful-deletion/README.md

+  - Will enabling / disabling the feature require downtime or reprovisioning of a node? No
+
+**Rationale for no feature gate**: This change is purely additive and backward
+compatible. It only sets an additional metadata field that existing controllers


I don't know that we have tested all API clients to verify this.

lmktfy · 2025-11-10T19:47:24Z

keps/sig-api-machinery/5673-custom-resource-graceful-deletion/README.md

+**Rationale for no feature gate**: This change is purely additive and backward
+compatible. It only sets an additional metadata field that existing controllers
+can safely ignore. The worst-case scenario is that the field is set but not used,
+which poses no risk to cluster stability or functionality.


[citation needed]

lmktfy · 2025-11-10T19:48:36Z

keps/sig-api-machinery/5673-custom-resource-graceful-deletion/README.md

+
+###### How can an operator determine if the feature is in use by workloads?
+
+1. Check if CRs have `deletionGracePeriodSeconds` set:


Does this work if there are no recent deletes?

it's a pointer, it would be nil if there is no value.

This KEP proposes adding support for graceful deletion to Custom Resources (CRs) by honoring the `--grace-period` flag (and `DeleteOptions.GracePeriodSeconds`) during deletion operations. Currently, when users delete a Custom Resource with a specified grace period, the value is ignored, leading to unexpected behavior for controllers and operators that rely on finalizers and graceful shutdown logic. This enhancement will ensure that when a Custom Resource is deleted with a grace period, the `metadata.deletionGracePeriodSeconds` field is properly set, allowing finalizers to observe this value and implement appropriate graceful shutdown behavior. This KEP proposes implementing this as stable behavior without a feature gate, as the change is purely additive, backward compatible, and poses minimal risk. Signed-off-by: Daniel Sionov <[email protected]>

k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Oct 29, 2025

k8s-ci-robot added the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Oct 29, 2025

k8s-ci-robot added the kind/kep Categorizes KEP tracking issues and PRs modifying the KEP directory label Oct 29, 2025

k8s-ci-robot requested review from deads2k and fedebongio October 29, 2025 16:29

k8s-ci-robot added sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Oct 29, 2025

lmktfy reviewed Oct 30, 2025

View reviewed changes

lmktfy reviewed Nov 10, 2025

View reviewed changes

dasionov force-pushed the gracefull_deletion_crs branch from c6355aa to 7962067 Compare November 11, 2025 11:28

dasionov force-pushed the gracefull_deletion_crs branch from 7962067 to 76c9509 Compare November 11, 2025 11:30

		- Potential for confusion about when grace periods apply (only with finalizers)

		## Alternatives


		Decision: Rejected. Grace periods should only apply when there's work to do
		(i.e., finalizers present), consistent with Pod behavior.


		###### How can an operator determine if the feature is in use by workloads?

		1. Check if CRs have `deletionGracePeriodSeconds` set:

KEP-5673: add Graceful Deletion support for Custom Resources #5674

Are you sure you want to change the base?

KEP-5673: add Graceful Deletion support for Custom Resources #5674

Conversation

dasionov commented Oct 29, 2025

Uh oh!

k8s-ci-robot commented Oct 29, 2025

Uh oh!

k8s-ci-robot commented Oct 29, 2025

Uh oh!

k8s-ci-robot commented Oct 29, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lmktfy Nov 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

lmktfy Nov 10, 2025 •

edited

Loading