Service controller tries to add finalizer to service being deleted, causing errors

[wpross]

Used ccm version v33.1.1
K8S version v1.33.5

Issue

The service controller seems to have a race condition where it attempts to add a finalizer to a service that is already being deleted. This occurs when a reconciliation event (e.g., node update) happens after the load balancer cleanup has completed and the finalizer has been removed, but before the service object is fully deleted from etcd.

Steps to Reproduce

1. Create a LoadBalancer service with a large cluster
2. Delete the service
3. While the service is being deleted, trigger a node update event
4. Errors in cloud-controller-manager logs

Expected Behavior

The service should be deleted without errors. Once the finalizer is removed, subsequent reconciliation events should recognize the service is being deleted and either skip processing or continue with cleanup.

Actual Behavior

The controller attempts to add the finalizer back to a service that has deletionTimestamp set, resulting in:

{"log":"\"Unhandled Error\" err=\"error processing service default/my-loadbalancer-service (retrying with exponential backoff): failed to add load balancer cleanup finalizer: Service \\\"my-loadbalancer-service\\\" is invalid: metadata.finalizers: Forbidden: no new finalizers can be added if the object is being deleted, found new finalizers []string{\\\"service.kubernetes.io/load-balancer-cleanup\\\"}\" logger=\"UnhandledError\"","pid":"1","severity":"ERR","source":"controller.go:302"}

Additional log context:

2026-02-09 09:05:02 {"log":"\"Event occurred\" object=\"default/my-loadbalancer-service\" fieldPath=\"\" kind=\"Service\" apiVersion=\"v1\" type=\"Warning\" reason=\"SyncLoadBalancerFailed\" message=\"Error syncing load balancer: failed to add load balancer cleanup finalizer: Service \\\"my-loadbalancer-service\\\" is invalid: metadata.finalizers: Forbidden: no new finalizers can be added if the object is being deleted, found new finalizers []string{\\\"service.kubernetes.io/load-balancer-cleanup\\\"}\"","pid":"1","severity":"INFO","source":"event.go:389"}

2026-02-09 09:05:02 {"log":"\"Unhandled Error\" err=\"error processing service default/my-loadbalancer-service (retrying with exponential backoff): failed to add load balancer cleanup finalizer: Service \\\"my-loadbalancer-service\\\" is invalid: metadata.finalizers: Forbidden: no new finalizers can be added if the object is being deleted, found new finalizers []string{\\\"service.kubernetes.io/load-balancer-cleanup\\\"}\" logger=\"UnhandledError\"","pid":"1","severity":"ERR","source":"controller.go:302"}

2026-02-09 09:05:02 {"log":"\"Event occurred\" object=\"default/my-loadbalancer-service\" fieldPath=\"\" kind=\"Service\" apiVersion=\"v1\" type=\"Normal\" reason=\"EnsuringLoadBalancer\" message=\"Ensuring load balancer\"","pid":"1","severity":"INFO","source":"event.go:389"}

2026-02-09 09:05:02 {"log":"Adding finalizer to service default/my-loadbalancer-service","pid":"1","severity":"INFO","source":"controller.go:958"}

2026-02-09 09:05:02 {"log":"Ensuring load balancer for service default/my-loadbalancer-service","pid":"1","severity":"INFO","source":"controller.go:402"}

2026-02-09 09:04:57 {"log":"\"Event occurred\" object=\"default/my-loadbalancer-service\" fieldPath=\"\" kind=\"Service\" apiVersion=\"v1\" type=\"Warning\" reason=\"SyncLoadBalancerFailed\" message=\"Error syncing load balancer: failed to add load balancer cleanup finalizer: Service \\\"my-loadbalancer-service\\\" is invalid: metadata.finalizers: Forbidden: no new finalizers can be added if the object is being deleted, found new finalizers []string{\\\"service.kubernetes.io/load-balancer-cleanup\\\"}\"","pid":"1","severity":"INFO","source":"event.go:389"}

2026-02-09 09:04:57 {"log":"\"Unhandled Error\" err=\"error processing service default/my-loadbalancer-service (retrying with exponential backoff): failed to add load balancer cleanup finalizer: Service \\\"my-loadbalancer-service\\\" is invalid: metadata.finalizers: Forbidden: no new finalizers can be added if the object is being deleted, found new finalizers []string{\\\"service.kubernetes.io/load-balancer-cleanup\\\"}\" logger=\"UnhandledError\"","pid":"1","severity":"ERR","source":"controller.go:302"}

2026-02-09 09:04:57 {"log":"\"Event occurred\" object=\"default/my-loadbalancer-service\" fieldPath=\"\" kind=\"Service\" apiVersion=\"v1\" type=\"Normal\" reason=\"EnsuringLoadBalancer\" message=\"Ensuring load balancer\"","pid":"1","severity":"INFO","source":"event.go:389"}

2026-02-09 09:04:57 {"log":"Updating backends for load balancer default/my-loadbalancer-service with 138 nodes: [node-1 node-2 node-3 node-4 node-5 node-6 node-7 node-8 node-9 node-10 node-11 node-12 node-13 node-14 node-15 node-16 node-17 node-18 node-19 node-20 <118 more>]","pid":"1","severity":"INFO","source":"controller.go:846"}

2026-02-09 09:04:57 {"log":"Adding finalizer to service default/my-loadbalancer-service","pid":"1","severity":"INFO","source":"controller.go:958"}

2026-02-09 09:04:57 {"log":"Ensuring load balancer for service default/my-loadbalancer-service","pid":"1","severity":"INFO","source":"controller.go:402"}

2026-02-09 09:04:01 {"log":"\"Event occurred\" object=\"default/my-loadbalancer-service\" fieldPath=\"\" kind=\"Service\" apiVersion=\"v1\" type=\"Normal\" reason=\"DeletedLoadBalancer\" message=\"Deleted load balancer\"","pid":"1","severity":"INFO","source":"event.go:389"}

2026-02-09 09:04:01 {"log":"Patching status for service default/my-loadbalancer-service","pid":"1","severity":"INFO","source":"controller.go:999"}

2026-02-09 09:04:01 {"log":"Removing finalizer from service default/my-loadbalancer-service","pid":"1","severity":"INFO","source":"controller.go:973"}

2026-02-09 09:03:31 {"log":"DeleteExternalTargetPoolAndChecks(a1b2c3d4e5f6(default/my-loadbalancer-service)): Deleting health check a1b2c3d4e5f6.","pid":"1","severity":"INFO","source":"gce_loadbalancer_external.go:429"}

2026-02-09 09:03:21 {"log":"ensureExternalLoadBalancerDeleted(a1b2c3d4e5f6(default/my-loadbalancer-service)): Deleting target pool.","pid":"1","severity":"INFO","source":"gce_loadbalancer_external.go:387"}

2026-02-09 09:02:51 {"log":"ensureExternalLoadBalancerDeleted(a1b2c3d4e5f6(default/my-loadbalancer-service)): Deleting IP address.","pid":"1","severity":"INFO","source":"gce_loadbalancer_external.go:377"}

2026-02-09 09:02:51 {"log":"ensureExternalLoadBalancerDeleted(a1b2c3d4e5f6(default/my-loadbalancer-service)): Deleting firewall rule.","pid":"1","severity":"INFO","source":"gce_loadbalancer_external.go:363"}

2026-02-09 09:02:51 {"log":"ensureExternalLoadBalancerDeleted(a1b2c3d4e5f6(default/my-loadbalancer-service)): Deleting forwarding rule.","pid":"1","severity":"INFO","source":"gce_loadbalancer_external.go:381"}

2026-02-09 09:02:50 {"log":"\"Event occurred\" object=\"default/my-loadbalancer-service\" fieldPath=\"\" kind=\"Service\" apiVersion=\"v1\" type=\"Normal\" reason=\"DeletingLoadBalancer\" message=\"Deleting load balancer\"","pid":"1","severity":"INFO","source":"event.go:389"}

2026-02-09 09:02:50 {"log":"Deleting existing load balancer for service default/my-loadbalancer-service","pid":"1","severity":"INFO","source":"controller.go:383"}

[k8s-ci-robot]

This issue is currently awaiting triage.

If the repository mantainers determine this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.