Hostpath PVs remain dangling when the node is deleted before OpenEBS can reconcile

[Tristan-Otterpohl-Forter]

Context

We are running Elasticsearch on Kubernetes using the OpenEBS Dynamic LocalPV (hostpath) provisioner.
Under normal circumstances, when a PVC and its associated pod are deleted, OpenEBS deletes the PV and the hostpath data from the node as expected.

However, when a node is deleted before OpenEBS reconciles the PV deletion, the PV and PVC remain in a dangling state. The provisioner logs errors similar to the following:

failed to delete volume pvc-a0a48f74-2136-4264-8806-7f358f3fdbbb:
Unable to get the Node with the Node Labels {map[kubernetes.io/hostname:ip-172-31-43-116.ec2.internal]}

This happens because the node object no longer exists in the Kubernetes API, and the provisioner cannot determine where the hostpath volume resides.

---

Expected vs Actual Behavior

Action	Expected	Actual
PVC deleted while node exists	PV and hostpath data deleted successfully	Works as expected
Node deleted before PV cleanup	PV removed automatically	PV and PVC remain dangling, repeated “Unable to get the Node” errors

---

Environment

• Kubernetes version: v1.34
• OpenEBS provisioner: dynamic-localpv-provisioner v4.1.1
• StorageClass: openebs-hostpath
• Platform: AWS (nodes managed by Karpenter)

This typically occurs during node rotations or Kubernetes upgrades, when Karpenter replaces EC2 instances and deletes their corresponding Node objects quickly after draining.

---

Observations

• The issue only appears when the Node object is deleted before the PVC deletion is reconciled.
• The behavior is understandable given the node-local nature of hostpath volumes.
• The dangling PVs need manual cleanup by removing the finalizer and deleting the PV.

---

Existing Mitigation

AppsFlyer has built a tool called the PVC Releaser, which is designed to handle dangling PVC's automatically in these scenarios. But nothing manages dangling PV's. it may be useful to have that functionality built into OpenEBS.

---

Suggested Actions

This doesn’t necessarily need to be “fixed” since it’s a logical limitation, but it would help if this behavior were better documented.

Possible improvements:

1. Add a note in the OpenEBS LocalPV Hostpath documentation under “Known Limitations” or “Operational Notes”.
2. Make the log message clearer (for example, “Node object not found — skipping cleanup; manual PV deletion may be required”).
3. Optionally provide a small guide or script to identify PVs referencing deleted nodes.

---

Reproduce Issue

This is easiest to reproduce by deleting a node without draining/cordoning.

---

Summary

When a node is deleted before OpenEBS can reconcile a hostpath PV deletion, the PV and PVC remain dangling because the provisioner cannot locate the node to perform cleanup.
This behavior is expected but not clearly documented, and making it more visible would help operators running node-local storage on clusters with frequent node rotations.

[tiagolobocastro]

We've had a similar scenario here
Probably we can try similar approach here, on the delete path we can check if the node exists before attempting to launch the pod to the scrub the PV data folder.

[tiagolobocastro]

Though not sure if this should be default behaviour. I think I had suggested for lvm 2 options:

1. helm var to enable this
2. a plugin command

Though the only downside of not doing it by default is in case the node hostname label changes for example, and we might not cleanup the data from the host in such case. Maybe not a good enough reason to not do this by default?