Question: Best practice for cleaning up resources from a single cluster with stopMatchingBehavior: LeavePolicies?

[kahirokunn]

Hello ✋

I came across an interesting scenario while testing ClusterProfile behavior and wanted to ask about the recommended approach.

Scenario

During my testing with a ClusterProfile that has stopMatchingBehavior: LeavePolicies, I accidentally matched it to a cluster where I didn't intend to deploy. This got me thinking about the best way to clean up resources from just that specific cluster without affecting other clusters where the profile is correctly deployed.

The challenge I noticed

1. Removing the label from the cluster: The ClusterSummary gets deleted, but resources remain in the cluster (which is expected behavior with LeavePolicies)
2. Temporarily changing the ClusterProfile's stopMatchingBehavior: This could potentially impact all other clusters where the profile is correctly deployed

Question

Is there a recommended procedure or best practice for this kind of situation? I'm curious if there's a clean way to remove resources from a single mismatched cluster while keeping the stopMatchingBehavior: LeavePolicies setting intact for the ClusterProfile.

Thanks for your help! 🙏

[gianlucam76]

Isolating and Draining a Single Cluster

Step 1: Tag the Target Cluster

Add a unique, temporary extra label to the cluster you intend to isolate and remove resources from.

Example: Label the cluster with key:

drain−target: temp−isolate

Step 2: Create a Temporary Draining Profile

Create a new ClusterProfile that is a copy of your original profile but with three critical modifications:

Lower Tier: Assign a lower tier value than the original ClusterProfile. This ensures the new profile takes precedence over the original.

Specific Match: Configure the profile to match only the one cluster you tagged in Step 1 (using the new extra label).

Withdraw Behavior: Set the matching behavior to stopMatchingBehavior:WithdrawPolicies. This instructs the system to remove all applied resources when the cluster no longer matches this profile.

Step 3: Remove Original Match Labels

Since the new, lower-tier ClusterProfile is now in effect for the target cluster, you can safely remove the original labels from the cluster that caused it to match the original ClusterProfiles.

The target cluster now only matches the new temporary profile.

Step 4: Final Resource Deletion

Delete the temporary ClusterProfile created in Step 2.

Because the cluster's labels no longer match this profile, the WithdrawPolicies behavior is triggered, which removes all policies and resources from the cluster.