Enable Falcon Operator to perform in-place Falcon Node Sensor upgrades in existing namespaces

[felimendi]

The current behavior of Falcon Operator 1.8.0, which blocks FalconNodeSensor updates with the error "ERROR FalconNodeSensor is attempting to install in a namespace with existing pods," (from falcon-operator/internal/controller/falcon_node/falconnodesensor_controller.go) creates a significant operational challenge for customers managing large, production Kubernetes fleets.

The core of this idea is to enhance the Falcon Operator's reconciliation logic to intelligently handle updates to an existing FalconNodeSensor Custom Resource (CR) in a non-blocking, in-place manner.

Intelligent Reconciliation (Primary Mechanism):

    When the Falcon Operator detects an Update event for a FalconNodeSensor CR (e.g., a change to spec.node.image or spec.node.version), it should first verify that the CR is already managing a DaemonSet and pods in the spec.installNamespace.

    If the CR already "owns" resources in that namespace, the operator should bypass the "namespace with existing pods" check.

    Instead, it should proceed to update the underlying DaemonSet's pod template (e.g., changing the image or other spec.node parameters).

    It should then rely on Kubernetes' native DaemonSet rolling update capabilities to gracefully replace the old sensor pods with new ones using the updated configuration. This ensures that nodes are updated one by one, maintaining security coverage throughout the process.

Distinguishing "Install" vs. "Update": The operator's logic needs to differentiate:

    Initial Installation: If a FalconNodeSensor CR is Created, and the spec.installNamespace contains any pods that are not already managed by an existing FalconNodeSensor CR, then the "namespace with existing pods" error is valid and should continue to block. This prevents accidental overwrites or conflicts.

    In-Place Update: If a FalconNodeSensor CR is Updated, and the operator already manages resources within the spec.installNamespace for that specific CR, then the validation should be relaxed to allow the update to proceed.

Optional: Explicit forceUpdate Parameter (Secondary Mechanism for Edge Cases): While intelligent reconciliation should be the default, a dedicated spec.update.forceUpdate: true (or similar) parameter could be useful for very specific edge cases. This would explicitly tell the operator: "I understand the risks, please force the update even if there are unexpected pods in the namespace, or if the operator doesn't recognize that it owns them."

[gpontejos]

Hi! Can you clarify the process you are using with in-place upgrades? Is this occurring when updating the existing instance of FalconNodeSensor or is there an attempt to apply a new FalconNodeSensor before deleting the old one? In large clusters we commonly see instances of the cleanup daemonset persisting for a long time and if a new FalconNodeSensor is applied shortly after, the overlap in deployments will cause the "namespace with existing pods" error.

[felimendi]

Hi! Thank you for the follow-up, we are attempting to perform an in-place update of an existing FalconNodeSensor Custom Resource (CR). We are not trying to apply a new FalconNodeSensor CR before deleting the old one.

Our workflow involves:

1. Modifying the spec.node.image field (or spec.falcon.version if using API-driven updates) within our existing FalconNodeSensor CR definition.

2. Using qbec apply to push this updated CR to the cluster. This operation performs a strategic merge patch or update on the single, existing falcon-node-sensor CR in the API server.

The "namespace with existing pods" error occurs when the Falcon Operator attempts to reconcile this updated existing FalconNodeSensor CR. It appears the operator's validation logic, originating from falcon-operator/internal/controller/falcon_node/falconnodesensor_controller.go:134, treats any existing pods in the falcon-system namespace (even its own older sensor pods) as a blocking condition for processing the update.

Therefore, our issue is not about an "overlap in deployments" in the sense of applying a second FalconNodeSensor CR. It's about the operator's inability to perform an in-place rolling update of the DaemonSet (managed by the same FalconNodeSensor CR) because the target namespace (falcon-system) is not empty of the currently running older sensor pods.