feat: add cephadm maintenance playbooks

[jackhodgkiss]

Add two playbooks for entering and exiting maintenance mode for a given Ceph node.

kayobe playbook run $KAYOBE_CONFIG_PATH/ansible/cephadm-enter-maintenance.yml --limit ceph-mon-01

kayobe playbook run $KAYOBE_CONFIG_PATH/ansible/cephadm-exit-maintenance.yml --limit ceph-mon-01

Note these playbooks use stackhpc.cephadm.commands which will delegate the command to the first mon within your inventory. If this node is in maintenance you must specify --cephadm_delegate_host and provide another mon.

kayobe playbook run $KAYOBE_CONFIG_PATH/ansible/cephadm-exit-maintenance.yml --limit ceph-mon-01 --cephadm_delegate_host ceph-mon-02

Note: this relies on something such as stackhpc/ansible-collection-cephadm/pull/109 being merged with some additional changes.

[Alex-Welsh]

Note these playbooks use stackhpc.cephadm.commands which will delegate the command to the first mon within your inventory. If this node is in maintenance you must specify --cephadm_delegate_host and provide another mon.

Is there a way of determining the hosts that are not in maintenance and selecting one of them? That would make things a lot more simple.

[jackhodgkiss]

Note these playbooks use stackhpc.cephadm.commands which will delegate the command to the first mon within your inventory. If this node is in maintenance you must specify --cephadm_delegate_host and provide another mon.

Is there a way of determining the hosts that are not in maintenance and selecting one of them? That would make things a lot more simple.

I suppose it is possible. However I see this as no different to how we handle controllers and VIP and intentionally avoid the VIP until the end.

My concern would be if a host is in maintenance the command gets trapped as it will proceed to authenticate with the cluster and silently fail so it would involve timeouts and other work arounds.

[Alex-Welsh]

Note these playbooks use stackhpc.cephadm.commands which will delegate the command to the first mon within your inventory. If this node is in maintenance you must specify --cephadm_delegate_host and provide another mon.

Is there a way of determining the hosts that are not in maintenance and selecting one of them? That would make things a lot more simple.

I suppose it is possible. However I see this as no different to how we handle controllers and VIP and intentionally avoid the VIP until the end.

My concern would be if a host is in maintenance the command gets trapped as it will proceed to authenticate with the cluster and silently fail so it would involve timeouts and other work arounds.

Does it error gracefully if the node is in maintenance? If not it might be worth adding "precheck" task to verify

[jackhodgkiss]

Note these playbooks use stackhpc.cephadm.commands which will delegate the command to the first mon within your inventory. If this node is in maintenance you must specify --cephadm_delegate_host and provide another mon.

Is there a way of determining the hosts that are not in maintenance and selecting one of them? That would make things a lot more simple.

I suppose it is possible. However I see this as no different to how we handle controllers and VIP and intentionally avoid the VIP until the end.
My concern would be if a host is in maintenance the command gets trapped as it will proceed to authenticate with the cluster and silently fail so it would involve timeouts and other work arounds.

Does it error gracefully if the node is in maintenance? If not it might be worth adding "precheck" task to verify

I will have to check but I think ceph has a tendency to return 0 regardless of it being successful or not.

[cityofships]

Won't be possible for any host holding RGW services - gets:

WARNING: Removing RGW daemons can cause clients to lose connectivity.
Note: Warnings can be bypassed with the --force flag

Of course --force defeats the purpose of other checks and is not viable here.

[markgoddard]

I reworked these into roles in the cephadm collection: stackhpc/ansible-collection-cephadm#153. Once that merges I'll propose some playbooks in SKC.