Stop keeping the state machine in status
The state machine stores its current state in the server's status. That is the wrong place. Status is observed state, and anything can wipe it: a status subresource reset, a restore from backup, a migration. If the status
disappears, the server falls back to the initial state, and from there the reconciler happily re-runs discovery on a machine that might be claimed and running a workload. In the worst case it powers the box off. Deletion handling reads the state from status too, so a wiped status can also unblock deletion of a server that is in maintenance.
The lifecycle state is not something we observe from the hardware. It is our own bookkeeping, and it should be treated that way. Either derive it from the things we control (claim ref, maintenance ref, boot configuration) or persist it in metadata, e.g. an annotation, which survives a status reset. The transitions themselves stay as they are.
Return early after any modification
The reconciler should stop working the moment it has modified the server object, in spec, status or metadata. Every write triggers a new reconcile event, and the next pass gets a fresh object from the cache. Continuing after a write means acting on stale data for the rest of the pass.
Today we do this in some places but not others, and the inconsistency shows. There is even a spot where we re-read the server directly from the API server in the middle of a handler because we no longer trust our own copy. If every write is followed by an immediate return, that workaround and the whole class of stale-object bugs go away. The state handlers become single-step: do one thing, return, let the next event continue.
Both changes point the same direction, so doing them in one go makes sense.
Stop keeping the state machine in status
The state machine stores its current state in the server's status. That is the wrong place. Status is observed state, and anything can wipe it: a status subresource reset, a restore from backup, a migration. If the status
disappears, the server falls back to the initial state, and from there the reconciler happily re-runs discovery on a machine that might be claimed and running a workload. In the worst case it powers the box off. Deletion handling reads the state from status too, so a wiped status can also unblock deletion of a server that is in maintenance.
The lifecycle state is not something we observe from the hardware. It is our own bookkeeping, and it should be treated that way. Either derive it from the things we control (claim ref, maintenance ref, boot configuration) or persist it in metadata, e.g. an annotation, which survives a status reset. The transitions themselves stay as they are.
Return early after any modification
The reconciler should stop working the moment it has modified the server object, in spec, status or metadata. Every write triggers a new reconcile event, and the next pass gets a fresh object from the cache. Continuing after a write means acting on stale data for the rest of the pass.
Today we do this in some places but not others, and the inconsistency shows. There is even a spot where we re-read the server directly from the API server in the middle of a handler because we no longer trust our own copy. If every write is followed by an immediate return, that workaround and the whole class of stale-object bugs go away. The state handlers become single-step: do one thing, return, let the next event continue.
Both changes point the same direction, so doing them in one go makes sense.