New instances shouldn't be placed on incompletely time-synchronized sleds

[askfongjojo]

A newly added sled which didn't succeed in getting its NTP zone created (#5502) was still in use by nexus for new instance placement. Its state was set to active in the sled table once it has gone through the omdb --destructive nexus sleds add step.

According to RFD 457, the process should involve a state transition from initializing to in-service to prevent using a sled that is partially initialized for customer workload or other service zones.

[jgallagher]

According to RFD 457, the process should involve a state transition from initializing to in-service to prevent using a sled that is partially initialized for customer workload or other service zones.

We don't yet have an initializing sled state; having one (and using it correctly) should fix this.

I wanted to clarify one thing though - particularly because of #5502, this sled should not have been eligible for new crucible regions - the sled was only chosen to run instances, not place disks, right?

[askfongjojo]

That's correct. Disks didn't land on the sled because the dataset records haven't been inserted into the CRDB table which disk placement query goes against.

[davepacheco]

In today's update call we discussed whether this was an R8 blocker and concluded that it wasn't because in practice for this release we'll be adding a sled during a maintenance window when provisioning will not be enabled. We won't re-enable provisioning in this intermediate state.

[morlandi7]

Something similar was seen on R17 dogfood update. This issue will be updated to more generally track a fix for preventing VMs from being placed on sleds that are still being time-synced