Perform reconciliation if all three downstairs are in live-repair

[mkeeter]

This is staged on top of the merge of #1783 and #1777, so it's hard to review right now; once those PRs are merged, I'll rebase this one.

Fixes #1556 by adding a new transition from "three downstairs in LiveRepairReady" to reconciliation.

I tested this using the same sequence as #1783 to get all three downstairs faulted, then bringing them back online. Watching upstairs_info.d shows that we successfully go through reconciliation (REC) then back to active:

   PID  SESSION DS0 DS1 DS2   UPW   DSW      JOBID   WRITE_BO    IP0   IP1   IP2     D0    D1    D2     S0    S1    S2
 60907 ae1ef96e FLT FLT FLT     0     0       8817          0      0     0     0      0     0     0      0     0     0
 60907 ae1ef96e LRR FLT FLT     0     0       8817          0      0     0     0      0     0     0      0     0     0
 60907 ae1ef96e LRR FLT FLT     0     0       8817          0      0     0     0      0     0     0      0     0     0
 60907 ae1ef96e LRR FLT FLT     0     0       8817          0      0     0     0      0     0     0      0     0     0
 60907 ae1ef96e LRR LRR FLT     0     0       8817          0      0     0     0      0     0     0      0     0     0
 60907 ae1ef96e LRR LRR FLT     0     0       8817          0      0     0     0      0     0     0      0     0     0
 60907 ae1ef96e LRR LRR FLT     0     0       8817          0      0     0     0      0     0     0      0     0     0
 60907 ae1ef96e LRR LRR FLT     0     0       8817          0      0     0     0      0     0     0      0     0     0
 60907 ae1ef96e LRR LRR FLT     0     0       8817          0      0     0     0      0     0     0      0     0     0
 60907 ae1ef96e LRR LRR FLT     0     0       8817          0      0     0     0      0     0     0      0     0     0
 60907 ae1ef96e LRR LRR FLT     0     0       8817          0      0     0     0      0     0     0      0     0     0
 60907 ae1ef96e REC REC REC     0     0       8817          0      0     0     0      0     0     0      0     0     0
 60907 ae1ef96e REC REC REC     0     0       8817          0      0     0     0      0     0     0      0     0     0
 60907 ae1ef96e REC REC REC     0     0       8817          0      0     0     0      0     0     0      0     0     0
 60907 ae1ef96e REC REC REC     0     0       8817          0      0     0     0      0     0     0      0     0     0
 60907 ae1ef96e ACT ACT ACT     1   221       9081          0      1     1     1    220   220   220      0     0     0
 60907 ae1ef96e ACT ACT ACT     1   529       9807          0      1     1     1    528   528   528      0     0     0

[leftwo]

Just a thought on function naming.

I'm excited to see this back, as it unlocks a test I wrote years (and years) ago that I've been waiting to run.

[leftwo]

I'm going to want a stat for DTrace when we do this. Just to see how often it happens. I'm happy to add that myself in a later PR (as I'll be updating the DTrace scripts to print it)

[leftwo]

I did not reall like the name of this function before, and more so now that will also do reconciliation from inside here if we deem its necessary.

We are:
Checking to see if we need to do live repair, and possibly checking to see if we need to do reconciliation.

Maybe call this verify_downstairs_consistency()?
I'm open to other suggestions too.
The comment where we call this should be updated to indicate that we are cheking for LR or Reconciliation.

[mkeeter]

Hmmm, it also starts live-repair, so just verify isn't great (neither is check for that matter).

What about ensure_downstairs_consistency?

[leftwo]

Yesssssssssssss.

[mkeeter]

Renamed, and comments are now updated (please let me know if I missed any!)