Runs are marked as lost when they should be re-enqueued

[taylordowns2000]

What's happening now can be reproduced on main by adding Process.sleep(10_000) right after line 39 of worker_channel.ex then running any run. Here's how the bug works:

1. The ws-worker joins the worker channel and requests work via the "claim" API.
2. Every once in a while, the DB poops itself and the query takes a very long time. (This is what we replicate with the Process.sleep)
3. The ws-worker goes away, after telling us ✘ TIMEOUT on claim. Runs may be lost.
4. The DB query then finishes, setting the run to :claimed and replying to Nobody At All that it's got a run ready to be executed.
5. Nobody At All does anything (the worker is long gone at this point) and the run sits there as :claimed for N minutes—controlled by the Janitor's grace period.
6. The Janitor (after N minutes) comes by and marks this run as :lost, secretly giving it the status "LostAfterClaim".
7. We tell the user that we don't know what happened, apologizing profusely for losing the run and saying, "Sorry, we have no more info and there's nothing you can do."

Really, it seems like there is something (or there are a couple of somethings) we can do here:

1. In the worker_channel, Make the socket timeout GREATER OR EQUAL to the query timeout so that we don't shut the door before the positive reply comes.
2. In the worker_channel Check if the connection to the worker is still alive before we {:reply, {:ok, %{runs: runs}}, socket}. If it's not still alive, then roll back the transaction and/or (more simply) set the run state back to :available so another worker can come along and claim it.
3. In the janitor.ex, don't set runs to :lost (with LostAfterClaim), set them back to :available so they get picked up. If we're certain that the run hasn't been started (:claimed is a different state than :started) then simply put it back in the queue... it's not lost, it's still waiting for some worker to work on it.

@stuartc , @josephjclark , @rorymckinley , a penny for your thoughts here? Of the last 230 lost runs we know of, 212 were LostAfterClaim and it seems that implementing any/all of these three enhancements might put a real dent in that.

I've opened an illustrative fix for concept 2 here: #3566 - if it has legs I'll write tests. so far just testing manually and it works a treat.

[rorymckinley]

@taylordowns2000

Responses to the suggested solution followed by a general comment on the current state of play.

1. "Make the socket timeout GREATER OR EQUAL" --> yes, I think this is a good idea, although I think @josephjclark may have done this already - as I did set a timeout for him on the worker pods last week.
2. "Check if the connection to the worker is still alive" Also a good idea - if we never 'claim' the run in the first place, then the problem never occurs - I will look at the PR.
3. "If we're certain that the run hasn't been started" - this part I am less certain about (i.e. I am uncertain about our certainty :P). I have compiled data for every lost run since July 27th - I have found cases where, to my untrained eye, it appears that the worker may have started doing something but the run is still LostAfterClaim. For example, this may be due to the worker losing the connection to Lightning, but maybe also when we run out of heap memory on the worker. I will send you the link to the data.

As far as I can tell, we have not had a Lost Run that resulted from query performance issues since the morning of August 25th. This may change as Joe gets a chance to work through the current issues - i.e. we may still be having issues caused by poor DB performance, I am just nor recognising them as such. Eyeballing the data suggests that heap memory may currently be the most prevalent cause.

I do not think that this invalidates your plan of attack, as it is unclear how much time the DB interventions have bought us, but it does mean that these changes may not result in an immediate reduction in lost runs.