Async actor lifecycle handlers (PostStop, PreStart, etc.)

[Aaronontheweb]

Summary

Currently actor lifecycle methods (PreStart, PostStop, PreRestart, PostRestart) are synchronous. This makes it awkward to handle async cleanup in PostStop when an actor has pending state to flush.

Current Workaround

protected override void PostStop()
{
    if (_pendingDelta != 0)
    {
        try
        {
            // Forced to block on async operation
            _store.UpdateAsync(_pendingDelta)
                .GetAwaiter().GetResult();
        }
        catch (Exception ex)
        {
            Context.GetLogger().Error(ex, "Failed to flush");
        }
    }
    base.PostStop();
}

This works but:

• Blocks a thread during shutdown
• Feels wrong in an async-first codebase
• Easy to accidentally fire-and-forget (which loses data)

Proposed API

protected override async Task PostStopAsync()
{
    if (_pendingDelta != 0)
    {
        await _store.UpdateAsync(_pendingDelta);
    }
    await base.PostStopAsync();
}

The actor system would await these during the stop sequence.

Use Cases

• Flushing buffered writes to database before passivation
• Graceful disconnect from external services
• Async resource cleanup

Questions

• Would this require changes to the mailbox/dispatcher?
• Should this be opt-in (new base class) or added to ReceiveActor/UntypedActor?
• Any concerns about deadlocks during coordinated shutdown?

[to11mtm]

I like this idea however we will need to be considerate of how to handle certain things:

Any concerns about deadlocks during coordinated shutdown?

I'll give one to consider (I don't consider it a deal-breaker); people not being aware of timings and volume of things in clustered scenarios.

As a sort of example, let's imagine you've got hundreds or thousands sharded actors and the node is trying to gracefully handle shutdown; Unless the users are -very- cognizant of what they are doing talking to a database/etc, a thundering herd problem is very possible, and in the case of clustered scenarios it can be very tricky to handle the correct timings for handling that scenario but still allowing handoff to successfully complete.

Want to be clear, I want this too :) but we should document the risks for other users.

I suppose my one, more (less?) actionable suggestion to help mitigate that, is that in ideal state we provide some form of CancellationToken to the PostStopAsync method, and in documentation both encourage frequent checking of the token if there is more than one async op, as well as reminding users that not all things properly use the token. That said it could at least provide a 'way out' for the case of someone trying to do DB stuff (e.x. try to do the DB stuff in a transaction, but if token is cancelled before commit stage they know it's a lost cause and shutdown timed out.)

... That may however require additional plumbing.... e.x. config values might be a way to MVP it... but at least as far as Coordinated Shutdown goes, I'd hope we could just pass a special stop message with a CancellationToken included that's based on it's timeout value for that stage...

[Aaronontheweb]

I suppose my one, more (less?) actionable suggestion to help mitigate that, is that in ideal state we provide some form of CancellationToken to the PostStopAsync method, and in documentation both encourage frequent checking of the token if there is more than one async op, as well as reminding users that not all things properly use the token. That said it could at least provide a 'way out' for the case of someone trying to do DB stuff (e.x. try to do the DB stuff in a transaction, but if token is cancelled before commit stage they know it's a lost cause and shutdown timed out.)

That's a great idea, actually - a system-provided CTS. "You get this long to shut down gracefully before we do it violently"