[Bug] BasicPublishAsync hangs after RabbitMQ server restart

[MarcoBarbieri90]

Describe the bug

Environment

• Client: RabbitMQ .NET Client 7.2.1
• Server (tested on): RabbitMQ 3.13.4 + Erlang 26.2.5.1 / RabbitMQ 4.0.25 + Erlang 27.3.4.10
• Server configuration: Default
• OS: Windows (RabbitMQ running as a Windows Service)

Description

We are experiencing a hang in BasicPublishAsync after restarting the RabbitMQ Windows service while publishing at high throughput (~50,000 messages every 7 seconds) with AutomaticRecoveryEnabled = true. The severity of the issue depends on whether publisher confirms are enabled or not.

When publisher confirms are disabled (publisherConfirmationsEnabled: false, publisherConfirmationTrackingEnabled: false), BasicPublishAsync silently hangs for approximately 3 minutes before automatically resuming. During this time, IConnection events (ConnectionShutdownAsync, RecoverySucceededAsync, ConnectionRecoveryErrorAsync) are not fired immediately — they are delayed by the same ~3 minutes, making them effectively useless for any timely recovery logic.

When publisher confirms are enabled (publisherConfirmationsEnabled: true, publisherConfirmationTrackingEnabled: false), the situation is worse. The IConnection events are still delayed by ~3 minutes and the connection does eventually recover, but BasicPublishAsync hangs indefinitely — even after full recovery, the publisher never resumes and no exception is ever thrown.

Root Cause

Both scenarios share the same underlying root cause, observed via the Visual Studio Task Debugger.
During the hang, the Task window reveals the following deadlock-like situation:

In the publisher without confirms, the Task window reveals the following call chain blocked at shutdown:

HandleConnectionCloseAsync(cmd, cancellationToken) inside Channel.cs is blocked attempting to write a frame to the SocketFrameHandler.
The internal Channel (bounded capacity: 128, FullMode: Wait) is full.
The reader task (WriteLoopAsync()) that drains this channel exits immediately when the RabbitMQ Windows service is stopped, leaving the channel permanently full and all writers blocked.

This means that when the server goes down, the write loop exits but the channel is never drained, causing any subsequent write attempt to wait indefinitely until automatic recovery eventually reconnects and unblocks the pipeline (~3 minutes later).

In the publisher confirms scenario, an additional problem compounds the above.
The Task window reveals the following call chain blocked at shutdown:

MaybeHandlePublisherConfirmationTcsOnChannelShutdownAsync
└─> SemaphoreSlim.WaitUntilCountOrTimeoutAsync ← blocked indefinitely

OnChannelShutdownAsync triggers MaybeHandlePublisherConfirmationTcsOnChannelShutdownAsync, which tries to acquire a SemaphoreSlim to complete or cancel all pending publisher confirm TaskCompletionSources.
This SemaphoreSlim is never released during shutdown (likely held by a write operation that is itself blocked — same root cause as issue #XXXX).
As a result, all pending BasicPublishAsync calls remain awaiting a confirm that will never arrive, even after the connection and channel are recovered.

Reproduction steps

1. Start a RabbitMQ server (3.13.x or 4.0.x) with default configuration.
2. Run the attached publisher targeting a quorum queue at ~50,000 msg/7s throughput, with AutomaticRecoveryEnabled = true.
3. While publishing, restart the RabbitMQ Windows service.

Scenario A — confirms disabled (publisherConfirmationsEnabled: false, publisherConfirmationTrackingEnabled: false):

Publisher - No Confirms.txt

Observe:

After ~3 minutes, IConnection events fire and the connection recovers.
BasicPublishAsync hangs silently for ~3 minutes.
After ~3 minutes, publishing resumes automatically.

Scenario B — confirms enabled (publisherConfirmationsEnabled: true, publisherConfirmationTrackingEnabled: false):

Publisher - Confirms.txt

Observe:

BasicPublishAsync hangs.
After ~3 minutes, IConnection events fire and the connection recovers.
BasicPublishAsync remains blocked indefinitely — the publisher never resumes.
No exception is ever thrown.

Expected behavior

1. IConnection events should fire promptly when the server goes down, not after a multi-minute delay.
2. BasicPublishAsync should either throw an exception immediately upon disconnection — allowing the caller to handle the error and retry — or recover quickly without hanging.

Additional context

No workaround has been found so far.
Are there any recommended workarounds while this is being investigated?

[michaelklishin]

IConnection events should fire promptly when the server goes down, not after a multi-minute delay.

This is not a bug, this how TCP works. See Heartbeats, including the part on low values and the risk of false positives, and Publisher Confirms.

[MarcoBarbieri90]

Thank you for the clarification.
I now understand that the ~3 minute delay is expected TCP behavior.
That said, I noticed that when enabling publisher confirms on the channel (publisherConfirmationsEnabled: true,
publisherConfirmationTrackingEnabled: false), the channel hangs permanently after the first server restart — even after the connection is fully recovered, BasicPublishAsync never resumes and no exception is thrown.
Am I handling the channel or the confirms incorrectly in this scenario?
The only configuration that gives us the expected behavior after a restart is publisherConfirmationTrackingEnabled: true, but this comes with a significant throughput penalty dropping from ~6,000 messages/second down to ~450 messages/second.
Is there a recommended way to use publisher confirms that handles server restarts correctly without such a large impact on throughput?

[ObyTan]

IConnection events should fire promptly when the server goes down, not after a multi-minute delay.

This is not a bug, this how TCP works. See Heartbeats, including the part on low values and the risk of false positives, and Publisher Confirms.

I’m seeing the same behavior on my side with a similar setup, and I arrived at essentially the same conclusions described in the message above from MarcoBarbieri90.

I understand the explanation regarding the delay being related to TCP/heartbeats, but what still concerns me is how the client behaves on top of that, especially in a typical RabbitMQ use case.

In a real-world scenario, RabbitMQ is often used to decouple real-time systems from asynchronous processing. In that context, having:

• delayed visibility of the connection loss, and
• BasicPublishAsync calls that can remain blocked indefinitely after a restart (with confirms enabled)

makes it very hard to design reliable and deterministic recovery logic on the client side.

Also, compared to the 5.x client, connection shutdown used to be surfaced immediately, which made this easier to handle (even though that version had other limitations).

I’m trying to understand what the expected behavior is here:

• should pending publish operations be expected to remain blocked in these conditions?
• and is there a recommended way to use publisher confirms under load that avoids both long detection delays and indefinite blocking?

More generally, is there a best practice for handling this scenario? It’s possible we’re missing something subtle or not well documented in how the client is intended to be used under failure/recovery conditions.

In these conditions, it leans more towards what appears to be a client library issue rather than just expected behavior.

[lukebakken]

Thank you for the reproduction steps @MarcoBarbieri90 - please continue discussion in #1930