Delivery limit validation rejects queues declared with `x-delivery-limit: -1`, RabbitMQ's documented way to get an unlimited delivery limit

[ramonsmits]

Describe the bug

Description

The delivery limit validation introduced for RabbitMQ 4.0 fails endpoint startup when the input queue was declared with the x-delivery-limit: -1 queue argument — even though this is the method the RabbitMQ documentation describes as the only guaranteed way to get a truly unlimited delivery limit (see the RabbitMQ docs change referenced in #1688: "The value of -1, which disables the limit altogether, cannot be currently set through a policy - this can only be done with x-delivery-limit=-1 queue argument during queue declaration.").

On RabbitMQ 4.x, a quorum queue declared with x-delivery-limit: -1 is reported by the management API with "delivery_limit": "unlimited", which the transport parses as -1. BrokerVerifier.ShouldOverrideDeliveryLimit then:

1. does not match the BigValueInsteadOfActuallyUnlimited (100000) early exit,
2. does not match the limit == -1 && IsTransportPolicy(...) branch (the value comes from a queue argument, not a transport-created policy), and
3. falls into the "explicitly configured delivery limit" guard, which throws.

https://github.com/Particular/NServiceBus.RabbitMQ/blob/10.1.8/src/NServiceBus.Transport.RabbitMQ/Administration/BrokerVerifier.cs#L166-L191

The result is that a queue configured to have exactly the semantics the validation is trying to guarantee (messages are never dropped by the broker due to repeated redeliveries) is rejected as misconfigured.

This particularly affects users who pre-provision topology outside the transport (deployment pipelines, RabbitMQ Kubernetes messaging topology operator manifests, infrastructure-as-code) with installers disabled and least-privilege runtime credentials (no policymaker). Anyone following the RabbitMQ documentation will naturally declare their queues with x-delivery-limit: -1 and then find that no endpoint can start. Because quorum queue arguments are immutable, recovering requires deleting and recreating every affected queue — the misconfiguration cannot be corrected in place, so this is expensive to discover after provisioning.

Expected behavior

Either:

• On brokers >= 4.0, a quorum queue whose effective delivery limit is reported as unlimited (-1) passes validation regardless of whether the value came from a transport policy or a queue argument; or
• The rejection is a deliberate design decision (see "Additional information" for a possible rationale around 3.x behavior), in which case it should be documented on the delivery-limit validation docs page, and the exception message should explain the actual remediation.

Actual behavior

Endpoint startup fails with:

System.InvalidOperationException: The delivery limit for '{queue}' is set to the non-default value of '-1'. Remove any delivery limit settings from queue arguments, user policies or operator policies to correct this.

The message is misleading in this case: -1 is not a limiting value, and "remove any delivery limit settings from queue arguments" is not actionable for quorum queues, whose arguments are immutable after declaration — the queue must be deleted and recreated.

Versions

• NServiceBus.RabbitMQ 10.1.8 (behavior introduced with delivery-limit validation in 10.0.0; unchanged on current master, so also affects 11.x/12.x)
• RabbitMQ broker 4.1.x
• Reproduced against rabbitmq:4.1-management

Steps to reproduce

1. Run a RabbitMQ 4.x broker: docker run -d --name rmq -p 5672:5672 -p 15672:15672 rabbitmq:4.1-management
2. Declare a quorum queue with an unlimited delivery limit, per the RabbitMQ documentation:

   curl -u guest:guest -X PUT http://localhost:15672/api/queues/%2F/my-endpoint \
     -H 'content-type: application/json' \
     -d '{"durable":true,"arguments":{"x-queue-type":"quorum","x-delivery-limit":-1}}'
   

   The management API now reports "delivery_limit": "unlimited" for this queue.
3. Start an NServiceBus endpoint named my-endpoint using NServiceBus.RabbitMQ 10.1.8 with QueueType.Quorum, installers disabled, and delivery limit validation enabled (the default).
4. Startup throws the InvalidOperationException above.

Relevant log output

System.InvalidOperationException: The delivery limit for 'my-endpoint' is set to the non-default value of '-1'. Remove any delivery limit settings from queue arguments, user policies or operator policies to correct this.
   at NServiceBus.Transport.RabbitMQ.BrokerVerifier.ShouldOverrideDeliveryLimit(Queue queue)
   at NServiceBus.Transport.RabbitMQ.BrokerVerifier.ValidateDeliveryLimit(String queueName, CancellationToken cancellationToken)
   at NServiceBus.Transport.RabbitMQ.MessagePump.Initialize(...)

Additional Information

Workarounds

• Do not set x-delivery-limit at queue declaration; instead pre-create a policy with delivery-limit: 100000 (exactly Queue.BigValueInsteadOfActuallyUnlimited) matching the queue, which passes validation read-only. For already-declared queues this workaround is unavailable without recreating the queue, since the argument cannot be removed.
• DoNotValidateDeliveryLimits() — undesirable, as it disables the protection entirely.

Possible solutions

• Accept an effective delivery limit of -1/unlimited on brokers >= 4.0 irrespective of its source (queue argument or policy). On 4.x the management API reports the effective limit directly, so the transport does not need to reason about where it came from. This would arguably be more correct than the policy the transport creates itself, since RabbitMQ documents the queue argument as the only guaranteed unlimited mechanism (RabbitMQ may not respect the unlimited retry policy created by the transport #1688).
• If arg-based -1 should remain rejected because of RabbitMQ 3.x semantics (see below), gate acceptance on BrokerVersion >= 4.0.0 (already available in BrokerVerifier), and improve the exception message for the -1 case to state that quorum queue arguments are immutable and the queue must be recreated (or the limit overridden by policy).
• Independent of the fix: document the constraint on the delivery-limit validation page, which currently doesn't mention that x-delivery-limit: -1 fails validation.

Additional information

A possible rationale for distrusting arg-based -1: on RabbitMQ 3.x a negative x-delivery-limit is treated as 0 by the broker (noted in Queue.GetDeliveryLimit, https://github.com/Particular/NServiceBus.RabbitMQ/blob/10.1.8/src/NServiceBus.Transport.RabbitMQ/Administration/ManagementApi/Models/Queue.cs#L37-L73), so the same declaration is "unlimited" on 4.x but "discard after first delivery" on 3.x. That justifies rejecting it on 3.x brokers, but not on 4.x where the broker reports the effective limit as unlimited.

The limit == -1 && IsTransportPolicy(...) branch shows the validation already accepts -1 as a valid effective limit when the transport's own (pre-10.1.6) policy produced it, so the semantics of -1 are not in question — only the source. The queue-argument source appears not to have been considered when the guard was written; the PR #1689 discussion does not mention it, and there is no test covering x-delivery-limit: -1 (the existing tests only cover finite values, https://github.com/Particular/NServiceBus.RabbitMQ/blob/10.1.8/src/NServiceBus.Transport.RabbitMQ.Tests/BrokerVerifierTests.cs#L234-L276).

Related:

• Ensure safe handling of RabbitMQ 4.0 default delivery limit #1550
• RabbitMQ may not respect the unlimited retry policy created by the transport #1688
• Use large value instead of unlimited when setting delivery limit via policy #1689
• Unexpected delivery limit policy applying to queue #1707

[mauroservienti]

After discussing this with @bording and @ramonsmits, we concluded it's not a bug. Instead, it's the intended behavior. As such, I removed the Bug label.

There is a long story behind the reasons why the transport behaves this way. In short:

1. It must support two different major versions of the RabbitMQ broker (3.x and 4.x) with different behaviors and options when it comes to delivery limits. Notably, on 3.x a negative x-delivery-limit queue argument is treated by the broker as 0, so the same queue definition that means "unlimited" on 4.x means "discard after the first delivery attempt" on 3.x.
2. Setting the delivery-limit to -1 (unlimited) via policy in RabbitMQ 4.x is not guaranteed to work. The RabbitMQ documentation states it can only be done via the x-delivery-limit=-1 queue argument at declaration time, and we had a reported case of message loss where the policy value was not honored (RabbitMQ may not respect the unlimited retry policy created by the transport #1688) — even though it does appear to work on current 4.1.x brokers when tested. "Might work" is not good enough when the failure mode is silent message loss, depending on the recoverability configuration.
3. We want to preserve customers' ability to use policies for their own purposes, so the transport never overwrites or replaces a policy it didn't create.
4. We don't want the transport to set the value on the queue definition: quorum queue arguments are immutable, so once set, the only way to change the value is deleting and recreating the queue, with all the issues coming from in-flight messages.

When delivery limit support was initially introduced, the transport created a policy setting -1 (unlimited). When [2] manifested, the only option was to switch to a different value that is effectively unlimited (100000), and update the transport validation to:

• if the effective delivery limit is 100000, we don't mind where it's set (queue argument, user policy, NServiceBus-created policy); all is good, proceed
• if the effective limit is -1 and it comes from a policy created by NServiceBus (pre-10.1.6 versions created these), update that policy to 100000 — this requires the management API credentials to have policymaker permissions
• if a delivery limit is explicitly set to any other value — via queue arguments, user policies, or operator policies, and including -1 via queue argument — throw. A finite limit could lead to message loss; and -1 via queue argument, while safe on 4.x, cannot be distinguished from the dangerous 3.x interpretation by a single uniform rule, and being immutable it is also not something the transport could remediate
• if no delivery limit is set anywhere (i.e., the broker's 4.x default of 20 applies), the transport creates its own nsb.<queue>.delivery-limit policy setting 100000 — this also requires policymaker permissions, and throws if the broker is older than 4.0 or if another policy is already applied to the queue

For those pre-provisioning topology with least-privilege credentials (no policymaker at runtime): create a policy with delivery-limit: 100000 (exactly that value) applying to the endpoint queues as part of provisioning. Validation then passes as a read-only check. Do not use the x-delivery-limit: -1 queue argument, and keep in mind that only one policy applies per queue, so the delivery limit definition must be part of whatever policy wins on those queues.

I'm keeping this open for now so it serves as documentation for whoever stumbles upon that part of the transport code in the future.