[ServiceBus] Message received from `receive_deferred_messages` does not have a lock token.

[ashleysommer]

• Package Name:
  azure-servicebus

• Package Version:
  7.14.2

• Operating System:
  Windows 11, and Ubuntu 24.04

• Python Version:
  3.12

Describe the bug
I have deferred messages in my Service Bus Topic. I know the sequence number of the message I want.
I use receive_deferred_messages with the message sequence number to receive the messsage.
The Parsed message correctly show it is locked for 60 seconds (it has a valid locked_until_utc value) but it does not have a corresponding lock_token (it is None).
Attempting to settle this message (eg, to complete it) fails because the associated lock_token is None.

To Reproduce
Steps to reproduce the behavior:

1. Have a MessageBus topic with 1 or more deferred messages. Note their sequence numbers.
2. Create a ServiceBusReceiver with correct credentials and service mode "PEEK_LOCK".
3. run receiver.receive_deferred_messages([1234]) with the sequence number of your message.
4. check the lock_token value for each message received.

my_message_sequence_number = 1234
async with bus_client.get_subscription_receiver(topic, subscription, max_wait_time=5, prefetch_count=0) as receiver:
    msgs = await receiver.receive_deferred_messages([my_message_sequence_number], timeout=30)
    for msg in msgs:
        assert msg.lock_token is not None

Expected behavior
The python SDK should extract the "lock-token" value from the raw message when parsing the ServiceBus Message. I've verified AMQP protocol does have the correct "lock-token" property in the raw message, but the SDK code does not use it.

This is the relevant section of the code where the SDK parses the message content, but ignores the corresponding "lock-token" content.

azure-sdk-for-python/sdk/servicebus/azure-servicebus/azure/servicebus/_transport/_pyamqp_transport.py

Lines 811 to 830 in f379729

	def parse_received_message( # pylint:disable=docstring-keyword-should-match-keyword-only
	message: "Message", message_type: Type["ServiceBusReceivedMessage"], **kwargs: Any
	) -> List["ServiceBusReceivedMessage"]:
	"""
	Parses peek/deferred op messages into ServiceBusReceivedMessage.
	:param ~pyamqp.message.Message message: Message to parse.
	:param ~azure.servicebus.ServiceBusReceivedMessage message_type: Parse messages to return.
	:keyword ~azure.servicebus.ServiceBusReceiver receiver: Required.
	:keyword bool is_peeked_message: Optional. For peeked messages.
	:keyword bool is_deferred_message: Optional. For deferred messages.
	:keyword ~azure.servicebus.ServiceBusReceiveMode receive_mode: Optional.
	:return: List of service bus received messages.
	:rtype: list[~azure.servicebus.ServiceBusReceivedMessage]
	"""
	parsed = []
	if message.value:
	for m in message.value[b"messages"]:
	wrapped = decode_payload(memoryview(m[b"message"]))
	parsed.append(message_type(wrapped, **kwargs))
	return parsed

Screenshots
lock_token is None:

The raw pyAMQP payload does have the lock token (alongside the "message" payload), however it is not used:

Additional context
The root cause of this is that the SDK assumes the lock token will always be in the form of a "delivery-tag", that is in a header of the received message. However that is only true for ACTIVE messages. See the relevant section of the AMQP spec: https://learn.microsoft.com/en-us/azure/service-bus-messaging/service-bus-amqp-request-response#response-12
This shows that for DEFERRED messages, the lock-token is sent alongside the received messages.
See also the note about the usage of "delivery-tag" vs "lock-token" here https://learn.microsoft.com/en-us/azure/service-bus-messaging/service-bus-amqp-request-response#request

[ashleysommer]

Note, this issue has been present since at least 2019
This issue has the same cause #9010

And this issue is also related, discussing the different ways a response can include a lock token: #7336

[ashleysommer]

Another thing to note, I've tried with both the pyAMQP and the uAMQP transports (because they have different receiver/parser codepaths) and they both exhibit exactly the same bug.

[ashleysommer]

I have a simple fix for this here: #42471