Transport requires the serializer to be defined in the EndpointConfiguration or a custom serializer to be configured to handle message wrapping before sends

[mauroservienti]

Transport currently relies on the serializer defined in the EndpointConfiguration or a custom serializer configured to handle message wrapping before sends. The problem is that the serializer is defined using the SerializationDefinition Core type, which, in order to be used, requires an Endpoint with certain settings defined (MessageMetadataRegistry and Conventions). That behavior couples the transport to be used in the context of an NServiceBus endpoint, and makes the raw transport usage more complex. E.g., the following hacks are required:

NServiceBus.AzureStorageQueues/src/Transport/AzureStorageQueueTransport.cs

Lines 131 to 145 in 35e4cb2

	var serializerSettingsHolder = hostSettings.CoreSettings;
	if (serializerSettingsHolder == null)
	{
	//HACK:
	//in raw transport mode to set up the required serializer a settings holder
	//is needed to store MessageMetadataRegistry and Conventions instances.
	var tempSettingsHolder = new SettingsHolder();
	const BindingFlags flags = BindingFlags.Instance | BindingFlags.Public | BindingFlags.NonPublic | BindingFlags.CreateInstance;
	var conventions = tempSettingsHolder.GetOrCreate<Conventions>();
	var registry = (MessageMetadataRegistry)Activator.CreateInstance(typeof(MessageMetadataRegistry), flags, null, new object[] { new Func<Type, bool>(t => conventions.IsMessageType(t)) }, CultureInfo.InvariantCulture);
	tempSettingsHolder.Set(registry);
	serializerSettingsHolder = tempSettingsHolder;
	}

NServiceBus.AzureStorageQueues/src/Transport/AzureStorageQueueTransport.cs

Lines 237 to 260 in 35e4cb2

	static MessageWrapperSerializer BuildSerializer(SerializationDefinition userWrapperSerializationDefinition, ReadOnlySettings settings)
	{
	return userWrapperSerializationDefinition != null
	? new MessageWrapperSerializer(userWrapperSerializationDefinition.Configure(settings).Invoke(MessageWrapperSerializer.GetMapper()))
	: new MessageWrapperSerializer(GetMainSerializerHack(MessageWrapperSerializer.GetMapper(), settings));
	}
	internal static IMessageSerializer GetMainSerializerHack(IMessageMapper mapper, ReadOnlySettings settings)
	{
	if (!settings.TryGet<Tuple<SerializationDefinition, SettingsHolder>>(SerializerSettingsKey, out var serializerSettingsTuple))
	{
	throw new Exception("No serializer defined. If the transport is used in combination with NServiceBus, " +
	"use 'endpointConfiguration.UseSerialization<T>();' to select a serializer. " +
	"If you are upgrading, install the `NServiceBus.Newtonsoft.Json` NuGet package " +
	"and consult the upgrade guide for further information. If the transport is used in isolation, " +
	"set a HostSettings.CoreSettings to an empty SettingsHolder instance before starting the transport.");
	}
	var (definition, serializerSettings) = serializerSettingsTuple;
	var serializerFactory = definition.Configure(settings);
	var serializer = serializerFactory(mapper);
	return serializer;
	}

It would be good to decouple the transport from the SerializationDefinition concept and use the same approach used to define the message unwrapper, which is just a Func<,>:

NServiceBus.AzureStorageQueues/src/Transport/AzureStorageQueueTransport.cs

Lines 433 to 441 in 35e4cb2

	public Func<QueueMessage, MessageWrapper> MessageUnwrapper
	{
	get => messageUnwrapper;
	set
	{
	Guard.AgainstNull(nameof(MessageUnwrapper), value);
	messageUnwrapper = value;
	}
	}

It's important to keep in mind wire compatibility for in-flight messages.