Outbox: Additional latency due to outbox dispatching based on send/received control message

[ramonsmits]

When the transaction session completed the messages are not immediately dispatched. The control messages that is already in flight will be processed and retrieve the outbox record and only then starts dispatching and causes unneeded latency.

Poll based transports

This latency is very noticeable in a poll based transport like SQL transport. Especially if the endpoint is idle and the queue peek interval has the default or higher value. In that case the latency can be observed to double in length. Such "pick up" latency due to polling does not exist for push-based transports.

Queue backlog

If the control message is sent to a queue that has a backlog of messages then dispatching is delayed until the control messages gets processed. If the backlog presents a queue time of 5 seconds the messages that were sent in the API will have a +5 second delay.

Monitor the critical time metric to know if a queue backlog is causing the observed dispatch/delivery latency.

Busy storage

As the control message is already dispatched it can happen that the data is not yet available for reading when the control message is processed. Processing fails and the control message is retried and resulting in additional latency.

This can be mitigated by running the outbox/business data transaction in a pessimistic locking configuration.

[ramonsmits]

Solution suggestion

A solution strategy would be to immediately dispatch the transport operations. This does not have to block the API that invokes the transactional session. Dispatching can happen on a different thread immediately after the storage commit completes so that the API can already return.

The control message would still be sent but can be sent with a delay to act as a safety mechanism to still dispatch if an error occurred during dispatching or if even the whole process died.

Under normal operation, the control message would query the outbox and do nothing as the operations should already be dispatched.

1. Removes all latency scenarios mentioned in the description
2. Transport operation objects are still in memory and no deserialization penalty