My codebase uses Broadway with BroadwaySQS. I recently made a few changes to begin capturing a metric for the full duration of time a message spends in the pipeline, from when SQSClient.receive_messages/2 is called, to when the message is acknowledged via Acknowledger.ack/3.
To do this, I store a :received_at timestamp in the metadata of every Broadway.Message. For SQS messages specifically, :sent_timestamp and :approximate_first_receive_timestamp are insufficient. (:sent_timestamp is the timestamp of when the message was enqueued, and :approximate_first_receive_timestamp will be stale whenever the message is redelivered.)
Then, in my implementation of Acknowledger.ack/3, I subtract :received_at from the current system time to compute each message’s time delta, and emit a custom Telemetry event.
I imagine other users of Broadway may wish to capture a similar metric, so I’m wondering if it makes sense to add one or both of the following to Broadway itself:
- A new Telemetry event for acks, dispatched when a group of messages is acknowledged via
Acknowledger.ack/3
- A
:received_at (or :created_at?) field in the metadata of all messages, automatically set when a message is created by a producer
This essentially generalizes what I implemented. Thoughts? I’d be happy to put together a PR with these changes.
My codebase uses Broadway with BroadwaySQS. I recently made a few changes to begin capturing a metric for the full duration of time a message spends in the pipeline, from when
SQSClient.receive_messages/2is called, to when the message is acknowledged viaAcknowledger.ack/3.To do this, I store a
:received_attimestamp in themetadataof everyBroadway.Message. For SQS messages specifically,:sent_timestampand:approximate_first_receive_timestampare insufficient. (:sent_timestampis the timestamp of when the message was enqueued, and:approximate_first_receive_timestampwill be stale whenever the message is redelivered.)Then, in my implementation of
Acknowledger.ack/3, I subtract:received_atfrom the current system time to compute each message’s time delta, and emit a custom Telemetry event.I imagine other users of Broadway may wish to capture a similar metric, so I’m wondering if it makes sense to add one or both of the following to Broadway itself:
Acknowledger.ack/3:received_at(or:created_at?) field in themetadataof all messages, automatically set when a message is created by a producerThis essentially generalizes what I implemented. Thoughts? I’d be happy to put together a PR with these changes.