docs: additional notes on troubleshooting messages

Signed-off-by: Andrew Richardson <[email protected]>

Author: awrichar

doc-site/docs/troubleshooting/undelivered_messages.md

@@ -16,16 +16,20 @@ In general, FireFly messages come in three varieties:
 
 1. **Unpinned private messages:** private messages delivered directly via data exchange
 2. **Pinned private messages:** private messages delivered via data exchange, with a hash of the message recorded on the blockchain ledger
-3. **Pinned broadcast messages:** messages stored in IPFS, with a hash and reference to the message shared on the blockchain ledger
+3. **Pinned broadcast messages:** messages stored in IPFS, with a hash and reference to the message shared
 
-Note that all messages are batched for efficiency, but in cases of low throughput, you may frequently see batches
+All messages are batched for efficiency, but in cases of low throughput, you may frequently see batches
 containing exactly one message.
 
 "Pinned" messages are those that use the blockchain ledger for reliable timestamping and ordering. These messages have
 two pieces which must be received before the message can be processed: the **batch** is the actual contents of
 the message(s), and the **pin** is the lightweight blockchain transaction that records the existence and ordering of
 that batch. We frequently refer to this combination as a **batch-pin**.
 
+> Note: there is a fourth type of message denoted with the type "definition", used for things such as identitity claims
+> and advertisement of contract APIs. For most troubleshooting purposes these can be treated the same as pinned
+> broadcast messages, as they follow the same pattern (with only a few additional processings steps inside FireFly).
+
 ## Symptoms
 
 When some part of the multiparty messaging infrastructure requires troubleshooting, common symptoms include:
@@ -39,9 +43,13 @@ When troubleshooting one of the symptoms above, the main goal is to identify the
 experiencing an issue. This can lead you to diagnose specific issues such as misconfiguration, network problems, database
 integrity problems, or potential code bugs.
 
-In all cases, the **batch ID** is the most critical piece of data for determining the nature of the issue. This ID will be the
-same on all nodes involved in the messaging flow. The following two steps can be easily performed to check for the existence
-of the expected items:
+In all cases, the **batch ID** is the most critical piece of data for determining the nature of the issue. You can usually
+retrieve the batch for a particular message by querying `/messages/<message-id>` and looking for the `batch` field in the returned
+response. In rare cases, if this is not populated, you can also retrieve the message transaction via `/messages/<message-id>/transaction`,
+and then you can use the transaction ID to query `/batches?tx.id=<transaction-id>`.
+
+The batch ID will be the same on all nodes involved in the messaging flow. Therefore, the following two steps can be
+easily performed to check for the existence of the expected items:
 
 - query `/batches/<batch-id>` on each node that should have the message
 - query `/pins?batch=<batch-id>` on each node that should have the message (for pinned messages only)
@@ -50,18 +58,19 @@ Then choose one of these scenarios to focus in on an area of interest:
 
 #### 1) Is the batch missing on a node that should have received it?
 
-For private messages, this indicates a potential problem with **data exchange**. Check the sending node to see if the
-operations succeeded when sending the batch via data exchange, and check that the data exchange runtime is healthy.
+For private messages, this indicates a potential problem with **data exchange**. Check the sending node to see if the FireFly
+operations succeeded when sending the batch via data exchange, and check the data exchange logs for any issues processing it
+(the FireFly operation ID can be used to trace the operation through data exchange as well).
 If an operation failed on the sending node, you may need to retry it with `/operations/<op-id>/retry`.
 
-For broadcast messages, this indicates a potential problem with **IPFS**. Check the sending node to see if the
+For broadcast messages, this indicates a potential problem with **IPFS**. Check the sending node to see if the FireFly
 operations succeeded when uploading the batch to IPFS, and the receiving node to see if the operations succeeded when
 downloading the batch from IPFS. If an operation failed, you may need to retry it with `/operations/<op-id>/retry`.
 
 #### 2) Is the batch present, but the pin is missing?
 
 This indicates a potential problem with the **blockchain connector**. Check if the underlying blockchain node is
-healthy and mining blocks. Check the sending node to see if the operation succeeded when pinning the batch via the
+healthy and mining blocks. Check the sending FireFly node to see if the operation succeeded when pinning the batch via the
 blockchain. Check the blockchain connector logs (such as evmconnect or fabconnect) to see if it is
 successfully processing events from the blockchain, or if it is encountering any errors before forwarding those events
 on to FireFly.
@@ -71,7 +80,10 @@ on to FireFly.
 Check the pin details to see if it contains a field `"dispatched": true`. If this field is false or missing, it means
 that the pin was received but couldn't be matched successfully with the off-chain batch contents. Check the FireFly
 logs and search for the batch ID - likely this issue is in FireFly and it will have logged some problem while
-aggregating the batch-pin.
+aggregating the batch-pin. In some cases, the FireFly logs may indicate that the pin could not be dispatched because
+it was "stuck" behind another pin on the same context - so you may need to follow the trail to a batch-pin for a
+different batch and determine why that earlier one was not processed (by starting over on this rubric
+and troubleshooting that batch).
 
 ## Opening an issue