Reply-to wire format diverges from Columba — fields[0x30]/[0x31] vs fields[16] overload

[thatSFguy]

Summary

While inventorying the LXMF fields[16] (0x10) wire format across LXMF clients for a community-driven Reticulum/LXMF wire-format spec, I noticed that MeshChatX and Columba (torlando-tech/columba) agree on tap-back reactions but disagree on reply-to threading. This issue is about the reply-to divergence — the reactions side is in great shape on the MeshChatX implementation.

cc @markqvist as the LXMF protocol owner. None of this is a MeshChatX bug per se — upstream LXMF doesn't currently allocate any of 0x10, 0x30, or 0x31, so both clients are working within unallocated bytes. The question is whether the two clients want to converge before upstream blesses (or rejects) an allocation.

What works today — reactions

MeshChatX's src/backend/lxmf_utils.py:11 defines LXMF_APP_EXTENSIONS_FIELD = 16, and lxmf_utils.py:30 explicitly cites "Columba app_extensions" as the interop target. Send and parse both use the same shape:

fields[16] = {
    "reaction_to": "<target_message_id_hex>",
    "emoji":       "<unicode>",
    "sender":      "<source_identity_hash_hex>",
}

This matches Columba's send shape exactly, so MeshChatX → Columba reactions should land correctly. (There may be an open question about Columba's parse path — filed separately on torlando-tech/columba to confirm.) MeshChatX-to-MeshChatX is clean.

What conflicts — reply-to threading

MeshChatX uses two dedicated fields (meshchat.py:16697):

lxmf_message.fields[0x30] = bytes.fromhex(reply_to_hash)            # raw bytes hash
lxmf_message.fields[0x31] = reply_quoted_content.encode("utf-8")    # optional quoted text

Columba uses an overload of fields[16] (NativeMessageSender.kt):

fields[16] = {"reply_to": "<target_message_id_hex>"}

A Columba reply (fields[16] = {reply_to}) will not be detected as a reply by MeshChatX (which reads fields[0x30]), and vice versa. Once upstream LXMF allocates any of 0x10, 0x30, or 0x31 canonically, one of the two clients will be at odds with the canonical shape.

Tradeoffs as I see them

• MeshChatX's split approach uses two bytes but cleanly separates reply-target-hash from quote-text, and uses raw 32-byte hash (smaller than 64-char hex on the wire).
• Columba's overload approach uses one byte but couples reply-to to the same field as reactions, which is awkward semantically — and overloaded with fields[16] = {reaction_to, ...} requires the parser to dispatch on inner-dict keys.

If MeshChatX's split is the intended long-term shape, it'd be worth either getting upstream LXMF to allocate FIELD_REPLY_HASH = 0x30 / FIELD_REPLY_QUOTE = 0x31 (or whatever bytes upstream prefers), or coordinating with Columba so both clients use the same approach. The reverse direction (move MeshChatX onto Columba's fields[16] overload) is also a valid option.

I don't have a strong opinion on which approach should win — happy to support either consensus.

Cross-references

• Spec coverage discussion: Document fields[16] / [0x30] / [0x31] — tap-back reactions + reply-to (app extensions) thatSFguy/reticulum-specifications#8
• Proposed FIELD_REACTION = 0x10 patch for upstream LXMF (parked on a fork because markqvist/LXMF has interactions locked to collaborators): https://github.com/thatSFguy/LXMF/tree/field-reaction-0x10
• Companion issue on Columba about (1) a possible reaction send/parse mismatch and (2) the same reply-to conflict from Columba's side: Possible reaction send/parse shape mismatch in fields[16] — confirmation needed (+ reply-to divergence with MeshChatX) torlando-tech/columba#926

Source citations

• MeshChatX reaction field constant: src/backend/lxmf_utils.py:11
• MeshChatX reaction docstring referencing Columba interop: src/backend/lxmf_utils.py:30
• MeshChatX reaction send: meshchat.py:16805
• MeshChatX reaction parse: src/backend/lxmf_utils.py:352-358
• MeshChatX reaction tests: tests/backend/test_lxmf_reactions.py:18
• MeshChatX reply send: meshchat.py:16697
• Columba reaction send: reticulum/src/main/java/network/columba/app/reticulum/protocol/NativeReticulumProtocol.kt ~2230-2245
• Columba reaction parse (possible mismatched shape): app/src/main/java/network/columba/app/ui/model/MessageMapper.kt ~90-160
• Columba reply send: reticulum/src/main/java/network/columba/app/reticulum/protocol/NativeMessageSender.kt

[Sudo-Ivan]

I try to stick to the Zen of Reticulum with MeshChatX. While this is not always the case when first adding new features, I eventually do a second pass to ensure bandwidth efficiency. As for this issue, I prefer my approach over Columba's because it is more bandwidth-efficient.

[thatSFguy]

Thanks for the quick response — and good call invoking Zen of Reticulum, I should have led with that as the tiebreaker. Raw 32-byte hash + utf-8 quote bytes is meaningfully smaller than hex-string-in-dict-overload (~32+ bytes per reply saved, plus the msgpack dict-encoding savings for fields[0x30] vs fields[16] = {reply_to: ...}), so MeshChatX's approach has a real efficiency argument behind it on top of the cleaner field-separation.

I'll surface this on the Columba thread so @torlando-tech sees the rationale without having to dig — happy to let the two of you (and @markqvist) take it from there. No expectation that you move; if Columba ends up adopting the fields[0x30]/[0x31] split, the spec can describe that as the canonical shape for reply-to threading.

[torlando-tech]

Thanks, yeah I didn't realize MeshChatX had replies and reactions. I'll happily take the more bandwidth efficient approach in Columba, it'll just take some time.

More verbose version here: torlando-tech/columba#926 (comment)

[markqvist]

markqvist/LXMF@764758d