Pre-messages

Motivation

• Make background-fetch work reliably. On iOS, we somehow even have crashes because of timeout or memory exhaustion.
• This will also make calls arrive more reliably.
• Also, this will make download-on-demand more user-friendly. Currently, pre-messages are often assigned to the wrong chat (1:1 email chat with the sender, rather than group chat or 1:1 encrypted chat). Also, we'll be able to directly show more of the message, e.g. the text.
  • previous discussions, mainly focused on chat assignment We don't know any metadata of large messages before fully downloading #5888
• On the chatmail server, we'll be able to delete large messages more eagerly, because the recipient will at least have an indication that there was a missed message.

Implementation

• No UI changes should be needed in the beginning; from the UI side, everything should stay the same except that the pre-message is directly assigned to the correct chat. (and possibly that the text is already there before clicking "Download")
• Ideally, it will be backwards-compatible, i.e. messages sent with the new version of DC look OKish on old versions. It's probably OKish if message content is shown twice.
• In the beginning, we'll only care about big attachments, and ignore the case of very long text messages that exceed 160KB.
• Note that there may be existing message-stubs that are not downloaded yet. After updating, the "Download" button will probably become non-functional; this is fine. We should make sure that DC doesn't crash when clicking "Download".
• After this change all large messages without the new Chat-Is-Full-Message header are downloaded regardless of download setting, especially classical emails. But this only happens when the user has the app in foreground.
  • so the download size setting would only apply to messages sent by the new version of delta chat.

Steps

• Remove "Download maximum available until", because it's unreliable - chatmail servers may delete full-messages quicker. feat: Remove "Download maximum available until" and remove stock string DC_STR_DOWNLOAD_AVAILABILITY #7369

• Remove partial-download remove: partial downloads - creation of the stub messages #7373

• Sending messages.

  • All encrypted messages with an attachment get a pre-message. We can exclude attachments smaller than e.g. 10KB or 50KB or 100KB.
  • The pre-message needs to reference the full-message in an encrypted header, so that the full-message can be downloaded when the user requests it. The header can be called Chat-Full-Message-Id.
  • The full-message gets a cleartext header Chat-Is-Full-Message
  • The best place to check whether the message should be split up into two is probably create_send_msg_jobs().
  • The message's state must be set to MessageState::OutDelivered only after both messages are sent. If a read receipt is received, the message can be OutMdnRcvd or OutPending; let's just do whatever is easiest for now. Take care not to revert from OutMdnReceived to OutDelivered if we first receive a read receipt and then deliver the full message.
  • Autocrypt-gossip and selfavatar should never go into full-messages, because that would just make it unnecessarily large

• Receiving messages.

  • In receive_imf, the full-message can be handled similarly to Chat-Edit (grep for "ChatEdit" in receive_imf.rs).
  • As opposed to the current download-on-demand, we won't replace the whole msgs row, because this way, it will just work if an edit-message is received, and then later the user requests to download the attachment.
    • Need to change Viewtype, probably metadata params like image size
  • pre-message gets Viewtype::Text in the database until downloaded
  • background_fetch should not download messages with the Chat-Is-Full-Message header. If a large message without this flag is received, the UI should be notified via an event.
  • A read receipt should be shown already when the user saw the pre-message.

• check if we re-used partial_download_msg_body stock string, if not, then remove it

Future possibilities (out of scope right now)

• Nicer pre-message with an image thumbnail etc.
• Care about very long text messages that exceed 160kb:
  • We could just deny sending this large messages
  • Or we could introduce some extra complexity to make it just work.
  • We could emit an event when we ignore messages >1MB in background_fetch (both classical emails and very large text messages)
• Show pre-messages in the UI as having the correct file type, showing that this file isn't downloaded yet
• If the full-message only contains an attachment, include a blake3 hash of the attachment into the pre-message. Then, if we already have this file, we don't need to download it.
• Request full-messages from other devices via iroh (either your second device or the sender's device)

Download logic pseudo-code

Logic with Chat-Is-Full-Message header (or Chat-Has-Premessage or Chat-Can-Be-Safely-Ignored-For-Now),
unconditionally downloading classic email & messages from old clients:

// with this logic, classic email will be downloaded unconditionally, ignoring config.download_limit
// simplification is not only in this code, but mainly in the removed partial_download logic

download_when_normal_starts = [] // might be sql table "download"
available_full_msgs = [] // new sql table - all full-messages we've seen on the server but not downloaded yet. Remember to remove messages from here when deleted


background_fetch()
{
    while (msg = get_message_prefetch_header())
    {
        if !msg.fullMessage {
            if msg.size < 1MB {
                download_message() // may be a pre-message or a pure-text message
            } else {
                // This is e.g. a classical email
                // Queue for full download, in order to prevent missing messages
                download_when_normal_starts.push(msg.rfc724_mid)
            }
        } else {
            // This is a full-message
            if msg.size < config.download_limit {
                download_when_normal_starts.push(msg.rfc724_mid)
            }
            available_full_msgs.push(msg.rfc724_mid)
        }
    }

    // optionally, maybe later, download larger messages if time
    while (msg = download_when_normal_starts.pop()) {
        // ... (like in normal_fetch())
    }

    // optionally, in order to guard against lost pre-messages:
    while (msg = available_full_msgs){
        if !premessage_is_downloaded_for(msg.rfc724_mid) {
            // Download the full-message unconditionally,
            // because the pre-message got lost.
            // The message may be in the wrong order,
            // but at least we have it at all.
            download_msg()
        }
    }
}


normal_fetch()
{
    while (msg = get_message_prefetch_header())
    {
        if !msg.fullMessage {
            download_message()
        } else if msg.size < config.download_limit {
            download_message() // this may be a full-message replacing a pre-message
        } else {
            available_full_msgs.push(msg.rfc724_mid)
        }
    }

    while (msg = download_when_normal_starts) {
        let res = download_message()
        if res.is_ok() {
            download_when_normal_starts.remove(msg)
            available_full_msgs.remove(msg)
        }
        if res.is_err() {
            if !premessage_is_downloaded_for(msg.rfc724_mid) {
                warn!() // This is probably a classical email that vanished before we could download it
                download_when_normal_starts.remove(msg)
            } else if available_full_msgs.contains(msg.rfc724_mid) {
                // set the message to DC_DOWNLOAD_FAILURE - probably it was deleted on the server in the meantime
                download_when_normal_starts.remove(msg)
                available_full_msgs.remove(msg)
            } else {
                // leave the message in DC_DOWNLOAD_IN_PROGRESS;
                // it will be downloaded once it arrives.
            }
        }
    }

    // optionally, in order to guard against lost pre-messages:
    while (msg = available_full_msgs) {
        if !premessage_is_downloaded_for(msg.rfc724_mid) {
            // Download the full-message unconditionally,
            // because the pre-message got lost.
            // The message may be in the wrong order,
            // but at least we have it at all.
            res = download_msg()
            if res.is_ok() {available_full_msgs.remove(msg)}
        }
    }
}