Bulk Delete Messages with index.json fails.

[RyouFuchiKyou]

Undiscord Version

v5.2.3

Browser

1.60.110 Chromium: 119.0.6045.105 (Official Build) (64-bit)

Extension

TamperMonkey

Reproduction steps

Uploaded index.json
Error code.

Actual results

Squat. Error code.

Error searching messages, API responded with status 403!
{"message":"Missing Access","code":50001}
CoreException {}

Expected results

Start deleting all dm's.

Bug affects

• DMs
• DMs (group)
• Channels
• Channel (NSFW)

Additional information & file uploads

[RyouFuchiKyou]

Something is happening within the file. I had to personally edit it down to DM ID's only then paste it in by hand.

[SeraphCoding]

I did some digging.
Essentially:
The error handling in the script is quite lackluster - instead of skipping failed messages, the entire thing just crashes.

It will fail processing the index.json when the following is true:

• The name of the chat is "null" (a lot of them)
• The messages are on a server you do not have access to anymore (which will be a lot of them)
• The messages are in a thread that has been archived (also access)
• The messages are in a channel on a server that you DO have access to but the channel has been deleted/archived/changed in a way where you lost access

Those are the main causes I could identify.

[yazanzaid00]

Temporary Workaround

Due to limited time, I couldn’t thoroughly debug the issue and solve it more elegantly. It appears that undiscord doesn’t properly manage abandoned/deleted servers, channels, or messages. As a temporary solution, I’ve bypassed all errors, allowing the code to continue to the next message even if an error arises. Please be aware that this code should be used at your own risk. I didn’t review the entire undiscord code; I began troubleshooting from the recurring error message and attempted to catch the error before the program exited.

Modifications Made

1. I modified the runBatch function to ensure it continues to the next message even if an error is encountered:

async runBatch(queue) {
  if (this.state.running) return log.error('Already running!');

  log.info(`Runnning batch with queue of ${queue.length} jobs`);
  for (let i = 0; i < queue.length; i++) {
    const job = queue[i];
    log.info('Starting job...', `(${i + 1}/${queue.length})`);

    // set options
    this.options = {
      ...this.options, // keep current options
      ...job, // override with options for that job
    };

    try {
      await this.run(true);
    } catch (error) {
      log.error('Error in job', `(${i + 1}/${queue.length})`, error);
    }

    if (!this.state.running) break;

    log.info('Job ended.', `(${i + 1}/${queue.length})`);
    this.resetState();
    this.options.askForConfirmation = false;
    this.state.running = true; // continue running
  }

  log.info('Batch finished.');
  this.state.running = false;
}

2. I removed the explicit throw. I also employed a try-catch block on !resp.ok to prevent exceptions from being thrown:

	    if (!resp.ok) {
	      // searching messages too fast
	      if (resp.status === 429) {
			// existing error handling code...
	      }
		  // Replace only this current else block
		  else {
			const body = await resp.text();
			try {
			  const r = JSON.parse(body);
			  if (resp.status === 400 && r.code === 50083) {
				// existing error handling code...
			  } else {
				log.error(`Error deleting message, API responded with status ${resp.status}!`, r);
				log.verb('Related object:', redact(JSON.stringify(message)));
				this.state.failCount++;
				// Instead of throwing an error, just return a failure status
				return 'FAILED';
			  }
			} catch (e) {
			  log.error(`Fail to parse JSON. API responded with status ${resp.status}!`, body);
			  // Again, instead of throwing an error, just return a failure status
			  return 'FAILED';
			}
		  }
	    }

[SeraphCoding]

Temporary Workaround

Due to limited time, I couldn’t thoroughly debug the issue and solve it more elegantly. It appears that undiscord doesn’t properly manage abandoned/deleted servers, channels, or messages. As a temporary solution, I’ve bypassed all errors, allowing the code to continue to the next message even if an error arises. Please be aware that this code should be used at your own risk. I didn’t review the entire undiscord code; I began troubleshooting from the recurring error message and attempted to catch the error before the program exited.

Modifications Made

1. I modified the `runBatch` function to ensure it continues to the next message even if an error is encountered:

async runBatch(queue) {
  if (this.state.running) return log.error('Already running!');

  log.info(`Runnning batch with queue of ${queue.length} jobs`);
  for (let i = 0; i < queue.length; i++) {
    const job = queue[i];
    log.info('Starting job...', `(${i + 1}/${queue.length})`);

    // set options
    this.options = {
      ...this.options, // keep current options
      ...job, // override with options for that job
    };

    try {
      await this.run(true);
    } catch (error) {
      log.error('Error in job', `(${i + 1}/${queue.length})`, error);
    }

    if (!this.state.running) break;

    log.info('Job ended.', `(${i + 1}/${queue.length})`);
    this.resetState();
    this.options.askForConfirmation = false;
    this.state.running = true; // continue running
  }

  log.info('Batch finished.');
  this.state.running = false;
}

2. I removed the explicit throw. I also employed a try-catch block on !resp.ok to prevent exceptions from being thrown:

	    if (!resp.ok) {
	      // searching messages too fast
	      if (resp.status === 429) {
			// existing error handling code...
	      }
		  // Replace only this current else block
		  else {
			const body = await resp.text();
			try {
			  const r = JSON.parse(body);
			  if (resp.status === 400 && r.code === 50083) {
				// existing error handling code...
			  } else {
				log.error(`Error deleting message, API responded with status ${resp.status}!`, r);
				log.verb('Related object:', redact(JSON.stringify(message)));
				this.state.failCount++;
				// Instead of throwing an error, just return a failure status
				return 'FAILED';
			  }
			} catch (e) {
			  log.error(`Fail to parse JSON. API responded with status ${resp.status}!`, body);
			  // Again, instead of throwing an error, just return a failure status
			  return 'FAILED';
			}
		  }
	    }

Did something similar to cope with the failures and just skip them. In all honesty, explicit error handling for these messages is only beneficial for logging purposes in the first place, so the user "knows" that there are some unreachable messages.

So just skipping and continuing is the way to go.

[vanwarcho]

@yazanzaid00 Can you send the full modified code since this isnt working for me

[athaarv]

@yazanzaid00 Can you send the full modified code since this isnt working for me

Thanks to @yazanzaid00 and @SeraphCoding for the corrected code snippets above.

While testing those changes, I found that Discord starts rate limiting the script when it's re-executed on chats where it had previously stopped partway through. For instance, if the script successfully runs through chats 1–30 but only deletes part of chat 31 (due to the search results coming back empty from Discord’s API), it will move on prematurely to chat 32. If you later re-run the script using the index.json method, it ends up hammering Discord with a burst of requests on already-cleared chats, which triggers rate limiting and causes the entire process to fizzle out. To fix this, I've added a delay between each batch run to prevent being throttled.

I've incorporated all these fixes into my own fork of the repo, which you can find the fully modified version here for now:
https://github.com/athaarv/undiscord

I've opened a pull request (#722) and hope @victornpb accepts it

[SeraphCoding]

@yazanzaid00 Can you send the full modified code since this isnt working for me

Thanks to @yazanzaid00 and @SeraphCoding for the corrected code snippets above.

While testing those changes, I found that Discord starts rate limiting the script when it's re-executed on chats where it had previously stopped partway through. For instance, if the script successfully runs through chats 1–30 but only deletes part of chat 31 (due to the search results coming back empty from Discord’s API), it will move on prematurely to chat 32. If you later re-run the script using the index.json method, it ends up hammering Discord with a burst of requests on already-cleared chats, which triggers rate limiting and causes the entire process to fizzle out. To fix this, I've added a delay between each batch run to prevent being throttled.

I've incorporated all these fixes into my own fork of the repo, which you can find the fully modified version here for now: https://github.com/athaarv/undiscord

I've opened a pull request (#722) and hope @victornpb accepts it

This may be unrelated to this issue, but it now also makes sense to create a separate logfile/csv file with unreachable messages in the bulk deletion request format for Discords DPO - specifically for those messages that are actually present but the user does not have access to anymore.

Discords Data Privacy Officer no longer accepts bulk deletion requests for messages that are accessible by and therefore deletable by the user - at least they say they don't. Whether that is a legal denial of the request, I won't be able to judge here.

However - it is possible to check messages from an archive and why they error out. If they are unreachable, might as well write them to a separate file and send them to Discords DPO for bulk deletion, because you cannot delete them yourself.

Also good work implementing a fix, I am dumb and did not commit my fixes with similar changes to my backup fork lol @athaarv