Subfolders of Designated "Original" Folder Not Included In Search?

[ForestCat4]

Hi,
I have a very large "Sorted" message tree w/ a large number of subfolders, etc.
If I right-click and set that as Original, and then right-click another folder, "OldInbox" and then select Remove Duplicates on the context menu, none are found.
If instead I select (left-click) the Sorted folder, then CTRL-Left-Click the "OldInbox" to multi-select, then RightClick-->Remove Duplicates, a HUGE number of dups are found, as I'd expect.

Is this a known issue?

Thanks!

Windows 7/64
Thunderbird 102.15.1
Remove Duplicate Messages 0.5.4b5

[eyalroz]

No, it's not a known issue. But... when you set the originals - are you selecting the whole folder tree, or just its root folders? If you're doing the latter, try actually selecting all folder in that tree.

[ForestCat4]

Thanks. I upgraded to v115ESR over the weekend(trying to address .msf issues). Now I can't verify whether I had selected all folders, etc, since, for reasons only known to the developers and God, they appear to have removed the CTRL-Click function for folders. Does that mean that your "Original folder" funcion is also broken by TBird v115.xx? Does not appear to be working.

[ForestCat4]

I upgraded to v140xx to test further.

Windows 10
Betterbird v140.7.0esr-bb17
RemoveDuplicateMessages v0.6.4

Before I try to explain the behavior, Let's agree on some terminology, as it applies in the Windows environment.

Thunderbird Folder: An mbox file, which can contain messages but NOT subfolders.
they exist as NTFS files with the same name as the Thunderbird foldername within the account tree in the left pane. They normally would have no NTFS filename extension.

OS Folder: Appears in Thunderbird just like the mbox folders, however it is actual NTFS directory, which can contain mboxes, subfolders, or anything else allowed by the OS.
Thunderbird adds the .sbd extension to these folders when they are created within Thunderbird itself.

For this discussion, assume I am NOT using context menus, since they are still problematic in this version.

Assume we have two folder/message trees, under the main account:

SortedOriginal-
Contains many OS folders, and some Thunderbird mbox as well. Each of these OS folders may contain more OS folders, mboxes, etc. nested fairly deeply. Note that this folder has some deliberate duplication, for example email receipts for purchases of multiple items that fit in many categories. An example would be an Amazon email receipt for a toilet valve, spark plugs, and a pair of slippers. It would have been filed in Purchases\2021\Automotive, Purchases\2021\Clothing, Purchases\2021\Household. Hence the need to set this entire SortedOriginal folder tree as "Original" in the comparison & protect it from deletion of deliberate dups.

SortedIncoming- For the sake of argument, lt's call this an exact duplicate of SortedOriginal.

If I select(left-click) the SortedOriginal folder, then Tools-->Remove Duplicates,
It finds them.

If I select(left-click) the SortedIncoming folder, then Tools-->Remove Duplicates,
It finds them.

If I select(left-click) the SortedOriginal folder, then CTRL-Left-Click the SortedIncoming folder, then Tools-->Remove Duplicates,
It finds 20,000 dups. However without a way to mark/unmark by criteria such as parent folder, path, etc., there is no way to be sure the SortedOriginal folder folder is "protected".

So we have "Set Original Message Folders for next duplicate search", which is supposed to solve this issue.

However-
If I select(left-click) the SortedOriginal folder, then Tools-->Set Original Message Folders for next duplicate search, then select(left-click) the SortedIncoming folder, then Tools-->Remove Duplicates, It will NOT traverse the OS folders below the selected folders. It finds the dups with the Thunderbird(mbox) files at that highest level but no deeper.

Since RemoveDupes is DEFINITELY able to process deeply nested OS folders one at a time, or multi-selected, it seems something in the "Set Original Message Folders for next duplicate search" method is handling OS folder traversal differently?

I hope I've explained this well :-)

Thanks for reading!

[eyalroz]

, it seems something in the "Set Original Message Folders for next duplicate search" method is handling OS folder traversal differently?

Indeed it is. With originals folders, you have to mark the exact folders you want to consider for originals, and subfolders are not traversed as well. Now, given that it is possible for you to select all of the relevant subfolders - can you make an argument for why it is important to support doing this automatically? And remember that, for duplicate removal, we usually want to err on the side of caution, i.e. avoid finding dupes which we do not know for certain the user wants to be found; for which reason, if the user has not told us they want to find originals in the subfolders as well, it is prudent not to make that assumption.

[ForestCat4]

Hi, and thank you for getting back to me.  I think I remember seeing somewhere you had your hands full w/ some family-related stuff.  Sorry to hear about that, and apologies if I'm remembering wrong.  I have been buried in web searches related to removing duplicates in thunderbird for three entire months, I've lost track of what I've read/where. In any case really appreciate your work on this plugin.  Must be frustrating having the TBird developers breaking your code every couple of months.  I can relate. So... You mentioned: "Now, given that it is /possible/ for you to select all of the relevant subfolders" I'll be the first to admit that I can miss obvious things sometimes. In Thunderbird v140.xx, can you tell me how I can select a parent folder and EVERY subfolder, about 600 of them, nested dozens of levels deep? I have tried every combination of SHIFT, CTRL, etc I could think of.  All I have been able to do is select a range of "visible" folders, not the nested subfolders.  Even if there were some way to have Thunderbird completely expand the entire tree, "click top folder, scroll 6 feet, shift click the last visible folder" is a pretty poor workaround. If you know of a way to select a folder and all it sub-subfolders that you're willing to share, I would be very, very grateful. I've found no way to do this, even if I try to create a virtual "search folder", Thunderbirds folder selection dialog works contrary to every modern programming convention re: selecting/highlighting folders/subfolders.  While a do appreciate that Thunderbird is free, and the programmers are volunteers, I don't know what these people are thinking re: many of the ui design/function decisions. If there is NOT a way, then here is my argument/ use-case for this function in removeDupes: I have a huge, meticulously organized, deeply nested folder tree, called Archives. It has NO duplicates. It has exactly one copy of every email that I need to keep. I have MANY "unknown" profiles from multiple users/computers, that I am in the process of importing the mail folders from,  into the Thunderbird "archive" computer mentioned above. I need to compare these imported mail folders to the archive, and delete duplicates that I already have there, then deal with the remaining ones individually. I seems like a reasonable approach to me. Thank you for any thoughts/help on this.

___ Neil

On 3/20/2026 10:15 AM, Eyal Rozenberg wrote: *eyalroz* left a comment (eyalroz/removedupes#295) <#295 (comment)> , it seems something in the "Set Original Message Folders for next duplicate search" method is handling OS folder traversal differently? Indeed it is. With originals folders, you have to mark the exact folders you want to consider for originals, and subfolders are not traversed as well. Now, given that it is /possible/ for you to select all of the relevant subfolders - can you make an argument for why it is important to support doing this automatically? And remember that, for duplicate removal, we usually want to err on the side of caution, i.e. avoid finding dupes which we do not know for certain the user wants to be found; for which reason, if the user has not told us they want to find originals in the subfolders as well, it is prudent not to make that assumption. — Reply to this email directly, view it on GitHub <#295?email_source=notifications&email_token=ABPQKPW5WFP3COLXSVCJ2FT4RVHBBA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTIMBZHAZDMNBVHA42M4TFMFZW63VGMF2XI2DPOKSWK5TFNZ2KYZTPN52GK4S7MNWGSY3L#issuecomment-4098264589>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABPQKPXP2DY5K24F33EC5SL4RVHBBAVCNFSM6AAAAACRPGI5BSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHM2DAOJYGI3DINJYHE>. You are receiving this because you authored the thread.Message ID: ***@***.***>

-- Neil Nappe --==DiversiTech Consulting, LLC==-- ***@***.*** (732)583-4936

[eyalroz]

Please try your folder selection again with this beta build:

removedupes_0.6.6.0b3_tbird.xpi.zip