Email duplicates
Many individual emails and other documents appear multiple times within the dataset, so de-duplication is necessary for running queries and other analyses. We are identifying primary considerations and strategies at this point.
We have discussed a number of ways to identify duplicate emails. In general there are two ways a document could recur in the archive: if it was sent to multiple recipients whose emails were collected, or if it was included in a larger thread. Some emails or documents appear many times (probably over ten).
Reduce the list of total uts to only unique uts.
Different email clients may have display timestamps in varying specificities (minutes vs. seconds) and may round up or down. For example, an email actually sent at 5:30:17 PM may be displayed as such, with the seconds, or only to the minute and may round up or down to 5:30 PM or 5:31 PM respectively. To get around this, we round all uts to the hundreds place, so that any uts within 100 sec of another is interpreted as the same. e.g. 1431448357 and 1431448325 are both registered as 1431448300.
Another way to assure that two timestamps are actually duplicates of one another is to pair them with the senders associated with them and see if they are the same. It is unlikely that the same person sent multiple emails within 100 seconds (although I wouldn't put it past Brad). To do this, I wrote a code that sorts the data into a list of tuples, wherein the first item is the 'email'—represented by the uts and the sender—and the second item is a list of deq files where that email occurs.
I will explain how I sorted the data this way, following along with this notebook.
Step 1: I ran the function emails.getAll to get the text of the email. Importantly, this function does not just get the text of all the deqs, it also tags both the end of each page and the end of each bookmark with "---endPAGE" and "---BOOKMARKend---". This is important for step 2.
Step 2: I ran emailstext through the function machine. This function uses the page and bookmark endtags to split emailstext into a list of chunks, each chunk being either a bookmark (if 'bm' is indicated) or a page (if 'pg' is indicated).
For each chunk (bm or pg), the functions uts.getUnixTSr and dupl.getRawSenders_bm/pg generate a list of the rounded uts and senders found in that particular chunk. If the chunk is a page, then each of these lists will be put in a tuple with the page name (e.g. deq01_b33_3579_3579_1.txt). If the chunk is a bookmark however, the lists will be put in a tuple with the bookmark name or the 'root', where the pg number is replaced with an X to represent that this chunk includes all the pgs in the bookmark (e.g. deq01_b33_3579_3579_X).
The output is a list of tuples, where the unit (the bookmark or page file name) is the first item; the list of uts is the second item; and the list of senders is the third item. Ideally, we would have the same number of uts and senders, and uts[i] and the sender[i] would be of the same email (and this is the case for 95% of the emails right now).
Step 3: Why do we need to use both bookmarks and pages? For big bookmarks, we will use pages as the unit, but for small bookmarks we can use the bookmark itself. It is preferable to use entire bookmarks because oftentimes threads span several pages and a header may get split between two pages. In the enumeration method, this will mean that a uts and sender will end up associated with different file names.
However, when a bookmark is really big, a missed sender or uts will throw off the whole bookmark's enumeration. In this figure, sender2 is missing from the list (either it was not there in the first place, or it OCR'd improperly, or the function missed it). Thus, sender 3 will incorrectly be matched with uts2, and thus sender4 will also be matched incorrectly. For long lists, the ripple effect would be bad.
To compromise these two limitations, bookmarks with more than 10 pages will use pages and bookmarks with 10 or less pages will use bookmarks.
In the second part of step 3, we eliminate any lists that still have unequal list lengths. For now, this is about 5% of the files that need fixing. These two short and long lists are combined to give the output variable lpg_sbm.
lpg_sbm sorts the data by file.
Step 4: lpg_sbm is resorted using the function dupl.resortbyUTS which effectively just switches it so that the email is the unit of analysis now. The output em_deqf is a list of tuples, where each tuple is formatted as: ('email',[list of bookmarks/pgs where that email is present]). From this output we learn what is a unique email (unique pairing of rUTS and sender) and how many times that email occurs in the dataset (number of files in the list).
em_deqf sorts the data by email.
For TYPE A duplicates, we go to step 4A and for TYPE B duplicates to step 4B.
Step 5B:
As a note, for right now, any empty From: is given the placeholder NOSENDER and any empty Date:/Sent: is given the placeholder NOTIMESTAMP. These placeholders are necessary because they do represent the presence of an email header, but they cannot tell us about which ones are duplicates and which ones are unique. These are weeded out in the variable em_deqfB. Thus, we now have a list of unique emails and where they occur--those with more than 1 file are Type B duplicates.
Step 5A: This takes a similar rearranging and we end with the list in variable ems_deqfB. The output is structured similarly to em_deqfB, but instead of the unit being an individual email, it is a sequence of all the emails on that page or bookmark (which could still be only one). The output for em_deqfA is visualized below.
For files 1 and 4, there is only file (a pg for 1 and bm for 4) so it can be said to have no type B duplicates, unlike 2 and 3 which have multiple type B duplicates. 2 only contains one email, so all of the files listed are not only type B duplicates, but also type A duplicates. That email is also a type A duplicate to the first (highlighted) email in 3.