February 02, 2005

two of each

[He] held one finger up directly in from of Yossarian and demanded, "How many fingers do you see?"
"Two", said Yossarian.
"How many fingers do you see now?" asked the doctor, holding up two.
"Two", said Yossarian.
"And how many now?" asked the doctor, holding up none.
"Two", said Yossarian.
The doctor's face wreathed with a smile. "By Jove, he's right," he declared jubilantly. "He does see everything twice."

In Catch-22, by Joseph Heller.



For some mysterious reason, Thunderbird occasionally downloads my emails twice from the POP server. I think it may be related with the fact that I use to email clients (home and office) to suck emails from the server. But I also remember seeing this happen in Outlook in a previous life. So maybe it the server's fault rather than Thunderbird's.

Aaanyway, I decided to create an extension for Thunderbird that would delete duplicate emails automatically. Something that could be triggered from the context menu of a folder or account (and why not, as soon as an email arrives).

So the approach goes like this. The program iterates over every email in a folder or account and computes a hash for each email, using some function such as MD5. It then stores the hash in a map, associating it with the email (or rather, a reference to the email. We don't want to store the contents of every email in memory). But, if the map already contains an entry for that hash, it means we're probably facing a duplicate email. So the program compares the contents of both emails to rule out any chance of them having the same hash by coincidence (very unlikely, but possible).

So far so good. After a few cycles of writing and testing the Javascript code (writing mozilla extensions is messy, I tell you) I got to a point where it could detect duplicate emails (no deletion yet).

To my surprise, when I ran it the browser locked up for a few seconds. I eventually discovered what the problem was and I'll get to it after I describe how to program is structured.

There's a main function, deleteDuplicateMessages() that is invoked when the menu item is clicked on. This function iterates over the set of message headers (Thunderbird objects that contain metadata about each email) and invokes an asynchronous method to stream the contents of the email. The function takes an callback object as an argument. I believe they had to implement it like that to support IMAP, where getting the contents of an email could take a while.

The callback object gathers all the pieces of the email as it is streamed, strips out the headers and computes the MD5 hash. It then makes sure the entry does not exist in the hash map and exits.

The reason for the lockup is two-fold. First, there's the issue with the MD5 library I'm using, which takes a few seconds to compute the hash for really long strings.

The second issue seems to be that Javascript code in Mozilla executes in the UI thread, regardless of whether it's a callback function being invoked by a native component. So what's happening here is that the computation of the MD5 hash executes in the UI thread and prevents the Thunderbird from responding to user events.

I've been trying to find how to make that callback run in a separate thread, without much luck.

Another alternative would be to write the callback object in C++ as an XPCOM component, but I wanted to make the extension as portable as possible. Creating a native XPCOM component would mean that I'd have to compile it for every platform I wish my extension to support. Too bad.

The quest goes on.

19 Comments:

Anonymous Anonymous said...

Or just keep a hash of the Message-ID field, which is unique for each message. In your case, especially, where you are downloading the same exact message multiple times, this should be enough. #

It should also work even in cases where you're on the recipient list multiple times, like say a reply to a mailing list message, where you were cc'd personally on the reply. The advantage of using the Message-Id field is that it will be the same for those two messages, even though headers ("Received", etc) may be different (and thus your MD5's would be different).

should also be alot faster!

February 03, 2005 6:56 AM  
Anonymous Anonymous said...

I tried this on Netscape 7.2 and it works good.

sources of duplicate emails:
1) I have two email addresses: my company email address and a contractor email address for a different company. That company has software that is used for bug/question submission and unfortunately I get double emails everytime because when it collects info for the creation of the form it also collects all email addresses, adn then send that form to all email address.

2) I belong to several user group email aliases. When a person does a reply all to me many times I get an additional email copy since I am already part of the email alias.


I hate outlook

I could not find a program that could delete duplicate email messages in Netscape/Mozilla..until someone gave me your link

I was able to delete over 25,000 duplicate emails. The largest folder had 32000+ emails and it found just under 16,000 duplicate emails.

thank you!

May 31, 2005 9:58 PM  
Blogger Brian said...

Hey, you have a great blog here! I'm definitely going to bookmark you!

I have a business mom payout safelist site/blog. It pretty much covers business mom payout safelist related stuff.

Come and check it out if you get time :-)

October 26, 2005 4:46 PM  
Blogger Brian said...

Hey, you have a great blog here! I'm definitely going to bookmark you!

I have a free online classified advertising site/blog. It pretty much covers free online classified advertising related stuff.

Come and check it out if you get time :-)

October 29, 2005 1:06 PM  
Blogger Brian said...

Hey, you have a great blog here! I'm definitely going to bookmark you!

I have a free online advertising site/blog. It pretty much covers free online advertising related stuff.

Come and check it out if you get time :-)

November 03, 2005 2:20 AM  
Blogger Infactahost.com said...

Hey, you have a great blog here! I'm definitely going to bookmark you!

I have a advertising site/blog. It pretty much covers advertising related stuff.

Come and check it out if you get time :-)

November 03, 2005 9:51 PM  
Blogger Brian said...

Hey, you have a great blog here! I'm definitely going to bookmark you!

I have a guarantee marketing payplan safelist site/blog. It pretty much covers guarantee marketing payplan safelist related stuff. Plus you can advertise to billions using our guarantee marketing payplan safelist tools.

Come and check it out if you get time :-)

November 06, 2005 9:24 PM  
Blogger nasha said...

Why was there no follow on bankruptcy then? The bailout of AIG FP went to (wow power leveling) hedge funds that bound credit swaps on Lehman failing or others betting on rating (wow power leveling) declines. AIG has drained over 100 billion from the government. Which had to go to (wow power leveling) those who bet on failures and downgrades. Many of whom (power leveling)were hedge funds. I-banks that had offsetting swaps needed the money from the AIG bailout or they would have been caught. Its an (wow powerleveling) insiders game and it takes just a little bit too much time for most people to think (wow gold) through where the AIG 100 billion bailout money went to, hedge funds and players, many of whom hire from the top ranks of DOJ, Fed, Treasury, etc. ZHANG XIAO CHEN

April 19, 2009 9:56 PM  
Blogger danudan20 said...

r just keep a hash of the Message-ID field, which is unique for each message. In your case, especially, where you are downloading the same exact message multiple times, this should be enough. buy cialis , buy viagra , buy cialis , buy viagra , buy cialis , phentermine buy where to buy phentermine Buy Phentermine No Prescription Diet Pills Phentermine , swine flu level 6 , how much is insurance , Order Adipex Online, where to buy phentermine, buy phentermine 37.5, buy tramadol online It should also work even in cases where you're on the recipient list multiple times, like say a reply to a mailing list message, where you were cc'd personally on the reply. The advantage of using the Message-Id field is that it will be the same for those two messages, even though headers ("Received", etc) may be different (and thus your MD5's would be different).

June 20, 2009 11:28 AM  
Blogger Kayla said...

Cell phones from simple communication became a full-featured entertainment terminal . Can be said that cell phone china new milestone in the industry. This fully demonstrates that, cheap cell phones entertainment has become the most sought after consumer applications and has become the focus of the mobil phones market.

September 24, 2009 9:28 PM  
Blogger Kayla said...

Present-day society, the rapid development of science and technology makes cell phones wholesale is no longer a simple phone communications equipment only. Wholesale cell phones have many features, entertainment, scalability, and so have a good expression of the discount cell phones will be to maximize the effectiveness of multimedia technology.

September 24, 2009 9:29 PM  
Blogger Kayla said...

www.crazypurchase.com Web site for the majority of the china wholesalers, and buy products wholesale to provide information and buy wholesale from china information exchange platform dedicated to becoming China Wholesale.

September 24, 2009 9:29 PM  
Blogger Adi said...

Thank you for sharing.
Oes Tsetnoc | Mengembalikan Jati Diri Bangsa | Kenali dan Kunjungi Objek Wisata di Pandeglang | Oes tsetnoc | Online Marketing | Electronics Gadgets | etips solution | Travel Guide

October 25, 2009 8:11 AM  
Blogger The Jack said...

Thanks ever so much, very useful article. If you do not mind, please visit my article related to pandeglang district in Banten, Indonesia at Kenali dan Kunjungi Objek Wisata di Pandeglang or Kenali dan Kunjungi Objek Wisata di Pandeglang second and also Kenali dan Kunjungi Objek Wisata di Pandeglang Objek Wisata Air Terjun Curug Gendang or related to a leadership at Mengembalikan Jati Diri Bangsa and Oes Tsetnoc or Oes Tsetnoc the second and our hard work at Kerja Keras Adalah Energi Kita that's right Kerja Keras Adalah Energi Kita, and Kenali dan Kunjungi Objek Wisata di Pandeglang Memasuki Babak Akhir also Objek Wisata Taman Wisata Alam Carita, Kenali dan Kunjungi Objek Wisata di Pandeglang, or Kenali dan Kunjungi Objek Wisata di Pandeglang, also Kenali dan Kunjungi Objek Wisata di Pandeglang, or Kenali dan Kunjungi Objek Wisata di Pandeglang, also Kenali dan Kunjungi Objek Wisata di Pandeglang, or Kenali dan Kunjungi Objek Wisata di Pandeglang, also Kenali dan Kunjungi Objek Wisata di Pandeglang, or Kenali dan Kunjungi Objek Wisata di Pandeglang, very smart thank you!

October 28, 2009 11:01 PM  
Blogger Apa Saja Dah said...

Find and download what you need at Rapidshare Search Engine.
Top Site List Free Proxy Site Internet Marketing Tools Internet Marketing Auto Insurance Quotes Home Mortgage Loan Newest Gadgets Review Free Download mp3

October 31, 2009 9:23 PM  
Blogger Daniela said...

Find your great Travel News and sing the songs at Free Song Lyric or you can watch the drama at Korea Drama Online one of great korea drama is A Love to Kill if you go to travel to Indonesia learn Learn Indonesia Language first! And find your home cari rumah or make a blog Belajar membuat Blog find your home again rumah dijual and again at jual rumah then if you want buy a new laptop see the Laptop Price List or you can buy a New Blackberry and then take care your Health & Jewerly.

November 14, 2009 7:49 PM  
Blogger nike said...

Nice work and thanks!
Running
Adidas currently manufactures several running shoesNike shoes, including the adiStar Control 5, the adiStar Ride
Cheap nike shoes
Discount nike shoes
the Supernova Sequence and the Supernova Cushion 7, among others.
Nike shox r4
nike shox torch
nike shox shoes
Adidas also uses kangaroo leather to make their more expensive shoes.
Association football
One of the main focuses of Adidas is football kit and associated equipment.
puma cat
cheap sport shoes
Adidas also provides apparel and equipment for all teams in Major League Soccer. Adidas remain a major company in the supply of team kits for international football teams.
cheap nike shox
cheap nike max
Adidas also makes referee kits that are used in international competition and by many countries and leagues in the world. In the United States, referees wear the Adidas kits in MLS matches even though the primary referee supplier is Official Sports.
nike tn dollar
nike running shoes
The company has been an innovator in the area of footwear for the sport with notable examples including development of the Copa Mondial moulded boot on firm dry pitches for forty years.
nike air max tn
puma shoes
Adidas became renowned for advancing the "Predator" boot design.This design featured a ribbed rubber structure for the upper leather of the shoe, used to accent the movement of the ball.
discount puma shoes
puma mens shoes
The Predator also features the Craig Johnston invented "Traxion" sole. As the development and popularity of Football continued Adidas played a leading role in shaping the style of the play itself.
puma running shoes
puma shoes
FIFA, the sports governing body, commissioned specially designed footballs for use in its own World Cup tournaments to favour more attacking play.
ghd hair straighteners mk4
hair straightners
ghd iv styler hair straightener
ghd hair straightners
cheap ghd hair straighteners

December 09, 2009 5:50 PM  
Blogger nike said...

Clothing has always been a thing that has been given a great importance by human beings. It displays the attitude that the people exhibit.
ed hardy clothes
ed hardy shirts
Lots of brands have been existent in the clothing industry and a famous one among them is the Ed hardy clothing brand.
ed hardy jackets
ed hardy hoodies
The brand got its name from the famous American tattoo artist Ed Hardy. He was a very famous tattoo artist and has published many books on tattooing techniques.
ed hardy boots
ed hardy polo shirts
But his tattooing turned into a brand by the efforts of a company called Christian Audiger. This company was a very famous and very powerful company in the field of clothing.
ed hardy shoes
ed hardy jeans
They felt that it would be appropriate to create a brand called Ed hardy and use Hardy's art as the main selling point for the brand. This venture had turned out to be a very successful one and Ed hardy clothing is one of the most famous brands in the clothing industry.
ed hardy outerwear
ed hardy long sleeve shirts
The brand became very famous because it was worn by many famous celebrities like Madonna, Britney spears and also Sylvester Stallone.
ed hardy bags
ed hardy winter boots
The brand has clothes for men, women, and kids. They have also diversified their business by having a lot of accessories to support their clothing business.
ed hardy love kills slowly shirts
ed hardy love kills slowly shoes
ed hardy love kills slowly boots
for men they have a variety of products such as active wear, denim, outwear, swim trucks, sweaters, t-shirts, tops etc.
ed hardy trousers
ed hardy mens
the accessories include things such as belts, caps, scarves, shoes, socks, jeweler, sunglasses, ties and even wallets.
The art works is a unique combination of American and Japanese cultures. The unique selling proposition of the brand is the way in which the company has used the art works of Ed hardy.
ed hardy womens
ed hardy t shirts
ed hardy sunglasses

December 09, 2009 5:51 PM  
Blogger uwanna said...

gucci replica handbags
men gucci shoes
Gucci men sneakers
Gucci men moccasins
gucci women sneakers
gucci women boots
Gucci men boots
Gucci
Gucci shop
Gucci bags
Gucci shoes
Replica GUCCI SHOES
wholesale gucci shoes
cheap Gucci handbags
Gucci ON sale
Gucci Belts
Gucci small accessories
Gucci hats & scarves
Gucci wallets
Gucci Handbags
Women Gucci shoes
Men Gucci shoes
discount gucci shoes
cheap Gucci shoes

February 04, 2010 9:50 PM  

Post a Comment

Links to this post:

Create a Link

<< Home