February 04, 2005

Duplicate Message Remover 0.1

Here's the extension for Thunderbird I was working on. It removes duplicate messages in any given folder and moves them into the Trash (the operation can be undone). The function can be accessed by right clicking on a folder and selecting "Delete Duplicate Messages".

Installation instructions

  • Download the extension to your computer (Right-click on the link, and choose "Save Link As" if you're using Firefox. Otherwise, Firefox may try to install it itself)
  • In Thunderbird, go to Tools->Extensions and click on the Install button. Browse to the file you just downloaded and click on Open.
  • Restart Thunderbird... and you're done!

By the way, the extension requires Thunderbird 1.0

Known Limitations

  • The extension works on folders only. Executing the function on an account will do nothing.
  • It does not recurse into folders. If a folder contains subfolders you'll need to run it on each subfolder manually.
  • Since the extension analyzes each folder independently, messages duplicated across folders won't be detected.


Thanks to the anonymous poster for the suggestion regarding the message IDs. It turned out that the IDs can be obtained via Thunderbird APIs (they are stored in the metadata database). There's no need to parse the emails at all.

Update: a new version compatible with Thunderbird 1.5 can be found here.

52 Comments:

Blogger Dere said...

Thanks for the Extension.
As i have very great archives i will test it.
But i have one problem. Maybe we can improve the extension together. A few ideas from my point of view:
The problem is, there are mails, without msg-ids (very old, or imported mails from other programms) and mails with a modified header (due to import, or due to the fact of a later download than the first mail, so header dates differ) or mails with a removed attachment.

In my opinion, the best solution is your solution.
But, for users with a greater mailstorage, it may be a even better extension if there's a possibility to switch between a quick search and a detailed search for dupes.

Another idea: if you move both mails (the original one, and the dupe) to a seperate folder (which is selected by the user), the user is able to do a review by himself. Maybe this should be an option too. This would enable the processing to the original place, if all dupes are deleted, by filters.

In the end, if it works, and all bugs are cleaned out, it is possible to do the exact opposite.
And this opposite is called Sync. Select your mails, select another mailstorage, and Sync the mails, that are missing in the other folder....

I have planned an extension that does the same thing.
Maybe we could work together?

If you're interested, mail me.
And sorry for my bad english grammar,
it isn't my native language.

Another Thunderbird user
Dere

February 05, 2005 6:25 PM  
Anonymous Anonymous said...

not working on non-english character

February 05, 2005 9:17 PM  
Anonymous Anonymous said...

Great Extension!
But I had similar problems as dere: there where many messages without msg-ids and some of them got moved to trash although they are not duplicate - just the same subject.

I think deres ideas how to handle this are really good. One other idea would be to label the emails which are believed to be duplicate e.g. as ToDo so it would be easy to find them in den folder and remove them on your own.

As well sync would be really cool!

February 13, 2005 11:22 AM  
Blogger Ainslee said...

I'm not sure if I'm doing something wrong with the extension but it seems to be deleting emails that aren't even duplicates?

February 26, 2005 2:54 AM  
Anonymous Anonymous said...

Thanks, but it removed about 5000, out of the 6000 messages I had. Of course, I emptied the trash, and lost everything.

I guess all the messages I recently imported from Outlook wouldn't have message ID's, thus would get removed. Luckily I can reimport the .PST.

While you do move everything to the trash, I think you should mention that your extension does not work for imported mail in your extension description.

Perhaps you could code the extension to make two passes? One for message ID, the other for content, etc comparison.

Anyway, thanks for the great extension. I'll be checking frequently for updates...

March 07, 2005 11:06 AM  
Blogger Ben Klausner said...

This is a good start. But what about deleting dupes in the Trash folder? And/or maybe an option to dump dupes into /dev/null in the first place?

March 18, 2005 10:03 PM  
Anonymous Anonymous said...

Very nice extension - I'm using it to solve the duplicate RSS message problem in Thunderbird at the moment.

I've extended it a little to recurse into subdirectories, which means that you can invoke it on "News & Blogs" and get rid of all dupes in all of your RSS feeds at once.

Unfortunately, I don't know how to use code, pre or tt tags in blogger, so excuse the messy code (from deleteDuplicates.js):

function deleteDuplicateMessages()
{
  var folders = GetSelectedMsgFolders();
  for (var i in folders) {
    deleteDuplicateMessagesInFolder(folders[i]);
  }
}

function deleteDuplicateMessagesInFolder(folder)
{
  // depth first recursion
  if (folder.hasSubFolders) {
    var subFolders = folder.GetSubFolders();
    var done = false;
    while (!done) {
      var item = subFolders.currentItem();
      if (item) {
        var subFolder = item.QueryInterface(Components.interfaces.nsIMsgFolder);
        if (subFolder) {
          deleteDuplicateMessagesInFolder(subFolder);
        }
        try {
          subFolders.next();
        } catch (ex) {
          done = true;
        }
      }
    }
  }

  var messages;
  try {
    messages = folder.getMessages(msgWindow);
  } catch (ex) {
    // don't have any messages, just return
    return;
  }

  var messageTable = new Array();

  var messagesToDelete = Components.classes["@mozilla.org/supports-array;1"].createInstance(Components.interfaces.nsISupportsArray);
  
  while (messages.hasMoreElements()) {
    var message = messages.getNext().QueryInterface(Components.interfaces.nsIMsgDBHdr);
  
    var id = message.messageId;

    if (id == null) {
      alert("Message has no ID: " + this.uri + ":" + message.mime2DecodedSubject);
      continue;
    }

    if (messageTable[id] != undefined) {
      messagesToDelete.AppendElement(message);
    }
    else {
      var uri = folder.getUriForMsg(message);
      messageTable[id] = uri;
    }
  }

  folder.deleteMessages( messagesToDelete, msgWindow, false, false, null, true);

}

April 08, 2005 1:20 AM  
Anonymous Anonymous said...

Removal by MessageID may be fast, but that's not what most people think of as a duplicate message =)
I would like to ask for an option to do removal only based on Sender, Subject and content (probably what most people would consider a duplicate message if they match...). That would eg get rid of dupliucates caused by someone sending a message to you twice by accident.

April 15, 2005 2:06 AM  
Anonymous Anonymous said...

Dude,

This is a *vital* extension, you should log it into Mozdev so more can find it. I was looking just for this and it took a couple hours to find it, if on Mozdev it would be downloaded by thousands.

I've put something up on Mozdev to complement what you've done here. The 'sent' folder doesn't work with your extension due to a lack of headers. I've written something in VB that is just for this folder.

Thx for the work, it is a must have extension.

http://fatdrunkandstupid.org/mozilla/dupedelete.zip

April 15, 2005 6:37 PM  
Anonymous Anonymous said...

Hi,

i have many mails which I call duplicated as they have exactly the same contents, but they are send by two independant senders. But the text(s) of the mails are absolutely identically to each other.

I downloaded your extension as I thought it would remove those duplicated messages, but it doesn't...

Meanwhile, I know why -- you just delete copies of the same mail by the message-id but not two independant mails with the same contents.

I think you could offer an option (by radiobutton) which functionality the user wants to have...

Greetings,

Torsten

June 16, 2005 3:35 PM  
Anonymous Anonymous said...

Thank you very much for your work on this extension. It works well and it's highly recommended.

June 20, 2005 9:46 AM  
Anonymous Anonymous said...

Great tool!!! Congratulations!!! Is the best!!! :o)

I looking for a application as this since Netscape Navigator 4.5. Thank you!

If it will be easy, would like to suggest an optional config to colorize the messages duplicates/pairs in the proper folder using the system of labels of the Thunderbird!

Regards,

Andrey
Brazil

June 21, 2005 12:07 PM  
Anonymous Anonymous said...

Great tool. thanks a million
Murali

July 04, 2005 8:15 AM  
Anonymous Anonymous said...

This is a fabulous little piece of work. Thanks so much, and I agree wholeheartedly with the poster who mentions logging this into MozDev.

BTW, this works perfectly well with Mozilla Suite, and didn't even complain during the install.

As I test many builds of Moz, I tend to experience unexpected shutdowns (okay, crashes!) of the app more often than most. An unfortunate side-effect of this is that sometimes, popstate.dat gets wiped out in the process, and while I do have my server settings adjusted to keep messages on the server for several days, I get a large volume of messages. This usually results in my having to spend (waste) even more time culling through duplicate messages, and not just in my Inbox, as I have a multitude of filters configured. Oy!!

Anyway, again, thanks.

--
Lewis
------------------------------------------------------------
Lewis G Rosenthal, CNA, CLE
Rosenthal & Rosenthal, LLC
Accountants / Network Consultants
New York / Northern Virginia www.2rosenthals.com
eComStation Consultants www.ecomstation.com
Novell Users International www.novell.com/linux/truth

Warpstock 2005 - Hershey, Pennsylvania, October 6-9, 2005 www.warpstock.org
------------------------------------------------------------

Mozilla/5.0 (OS/2; U; Warp 4.5; en-US; rv:1.8b2) Gecko/20050626 MultiZilla/1.8.1.0h

September 11, 2005 8:00 PM  
Blogger Unknown said...

Interesting comment concerning 1.5.x. I use SeaMonkey 1.5a (which is based on the 1.9 branch); I'm assuming form the version that the earlier poster was referring to Thunderbird 1.5. This continues to work perfectly well for me under SeaMonkey 1.5a (2007-01-12 nightly).

--
Lewis
------------------------------------------------------------
Lewis G Rosenthal, CNA, CLP, CLE
Rosenthal & Rosenthal, LLC
Accountants / Network Consultants
New York / Northern Virginia www.2rosenthals.com
eComStation Consultants www.ecomstation.com
Novell Users Int'l www.novell.com/openenterpriseserver
Need a managed Wi-Fi hotspot? www.hautspot.com
------------------------------------------------------------

January 16, 2007 5:21 PM  
Blogger publichery25244 said...

This comment has been removed by a blog administrator.

April 10, 2007 2:35 AM  
Anonymous Anonymous said...

wholesale jewelryhandmade jewelryjewelry wholesalefashion jewelrycostume jewelry

May 06, 2009 2:25 AM  
Blogger Unknown said...

Thanks for the Extension.

June 20, 2009 4:44 AM  
Anonymous Anonymous said...

無料 出会い 競馬予想 無料 競馬予想 競馬予想 無料 競馬予想 無料 競馬予想 無料 近視 手術 メル友 出会い 出会い 出会い 出会い メル友 メル友 人妻 メル友 ギャンブル依存症 AV女優 無料 出会い 出逢い 掲示板 出会い系 無料 出会い 人妻 出会い 人妻 出会い セフレ 人妻 出会い セックスフレンド メル友 出会い SM 愛人 不倫 セフレ 無料 出会い 出会い系 無料 無料 出会い 富士山 写真 富士山 メル友 無臭性動画 カリビアムコム 一本堂 出会い 人妻 セックスフレンド ハメ撮り エッチな0930 メル友 無料 出会い 無料 出会い セフレ セフレ セフレ セックスフレンド セックスフレンド セックスフレンド 人妻 出会い 人妻 出会い 人妻 出会い 出会い系 出会い系 出会い系 カリビアンカム カリビアンカム

July 24, 2009 2:10 AM  
Blogger reen said...

Thanks for the plugin. It will be very useful. Thanks again. It is very hard to earn money chao!

August 07, 2009 1:21 AM  
Anonymous Anonymous said...

Wow...this was a really good article. Probably one of the best I've read on

online degree program | affordable degree | life experience degrees

August 26, 2009 12:35 AM  
Anonymous Anonymous said...

I have been reading your blog last couple of weeks and enjoy every bit. Thanks.


nursing degree | online doctoral degree

August 26, 2009 12:36 AM  
Anonymous Anonymous said...

Buy Viagra
hey i think that is a good idea keep up the good work!!

September 03, 2009 10:25 AM  
Anonymous Anonymous said...

メル友募集掲示板人妻 出会いを探す。
処女逆援助援交関係でSEXする。

October 13, 2009 7:37 PM  
Anonymous Anonymous said...

花蓮租車推薦,花蓮 租車,租車 花蓮,花蓮 租車 推荐,花蓮行易租車,花蓮市 租車,花?旅游网,花?租?,旅遊景點花蓮,一日遊行程,花蓮地圖,花蓮包車,花蓮租車網,花蓮 租車 服務,花蓮市租車,花蓮租車,花蓮租車,花蓮旅遊景點,租車,花蓮旅遊行程,花蓮旅遊地圖,花蓮包車,推薦 旅遊,花蓮住宿推薦,花蓮租車,花蓮賞鯨,花蓮旅遊,花蓮,花蓮民宿資訊網,花蓮租車,花蓮租車,花蓮租車,花蓮旅遊,花蓮租車行,花蓮租車優惠,花蓮賞鯨,花蓮行易租賃車行,花蓮車站 租車,花蓮賞鯨,花蓮泛舟,花蓮溯溪,花蓮旅遊,花蓮旅遊景點,花蓮旅遊行程,花蓮旅遊地圖,花蓮旅遊租車,花蓮市租車,花蓮租車 推薦,花蓮租車..,花蓮租車,花蓮租車,花蓮租車,花蓮租車,花蓮旅遊一日遊,行易租車,花蓮市租車,花蓮車站接送,花蓮,花蓮一日遊,花蓮租車,花蓮旅遊地圖,花蓮旅遊 包車,花蓮旅遊行程,花蓮旅遊景點

October 20, 2009 6:46 AM  
Anonymous Anonymous said...

無料 出会い系
無料 出会い系
無料 出会い系
無料 出会い系

人妻
人妻
人妻
人妻
人妻

出会い系
出会い系サイト
出会い

October 21, 2009 5:32 AM  
Anonymous Anonymous said...

celebrity movie archive

http://spycelebrity.com/ - here you can watch celebrity videos!
it is the greatest celebrity tube website! tons of naked, nude celebs, oops, scandals, sex tapes!
enjoy http://spycelebrity.com/

October 24, 2009 8:12 AM  
Anonymous Anonymous said...

celebrity oops
celebrity tube
naked celebrities

October 24, 2009 8:14 AM  
Anonymous Anonymous said...

出会い
熟女
メル友
出会い
出会い
家出少女
家出掲示板
メル友
出会い
出会い
出会い
出会い
出会い
出会い
出会い
出会い
出会い
出会い
出会い
出会い
出会い
出会い
出会い
出会い
出会い
出会い
出会い
出会い
出会い
出会い
出会い
出会い
出会い
出会い

October 27, 2009 4:42 AM  
Anonymous Anonymous said...

寧安橋,花蓮鯉魚潭,花蓮秀姑巒溪泛舟,蝴蝶谷渡假村,花蓮2日遊,花蓮赤柯山,

October 27, 2009 8:27 AM  
Anonymous Anonymous said...

ナンパ セフレ セックス
競馬予想ソフト 馬券術
セックス テクニック 潮吹き
競馬 投資 稼ぐ
恋愛 口説く ナンパ
パチンコ スロット ホルコン 攻略法
ED 勃起
セックス SEX
競馬 投資 稼ぐ
セフレ 肉便器 性奴隷 メス豚
投資 稼ぐ
早漏 対策
童貞 セックス オナニー
ナンパ セックス
キャバ嬢 風俗嬢 デリ嬢 SEX
セックスフレンド セフレ 方法
パチスロ 攻略法
社内恋愛 職場結婚 セックス
キャバクラ モテる
競艇 投資 競艇ソフト
風俗嬢 落とす セックス
彼女 プレゼント マニュアル
オンラインカジノ 攻略法
セックス テクニック 彼氏
出張ホスト 風俗 バイト
競馬 馬券術
モテる セックス sex
競馬 馬券術
ナンパ 出会い
競馬 馬券術
競馬 必勝法 JRA-VAN
彼氏 アラサー 独身 結婚 出会い
モテる ナンパ セックス
ED 勃起不全 バイアグラ
競馬 馬券術:
競馬 馬券術
復縁 大好きなあの人と復縁できる方法 出会い・恋愛・占い・おまじない
競馬 馬券術
競馬 投資 稼ぐ
ナンパ セックス 落とす
風俗嬢 恋愛 おとす
早漏 克服 対策 防止
競馬 稼ぐ 投資
競馬 馬券術
ネットナンパ mixi gree SNS 出会い系
セックス SEX モテる
ミニロト 予想
ナンバーズ 攻略法
セックス テクニック
セフレ セックス ナンパ
短期 ダイエット 方法
アダルトチャット
thanks!

October 27, 2009 12:27 PM  
Anonymous Anonymous said...

ghd hair straighteners,
cheap ghd hair straighteners,
hair straighteners,
pink ghd hair straighteners,
new ghd hair straighteners,

ugg boots,
cheap ugg boots,
discount ugg boots,
australia ugg boots,
wholesale ugg boots,
sheepskin ugg boots,
ugg boots 5815,
ugg boots size 9,
ugg snowboots,
womens ugg boots,
classic tall ugg boots ,
short ugg boots,
new ugg boots,
winter ugg boots,

November 12, 2009 10:49 PM  
Anonymous Anonymous said...

エロチャット
エロチャット
エロチャット
エロチャット
エロチャット

アダルトチャット
アダルトチャット
アダルトチャット
アダルトチャット
アダルトチャット

ツーショット
ツーショット
ツーショット
ツーショット
ツーショット

2ショット
2ショット
2ショット
2ショット

メル友
メル友
メル友
メル友

出会い
出会い
出会い
セックス
援助交際
援助交際
援助交際

November 18, 2009 11:51 PM  
Blogger Unknown said...

This is a great thing to help me delete duplicate folders, especially I have this way of downloading items when I forgot where I saved them and don't bother to locate where they are. This will help me clean my computer, aside from using disk cleanup of course. This will give mo more storage for my work. Sad that it only works on folders. Hope the next version can delete messages across the folders. bowflex

November 29, 2009 2:46 AM  
Anonymous Anonymous said...

KABUL, Afghanistan — Tainted by ed hardy mens clothing
a flawed election and ed hardy womens shoes
allegations of high-level corruption in his regime, ed hardy mens shoes
President Hamid Karzai ed hardy mens longsleeve
was inaugurated Thursday for a second term, ed hardy mens jackets
saying the Afghan Army should assume full control of the country’s security within five years. Ed Hardy chothing
“We will decrease the role of international forces,” ed hardy womens jackets
Mr. Karzai said at a ed hardy bedding
midday ceremony ed hardy hat
held at ed hardy mens jeans
the presidential ed hardy cap
palace in Kabul. ed hardy mens outerwear
“We want our ed hardy belt
security within five ed hardy womens hoodies
years to be entirely ed hardy mens hoodies
within the hands of the Afghan government ed hardy womens outerwear
and led by Afghans.”
The ceremony ed hardy jackets
was the culmination of a fraught ed hardy longsleeve
and chaotic electoral process that began on Aug. ed hardy cap
20 when Afghans went to the polls. ed hardy hat
Mr. Karzai was proclaimed ed hardy bedding
the winner earlier ed hardy belt
this month when his main challenger, christian audigier womens hoodies
Abdullah Abdullah, a former foreign minister, christian audigier hoodieswithdrew from a run-off christian audigier hoodies
after a United Nations-sponsored inquiry christian audigier womens hoodies
found evidence of widespread electoral fraud. ugg boots,
cheap ugg boots,
discount ugg boots,
australia ugg boots,
wholesale ugg boots,
sheepskin ugg boots,
ugg boots 5815,

November 30, 2009 7:01 PM  
Anonymous Anonymous said...

Life Experience Degree | online masters degree | associate degree

December 03, 2009 10:14 PM  
Anonymous Anonymous said...

Nike shoes
Cheap nike shoes
Discount nike shoes
Nike shox r4
nike shox shoes
puma cat
cheap sport shoes
cheap nike shox
cheap nike max
nike tn dollar
nike running shoes
nike air max tn
puma shoes
discount puma shoes
puma mens shoes
puma running shoes
puma shoes
ed hardy clothes
ed hardy shirts
ed hardy jackets
ed hardy hoodies
ed hardy boots
ed hardy polo shirts
ed hardy shoes
ed hardy jeans
ed hardy outerwear
ed hardy long sleeve shirts
ed hardy bags
ed hardy winter boots
ed hardy handbags
ed hardy love kills slowly shirts
ed hardy love kills slowly shoes
ed hardy love kills slowly boots
ed hardy trousers
ed hardy mens
ed hardy womens
ed hardy t shirts
ed hardy sunglasses
ghd hair straighteners mk4
hair straightners
ghd iv styler hair straightener
ghd hair straightners
cheap ghd hair straighteners

December 08, 2009 6:12 PM  
Blogger Unknown said...

This comment has been removed by the author.

December 17, 2009 5:15 PM  
Blogger ユメ said...

布団 通販
精力剤
仮性包茎 手術

January 08, 2010 9:25 AM  
Blogger Viagra said...

Hi, great post. thank you.Buy Viagra The best website in the world.Buy Viagra,CialisThe best on the world.Thanks!Buy Viagra online

January 16, 2010 4:03 AM  
Blogger Viagra said...

senza ricettaComprare Viagra generico The best on the world.Thanks!Comprar Viagra por internet

January 19, 2010 11:15 PM  
Blogger Unknown said...

Waugh... Great blog. Found it by accident when I search for securityalarm.
Belive me, I'll come back to this great blog. :)

March 09, 2010 11:08 AM  
Blogger Viagra said...

Your blog is very interesting and I like to read it!

March 11, 2010 2:29 AM  
Blogger Viagra said...

You have a very good blog that the main thing a lot of interesting and useful!

March 11, 2010 2:32 AM  
Blogger Viagra said...

The best on the world.Thanks!Viagra Online

March 11, 2010 2:33 AM  
Blogger Viagra said...

Cialis sur internet!Buy Cialis from canada
Comprar Propecia por internet
Comprare Lasix senza ricetta

March 11, 2010 2:35 AM  
Blogger Unknown said...

NFL Jerseys

Chaussure de Spor


Chaussures Sports

ed hardy clothing

tn requin

nike tn

Tn Requin

Air Shoes

mobile phone

March 15, 2010 11:07 PM  
Blogger Luca said...

I am really glad you did it
Dollarkurs | Goldpreis

March 18, 2010 6:43 AM  
Blogger Unknown said...

Thanks for the Extension.
As i have very great archives i will test it.
But i have one problem. Maybe we can improve the extension together. A few ideas from my point of view:NFL Jerseys
tn chaussures
chaussures puma
Tn Requin

March 18, 2010 8:16 PM  
Blogger Unknown said...

ed hardy clothing
Chaussures Sport
Tennis Racquet Shop
Cheap Polo Shirts
The North Face Jackets
cheap ed hardy
Chaussures Sport
Tennis Racquet
nike shox r4
ed hardy
cheap ed hardy
polo shirts
cheap polo
Remise Chaussures Sport
nike tn requin
ed hardy clothes
nike femmes chaussures

April 07, 2010 7:52 PM  
Blogger Unknown said...

This is a great post you have written. Having this information will prove to be very useful going forward. Thanks.
parental control software | parental control for blackberry | parental control for windows 7 | Android parental control | parental control for iphone 4Gs | parental control for windows xp | parental control ipad

April 09, 2010 7:38 PM  
Blogger keyboard said...

Latest film Clash of the Titans
In order to start the final struggle for power and fighting constantly with the King caused the human king and the gods of competition and confrontation, but only confrontation between the gods, it really could destroy the entire world, and let all have ceased to exist ... ... Zeus (with Muni Sen ornaments) the son of Perseus (Samuwoxin Cotton ornaments) While the identity of God is born into the human world, but was brought up as a human dependent - Even so, he was still no way to save his beloved family fled the Hardee's (Ralph Fiennes decoration) of revenge, he is from the God of Hell Hades. Perseus realized that he no longer has any care, he voluntarily assumed a
HP Pavilion DV5000 Battery|Dell Inspiron 1525 AC Adapter|DV5000 Battery|Dell PA-12 AC Adapter|Dell Inspiron 5100 Adapter|8600 Adapter|Dell Inspiron 1501 Adapter|HP 417220-001 Adapter|Dell inspiron 1300 AC Adapter

April 14, 2010 12:17 AM  

Post a Comment

<< Home