removing duplicate messages + rebuilt/compacted database

[Loren]

There are 2 parts to this question, but they are related.

Recently I've got the warning that your database is corrupted. (it was
about 900 Meg). I rebuilt the databse. Then later I got the same
message a second time, and rebuilt it again. Then as a preventative
measure I archived some 200 MB of msgs and compacted the database.

Each time I did this, when I checked mail, Entourage downloaded
duplicate messages from the server, going back about 3 weeks.

This is the Database Utility.log from the first corruption scan:

======================
Starting Scan
====================


ERROR: BlockID #66725 has an unknown checksum

ERROR: BlockID #80481 has an unknown checksum


So my question(s) are:

1. Related to the corruption:
a. What might the reason be that I got the corruption message in
the first place? Is there a way to get more info besides above?

b. Why would I get a corruption message very soon after rebuilding the
databse?

2. Related to duplicates:
a. if the database is rebuilt, will it always forget how much it needs
to retreive from server? Is there a way to avoid this?

b. how can I remove duplicates? There must be an easier way than to
do this manually.


Thank you, I especially need an answer to 2b right now.

Loren

 

[Barry Wainwright]

There are 2 parts to this question, but they are related.

Recently I've got the warning that your database is corrupted. (it was
about 900 Meg). I rebuilt the databse. Then later I got the same
message a second time, and rebuilt it again. Then as a preventative
measure I archived some 200 MB of msgs and compacted the database.

Each time I did this, when I checked mail, Entourage downloaded
duplicate messages from the server, going back about 3 weeks.

This is the Database Utility.log from the first corruption scan:

======================
Starting Scan
====================


ERROR: BlockID #66725 has an unknown checksum

ERROR: BlockID #80481 has an unknown checksum


So my question(s) are:

1. Related to the corruption:
a. What might the reason be that I got the corruption message in
the first place?

Because the database was corrupted?

Is there a way to get more info besides above?
No.


b. Why would I get a corruption message very soon after rebuilding the
databse?

Hard to say. Either the rebuild did not fix the corruption (it can happen,
but it gets less likely with each update since the rebuild engine does get a
lot of attention and improvement)

2. Related to duplicates:
a. if the database is rebuilt, will it always forget how much it needs
to retreive from server?
Yes.

Is there a way to avoid this?

No, because the rebuild process is robust mainly because it does not attempt
to use indexes or other extraneous information from the pre-rebuild state.
This, unfortunately, includes the message status flags.

b. how can I remove duplicates? There must be an easier way than to
do this manually.

Yes, with the script that is posted below.

Save the script as a compiled script & put it in the ŒEntourage Script Menu
Items¹ folder in your ŒMicrosoft User Data¹ folder. The script can be
manually run from the menu.

Thank you, I especially need an answer to 2b right now.

See below

--
Barry Wainwright
Microsoft MVP (see http://mvp.support.microsoft.com for details)
Seen the All-New Entourage Help Pages? - Check them out:
<http://www.entourage.mvps.org/>


-- Remove Duplicate Message v1.0
-- An Applescript by Barry Wainwright, 18th March 2002
-- Detects and deletes duplicate messages within a selected folder
-- works on Message-ID header - uniquely identifying duplicates

tell application "Microsoft Entourage"
set theMessages to current messages
if theMessages = {} then
set theFolder to selection
if class of theFolder is folder then
set mb to theFolder
else
display dialog "In the folder listing, please select the folder
you want to be scanned for duplicates" with icon stop buttons {"Quit"}
default button 1
return -99
end if
else
set mb to storage of item 1 of theMessages
end if
set theName to name of mb
say "Removing duplicates from mail folder: " & theName
set y to count messages of mb
say "Number of messages to check, " & y
set IDlist to {}
repeat with x from y to 1 by -1
try
set theHeaders to (get headers of message x of mb)
set AppleScript's text item delimiters to {"Message-"}
set temp to text item 2 of theHeaders
set AppleScript's text item delimiters to {return}
set theId to text 5 through -1 of text item 1 of temp
on error
set theId to ""
end try
if theId is in my IDlist then
delete message x of mb
else if theId ‚ "" then
copy theId to end of IDlist
end if
if x mod 100 = 0 then say "" & x
end repeat
set removedCount to y - (count messages of mb)
if removedCount is 0 then
say "Finished. No duplicates detected"
else
say "Finished. " & removedCount & " duplicates removed"
end if
set AppleScript's text item delimiters to {""}
end tell

 

[Loren]

Thank you I will try the script and report back problems. If no
response, it worked, and I thank you in advance. MS should build this
in, or include your script!

 

[Loren]

Barry... thanks I tried this script. I have a couple recommendations
for improvement:

1. (most important) It deleted the messages I had read or replied to,
rather than the "new" ones. Is it possible to re-write this so that in
the duplicate found set it prefers to keep ones marked read or replied
over new?

2. possibly it should be made a stay-open script, because long mailbox
folders might time out. I didn't have this problem but I can see it
happening. So far my limit mailbox was 5500 msgs, but I have ones with
more.

3. Perhaps an option to only check duplicates within the last d days
rather than the whole mailbox.

4. I modied the "mod" value to 500 because voice messages every 100
messages was annoying.

Thanks!

 

[Delete Me]

Hi,
Im a Mac noob, but I'm having the same prob with duplicates. When I cut and
paste the script code into Script Editor and compile, it hangs up at
default button 1
SYNTAX ERROR: Expected end of line, etc. but found identifier.

what am I doing wrong?
OSX 10.3.9