Entourage 2008 Database Rebuild OK but Program Doesn't Think So

[mattdoyel]

Hi,
I'm running Entourage 2008 12.1.0 on an iMac running OS 10.5.6 with 2
GB RAM and 55 GB of free space on a 300 GB drive.

Entourage hung while moving an email message from an IMAP folder to a
folder in Entourage. I got the spinning beach ball, waited a very
long time, and ultimately had to force quit Entourage.

When I restarted Entourage, I got a warning that it could not read my
database and that I needed to rebuild it. I used the Utility to first
verify the database--no problems reported but on restarting Entourage
I got the same message. I went back into the Utility and then rebuilt
the database--the utility reports that everything rebuilt fine.
However, when I launch Entourage I still get the message that it can't
read the database.

I've been through all six steps of the MS troubleshooting site,
including creating a new identity, new user data folder, repairing
disk permissions, creating a new user account, and deleting the
preference file that triggers Entourage reminders.

Is there anything else that I can try? MS tells me "...you may have
to use a third party recovery utility to restore your Entourage
database file." Can anyone recommend a third party recovery utility
for Entourage 2008? Or a recovery service that might be able to help
in this case?

Thanks in advance,
Matt

 

[Corentin Cras-Méneur]

When I restarted Entourage, I got a warning that it could not read my
database and that I needed to rebuild it. I used the Utility to first
verify the database--no problems reported but on restarting Entourage
I got the same message. I went back into the Utility and then rebuilt
the database--the utility reports that everything rebuilt fine.
However, when I launch Entourage I still get the message that it can't
read the database.

Try moving the Entourage preferences to the desktop and relaunch the
application. I suspect they are corrupted,


Corentin

 

[mattdoyel]

Try moving the Entourage preferences to the desktop and relaunch the

application. I suspect they are corrupted,

Corentin

Thanks for the suggestion, Corentin. I moved the following six files
from preferences to the desktop and restarted Entourage but still get
the message that Entourage can't read the database.

com.microsoft.AlertsDaemon.plist
com.microsoft.entourage.database_daemon.plist
com.microsoft.entourage.database_utility.plist
com.microsoft.entourage.office_reminders.plist
com.microsoft.Entourage.plist
Entourage Preferences

The problem is happening on my desktop iMac. Interestingly, if if I
take a copy of the Main Identity folder and put it on my MacBook Air,
replacing the Main Identity folder there, Entourage on the MacBook
will open okay, but none of my offline folders or messages appear.

Any thoughts or suggestions are appreciated.
Matt

 

[Diane Ross]

I moved the following six files
from preferences to the desktop and restarted Entourage but still get
the message that Entourage can't read the database.

There are many more preferences than this. You didn't even remove the
Microsoft folder in preferences.

<http://www.entourage.mvps.org/path/index.html#ent_prefs>

It's possible all your problems will go away if you remove ALL Microsoft
prefs.

The problem is happening on my desktop iMac. Interestingly, if if I
take a copy of the Main Identity folder and put it on my MacBook Air,
replacing the Main Identity folder there, Entourage on the MacBook
will open okay, but none of my offline folders or messages appear.

Are you sure you moved the correct folder over and not a blank Identity? Did
you allow your account to sync back up?

I work as a consultant and if you can't get things working again, you can
contact me off list.

BTW, what size if your Identity?

 

[mattdoyel]

It's possible all your problems will go away if you remove ALL Microsoft
prefs.

Thanks Diane. From my Preferences folder I removed all files
beginning with "com.microsoft" as well as the Microsoft folder.
Unfortunately that did not eliminate the problem.

The size of the Database file inside my identity is 4.49 GB.

Of interest is that I have some backups of the identity (created by
the Utility in months past when I've run it) and I cannot get
Entourage to start using any of them. Entourage will start if I
remove the MUD folder completely from my Documents folder.

Thx for the suggestions.
Matt

 

[Diane Ross]

Of interest is that I have some backups of the identity (created by
the Utility in months past when I've run it) and I cannot get
Entourage to start using any of them. Entourage will start if I
remove the MUD folder completely from my Documents folder.

Let's try this.

* Quit Entourage
* Move the Microsoft User Data folder to the desktop.
* Open Entourage and let it create a brand new Microsoft User Data folder
* Move one Identity back at a time to test.
* Use Switch Identities under Entourage in the Menu Bar to select the
Identity you want to open. (Entourage will only open the Identity that is
selected as the default)

The database utility creates a backup when it does a rebuild. It is named
like this:

Main [Backed up 4-21-2009 13.35]

After a rebuild if you are satisfied with the results, you can delete the
old Identity. I would not delete anything until we know we have a database
that can be salvaged. Sometimes the older database will have data that can
be recovered, where the newer Identity is too corrupted. It's a process to
recover corrupted data that takes many different options and tools.

Tip: Make a duplicate in the Finder first as additional backup. A rebuild
does make a duplicate, but sometimes in the process the original and backup
get merged resulting in total disaster. It's rare but it happens.

The size of the Database file inside my identity is 4.49 GB.

Export your Identity using File --> Export as Entourage archive (.rge) file
[All Items] This creates an .rge file. Compare the size of the .rge file to
the size of your database file. The difference is what's called 'wasted
space'. If this export as .rge file fails, this tells us there is still
corruption in your database. As long as you can open the Identity, you can
salvage your data.

Basics on 'wasted space'

Like many file systems & databases, deleting a message in Entourage only
deletes the index entry, and the actual disk space occupied by the message
is left intact as a 'hole' in the database, and will remain like that until
the space is re-used by fresh incoming content.

However, this is not all bad, since the wasted space is used up with new
data as it is added. This way, over a period of time (assuming that you are
deleting stuff as you go) the database will probably reach an optimum size,
after which the stuff you delete is roughly equal to the stuff you add, and
no further growth takes place.

 

[mattdoyel]

*  Open Entourage and let it create a brand new Microsoft User Data folder
*  Move one Identity back at a time to test.
*  Use Switch Identities under Entourage in the Menu Bar to select the
Identity you want to open. (Entourage will only open the Identity that is
selected as the default)

Diane, thank you very much--this method worked to identify a backup
Identity that allowed Entourage to open. I had seven backups in my
Office 2008 Identities folder, all created by the Utility in the past,
and all verified as okay by the utility currently, but only one of the
seven would open. That one is about six weeks old, but overall not
horrible since I have two weeks of email stored on my IMAP servers.
Net, about a month of data missing at this point.

Your method got me thinking that perhaps there was a more recent Time
Machine backup that I could try. [Over the course of this problem
I've now read your writings on Entourage and Time Machine. Not having
previously read them, Time Machine was backing up my MUD folder along
with everything else onto an external 1 TB hard drive. Going forward
I will implement a different backup system for Entourage.]

At any rate, I thought I might try some more recent Time Machine
backups of the Entourage database and see if I could find one that
worked. One at a time, using your method of launching Entourage with
a known good Identity, and then switching identities to test the
unknown identity, I was ultimately successful. Anecdotally, my
success rate was around 10%. One interesting thing I noticed was that
over the course of many days of Time Machine backups the date modified
stamp on the Database file would remain the same from day to day.
Then the date modified would jump to the date of a Time Machine
backup. I am guessing this is because I did not typically shut down
Entourage on a daily basis. I often only quit Entourage every 5 days
or so. (That will change with a new backup strategy).

Now, at this point, I am able to open Entourage using a recent
identity, however my attempts to create an export to an archive .rge
file have failed. The error message says that there is a problem with
one of the messages and that I should run the Utility to rebuild the
database. Of course, I've tried this and it doesn't work, and of
course the database verifies "okay" with the Utility.

It appears that the Utility can verify a database as "okay" when it is
actually not. Is this common?

At this point, there appears to be some kind of problem/corruption in
the database, even though Entourage will open and read the Database.
I will likely try stepping back to a previous copy of the Database
that Entourage can open and see if I can export to an archive .rge
file.

Is there another way to export? Or clean up my database?

Thanks for all your help. I'm very happy that I have a big chunk of
data at least viewable, even if there may still be corruption lurking
in the Database.
Matt

 

[mattdoyel]

Update: I've tried to export to an archive (.rge file) from each of
the Identities that Entourage will let me switch to, but each time the
archive fails, complaining that there is a problem with one of the
messages and that the database needs to be rebuilt. I am selectively
trying to drag folders to my desktop and have (so far) had good luck
with that method creating mbox files.

Thanks in advance for advice and/or opinions.
Matt

 

[Corentin Cras-Méneur]

Update: I've tried to export to an archive (.rge file) from each of
the Identities that Entourage will let me switch to, but each time the
archive fails, complaining that there is a problem with one of the
messages and that the database needs to be rebuilt. I am selectively
trying to drag folders to my desktop and have (so far) had good luck
with that method creating mbox files.

I feel your pain!!!
I hope you'll manage to get it all backed-up and ready to re-import soon
:-\


Corentin

 

[Diane Ross]

Now, at this point, I am able to open Entourage using a recent
identity, however my attempts to create an export to an archive .rge
file have failed. The error message says that there is a problem with
one of the messages and that I should run the Utility to rebuild the
database. Of course, I've tried this and it doesn't work, and of
course the database verifies "okay" with the Utility.

It appears that the Utility can verify a database as "okay" when it is
actually not. Is this common?

Let's just say it's not unusual. I think you have run into one of the
reasons that using Time Machine to backup your database might seem to work
but there is corruption lurking which will eventually result in data loss.

First thing I tell a user that is having problems. If you have a database
that will open, make a copy in the Finder. You don't know when the database
will go belly up and as long as you have a copy you can continue working to
get the data out. When I do recovery for a client, I can't tell you how many
times the database will crash and I have to revert to the copy to continue
the recovery.

Next, is the naming for the Identities. Give them specific names or you
won't know what you've done with them.

At this point, there appears to be some kind of problem/corruption in
the database, even though Entourage will open and read the Database.
I will likely try stepping back to a previous copy of the Database
that Entourage can open and see if I can export to an archive .rge
file.

I would forget that. Most likely I could even get your latest database
working, but if you follow directions and have a good chance to do this
yourself. Where most people fail is when they run into a problem, they don't
have alternatives.

Is there another way to export? Or clean up my database?

Just answered this for another user....

You next step is to find the corrupt message. You can use a script to export
as MBOX files by Jolly Roger (gets subfolders too). The script will fail on
the folder with corruption.

Export folders as MBOX files using a script
<http://www.entourage.mvps.org/database/export_mbox.html>

Once you identify the problem folder, create a subfolder and move half the
messages into the subfolder.

Try dragging the folder to the desktop to create an MBOX file. If this
works, then you know the corruption is in the other half. Most likely you
will have to continue dividing the folder(s) to get to the bad message(s).

Sometimes you can even drag the bad message to the desktop but then it might
not open, but you can see the contents then delete it.

Once you get all your data out of Entourage, you need to create a new
Identity and import your data. If you use a lot of categories, the .rge file
will only carry them over for messages and contacts. It is possible to get
everything but it takes time.

How to manually move your data. (when import fails and/or you need to move
to a new Identity same version or revert to an older version)

<http://www.entourage.mvps.org/install/manual_install.html>

 

[mattdoyel]

You next step is to find the corrupt message. You can use a script to export
as MBOX files by Jolly Roger (gets subfolders too). The script will fail on
the folder with corruption.

Thanks Diane, very helpful. Just dragging the folder to the desktop
doesn't indicate that it fails and only creates an MBOX file of part
of the folder. The script stopping on the folder where there is a
problem is a good pointer to the problem folder. I'm posting details
of what I've done in the case that anyone else has this challenge.

1. Running the Jolly Roger export script, the script stopped in the
middle of the Deleted Items folder (no failure warning but easy enough
to figure out where it stopped). Stopping in the Deleted Items folder
is an easy one, b/c I didn't need anything from that folder so I just
deleted all the messages in it.

2. I then tried to create a .rge backup and the progress bar
progressed considerably further than in the past but it eventually
failed.

3. I double checked the MBOX files output by the script and they were
all there. In a fit of craftiness, I used a text editor to look at
the last MBOX file and sure enough, it did not compare exactly to the
messages in the folder in Entourage (appearing to stop mid-way through
a message about three from the end of the folder).

4. Simple enough, I thought, and just deleted those last messages
from that last folder in Entourage, then permanently deleted the items
from the Deleted Items folder, restarted Entourage, and tried
another .rge archive. Unfortunately it still fails (after about 75%
of the progress bar marching across the window). I've re-run the
script again, looking for a potential failure point, and can find
none.

I was thinking that successfully creating a .rge file was a good
indication that I had found all corrupted messages in the database.
Now I think that it may not matter, since I will eventually abandon
this Identity.

Therefore I think my next step is to create a fresh Identity, and
start importing MBOX files, periodically checking to see if I can
successfully create a .rge file. I don't really need an .rge file at
this point, it just seems to be one indication of a lack of corruption
in the database. (Though I'm not sure that gives me high confidence
overall, combined with a fresh Identity and fresh MBOX import, I
should feel okay).

Thoughts?

Thanks again for all your help.
Matt

PS: I've briefly searched for a mass import capability to get numerous
MBOX files imported into Entourage but have not found anything. I'm
only dealing with about 30 folders so it's not horrible to do them one
at a time.

 

[Diane Ross]

You've made good progress.

I would go back and try running the MBOX script again now that you have
deleted the Deleted Items folder. If it fails, then drag each folder
manually to the desktop from the stopped position. I've seen cases where
even deleting the Deleted Items folder would stop an export so no further
folders can get caught by the script. It would be nice to be able to just
skip this folder all together.

There is no mass import for MBOX files. Entourage archive (.rge) files are
so much easier in that regard. You can drag the MBOX files in. They have to
go to the Inbox folder and not just to the folder list. Having to go through
the File --> Import window is time consuming.

I have never found an MBOX file be corrupt so checking along the way to
verify using the .rge file should not be necessary. Generally, all .rge
files are good, but I have run into cases where it exported but failed to
import so I always add the step to check by importing the .rge file.

Lastly, you might want to consider padding the database just a bit after you
complete your import by adding a big MBOX file several times. Once you get
the desired size, delete those folders. Now you have a space in one place on
your hd like a partition. If I didn't explain 'wasted' space in Entourage
before see this page:

<http://www.entourage.mvps.org/database/rebuild.html>

Stress Test for your Identity: I do this with all database recoveries that I
do.

1. Use the Entourage Database Utility to verify
2. On a copy, I run rebuild on the Identity to verify
3. Test by exporting all items as Entourage archive (.rge) file and import
that file into a new Identity.
4. Use the MBOX script to export all folders.

If a database passes all these tests I know it's good.

 

[mattdoyel]

There is no mass import for MBOX files. Entourage archive (.rge) files are
so much easier in that regard. You can drag the MBOX files in. They have to
go to the Inbox folder and not just to the folder list. Having to go through
the File --> Import window is time consuming.

That tip saved me a ton of time. Before reading your reply, I had
tried dragging directly to the folder list and when it didn't work was
resolved to the File-->Import path. I don't think I would have tried
directly to the Inbox if you hadn't told me.

Lastly, you might want to consider padding the database just a bit after you
complete your import by adding a big MBOX file several times.

I padded my database from 2.24 GB up to 5.03 GB (and then removed the
padding). My database size will grow over time, however this should
last me at least a year. I do delete quite a bit as I work, but I
also have a need to store quite a bit as well. Likely I will start
saving the previous year's emails in a different Identity (with a
backup of that Identity of course

Stress Test for your Identity...

All Stress Test steps completed and database appears to be free of
corruption.
Next step is to implement a script to quit all Office programs
(including the database daemon) and set up a cron job to run it. I've
read the pertinent pages on your blog and don't think I'll have any
problems.

Words can't express how appreciative I am of your expertise and
assistance--I truly appreciate it. Thank you for guiding me through
the process of recovering the data and getting a clean database set up
and verified. I hope that others who experience this problem will
find these posts helpful.

Matt

 

[mattdoyel]

I feel your pain!!!
I hope you'll manage to get it all backed-up and ready to re-import soon
:-\
Corentin

Thanks Corentin. It's all good now. I appreciate your suggestion on
deleting the Prefs, I was really hoping that would solve it (since the
Database passed the Utility but wouldn't open, Prefs made sense).
Regardless, it's all good now.
Thx,
Matt

 

[Diane Ross]

Next step is to implement a script to quit all Office programs
(including the database daemon) and set up a cron job to run it.

There is a script and a shell script on the blog to quit all. Links are in
this post.

Quit all Microsoft applications before backing up

<http://tinyurl.com/cdvqwy>

Since you are rolling your own, you might want to consider using a folder
action to keep your backups in a folder by date. See this article that uses
the folder action FolderOrgX to create snapshots.

Entourage and Time-Machine-like backup (can be used by Tiger users)

All Stress Test steps completed and database appears to be free of
corruption.

Feels good to know your database is truly in good shape?

Words can't express how appreciative I am of your expertise and
assistance--I truly appreciate it. Thank you for guiding me through
the process of recovering the data and getting a clean database set up
and verified. I hope that others who experience this problem will
find these posts helpful.

Glad I was able to help. It's a difficult process and not all users have the
experience to muddle through the process when things get complicated.