avoiding corruption in Entourage database backups

[arlomedia]

Version: 2008
Operating System: Mac OS X 10.5 (Leopard)
Processor: Intel
Email Client: imap

Hello,

I've been reading about problems with backing up Entourage databases when the database files are open for use, either by the Entourage application itself or the Microsoft Database Daemon used by Office Reminders. My coworkers routinely run their backups in the background, and this week we had a situation where we couldn't use a backup when we needed it, so I'm trying to find out exactly what's going on to avoid problems in the future.

I did several tests with our backup software, Synchronize Pro X from Qdea. This is a very configurable backup system, and we currently use it to run incremental backups of our home folders once a day. Here's what I found:

- Databases backed up while Office Reminders are off and both Entourage and the Microsoft Database Daemon (MDD) are not running are restorable. (When Office Reminders are off, the MDD only runs when Entourage is running.)

- Databases backed up while Office Reminders are on, and MDD is running, but Entourage is not running, are also restorable.

- Databases backed up while Office Reminders are on and both Entourage and MDD are running are still restorable, as long as the user is not working in Entourage.

- Databases backed up while the user is actively working in Entourage at the time of the backup, whether Office Reminders are on or off (the MDD will be running anyway) are NOT restorable; they are corrupted and not recoverable by the Microsoft Database Utility.

So, it appears that we don't need to worry about the Microsoft Database Daemon or Office Reminders, or leaving the Entourage application open, we just need to stop using Entourage while the backups are running. This is good because my users will understand this limitation and will be able to work around it, without requiring a new backup process.

I'm posting this here for others to use as a starting point for their own testing. By all means, don't rely on my testing when your data is at stake -- it's really not a good situation to pull a 5 GB database from a backup file only to find out that it's not usable, which is what happened to me this week. On the other hand, a good backup process may not need to be as complex as those described at various websites.

If you find additional info about when backups are and aren't usable, I hope you'll add it to this post.

Cheers,
-Arlo

 

[Diane Ross]

So, it appears that we don't need to worry about the Microsoft Database Daemon
or Office Reminders, or leaving the Entourage application open,

Sorry, but there is a fallacy in your statements.

The Microsoft Database Daemon is used Spotlight and Word, Excel, and
PowerPoint have features that can trigger Office notifications.

The advice from the developers is to quit the Microsoft Database daemon and
Entourage before backing up.

--
Diane
Entourage Help Page <http://www.entourage.mvps.org/>
Entourage Help Blog <http://blog.entourage.mvps.org/>
YouTalk <http://tinyurl.com/bzcrjy> <-- Entourage mailing list
Twitter: follow <http://twitter.com/entouragehelp>

 

[arlomedia]

Sorry, but there is a fallacy in your statements.

I'm reporting what I found through direct testing in my company with my backup software. I welcome reports of any other direct tests in other environments that could disprove my results.

That's why I started this thread -- to document some verifiable facts about exactly what does and doesn't work. The advice I had found repeated in other forum posts was either too vague or too impractical to inform a backup solution for my company.

 

[Diane Ross]

That's why I started this thread -- to document some verifiable facts about
exactly what does and doesn't work. The advice I had found repeated in other
forum posts was either too vague or too impractical to inform a backup
solution for my company.

I'm sorry if the developers word on this isn't good enough for you.

--
Diane
Entourage Help Page <http://www.entourage.mvps.org/>
Entourage Help Blog <http://blog.entourage.mvps.org/>
YouTalk <http://tinyurl.com/bzcrjy> <-- Entourage mailing list
Twitter: follow <http://twitter.com/entouragehelp>

 

[arlomedia]

This advice steered me in the right direction, but there are two problems: 1) this is general "advice", without any specific explanations of what doesn't work and why, and 2) I'm reading this on various message boards around the web, but never from any official Microsoft communication channels (are you an MS employee? I followed your signature links but couldn't tell).

If I compare this indirect info to my own direct testing, then I have to give my own testing some weight, too. But as I explained in my original post, I'm trying to compare my experience with that of others so I can develop as complete an understanding of this issue as possible.

I guess another way of looking at it is that if this oft-published info is complete enough for -your- purposes, then great; but it wasn't enough for mine and I still hope to collect additional info on this forum.

 

[Diane Ross]

2) I'm reading this on various message boards around the web, but never from
any official Microsoft communication channels (are you an MS employee? I
followed your signature links but couldn't tell).

No, I'm a Microsoft MVP and have been since Entourage was introduced. I'll
be sure to tell Andy Ruff who confirmed this for me (he's the lead program
manager for Entourage) that he should stop his work so he can verify this on
this newsgroup.

Sorry, if you guys wait for Microsoft to officially documents something you
might have to wait years. It took us over 4 years to get symbols used in
Entourage documented.

--
Diane Ross, Microsoft Mac MVP
Entourage Help Page <http://www.entourage.mvps.org/>
Entourage Help Blog <http://blog.entourage.mvps.org/>
YouTalk <http://tinyurl.com/bzcrjy> <-- Entourage mailing list
Twitter: follow <http://twitter.com/entouragehelp>

 

[Kerry]

but it wasn't enough for mine and I still hope to collect additional
info on this forum.

You've done excellent work and thanks for posting your results here.
It confirms my own testing but you have done more extensive work than
I have although my tests are in 5 database backup systems
(SuperDuper!, Retrospect, Data Backup, Time Machine and CrashPlan
Pro). I would agree that the information on the web is too general and
with something as critical as your data its probably well advised that
people have a well informed strategy especially if they are using
Entourage (very problematic in 08). I would not concern myself with a
reference to a Microsoft developer unless that developer posts this
specifically on the Microsoft official website (Mactopia or Microsoft
support) in a KnowledgeBase article. I rely on the forums for guidance/
help in resolving usability issues but with anything that has a
functional implication that can cause improper behavior or function of
a system I only rely on KnowledgeBase articles officially posted on
the vendors website or that which is told to me directly by the vendor
themselves (if I get an outside parties reference it might be valid
but I always confirm this - just what I do and that goes for listening
to me - if I say something try it but confirm this with an official
source).

Anyway, great work! Keep posting.

 

[Diane Ross]

If you try to confirm for example, whether the Microsoft Database daemon
being open or even Entourage being open while copied, you might get no
problems in the first 99 times, but the 100th time, it fails. If your data
is important, then I would do the simple thing of quitting to be sure I
didn't get caught with a corrupted database.

Hey, if you guys want to risk your data, it's no skin off my back. I'm just
trying to give you the best advice that I can.

--
Diane Ross, Microsoft Mac MVP
Entourage Help Page <http://www.entourage.mvps.org/>
Entourage Help Blog <http://blog.entourage.mvps.org/>
YouTalk <http://tinyurl.com/bzcrjy> <-- Entourage mailing list
Twitter: follow <http://twitter.com/entouragehelp>

 

[arlomedia]

I'm managing IT for everyone in my company, and if the process is too complicated, it will get done incorrectly or not at all. Quitting Entourage is easy enough, but then people talk about the database daemon, which has no interface -- you have to go into Activity Monitor and find it there to quit it -- and then people were talking about other invisible background processes, without specifying what they were. Then some of the solutions I found online involved using command-line scripts, or multiple backup applications. So where do I draw the line? That's why I started this thread, to find out for sure what is necessary. By confirming a few facts, I can sort through all these recommendations and settle on an approach that works but is also practical enough that my users will be able to do it.

I shouldn't have to justify trying to learn about how Entourage works, on a forum devoted to it ... but there you go.

 

[Diane Ross]

Quitting Entourage is easy enough, but then people talk about the database
daemon, which has no interface

The Microsoft Database daemon a.k.a Notifications/Reminders. Now it's not so
faceless. I think this has confused many users over the years.

Options to Quit:

* In Entourage Menu select "Turn off Office Notifications"
* Fire up 'Activity Monitor', look for 'Microsoft Database Daemon' and
choose 'quit process' from the processes menu (just do a normal quit, not a
forced quit).
* Easiest way.....use this script: (or download script
here)<http://tinyurl.com/auft8o>

tell application "Microsoft Database daemon" to quit

Save as either

Application (to use script from desktop)
Complied (to use from Script Menu)

The workflow I reference includes both scripts then copies the Identity for
backup to the location you select.

Workflow showing scripts

I shouldn't have to justify trying to learn about how Entourage works, on a
forum devoted to it ... but there you go.

Of course not, but you didn't seem to want to accept the advice from the
developers unless it came in some sort of official form.

There are no other faceless applications for Microsoft that I am aware of
that needs quitting. I would set the Automator workflow to run prior to when
your nightly backup is run, or exclude the Identity folder and make a copy
outside of the Microsoft User Data folder for backing up by Time Machine or
whatever software you are using. Since this is not active, you can copy it
at any time.

Does this help?

--
Diane Ross, Microsoft Mac MVP
Entourage Help Page <http://www.entourage.mvps.org/>
Entourage Help Blog <http://blog.entourage.mvps.org/>
YouTalk <http://tinyurl.com/bzcrjy> <-- Entourage mailing list
Twitter: follow <http://twitter.com/entouragehelp>

 

[Kerry]

I shouldn't have to justify trying to learn about howEntourageworks, on aforum devoted to it ... but there you go.

No, you shouldn't and as such I would take this all with a grain of
salt and for what it is.

Anyway, I think it is in each person's interest to develop their own
strategy based on their needs and assessment. Frankly, I have never
had any backup restore fail from Time Machine or any of the systems
I've mentioned and because rebuild doesn't work properly nor sync
services I've had to do way too many restores. Managing a multiple
user environment is difficult as it is without having to deal with all
this nonsense in Entourage but unfortunately using it requires all
this. Microsoft understands this but can't seem to resolve the
problems.

As I said, you've done the work and you understand this well. Its nice
to see.

 

[AlphaBovine]

I will just add my 2 cents here.

I run a small business at home that deals with a LOT of photos and what not. We do not have the money or infastructre for a dedicated FTP upload site or anything that will allow us to trade these 1MB+ files back and forth easily so we rely on Entourage/Outlook to do it for us until we get into a place where we can do that.

In the mean time we have learned to cope with 6GB+ databases and having to create new identities on a monthly bases. Like you we have to experiment and figure out what works for us like any IT team does. Sometimes what works falls under the recommendations from Microsoft's MBU, other times it does not.

As far as Dianne and the other MVP's go, they do offer good advice and will pretty much stick to what is recommended by MBU. However I do agree with your point. They are not official and they are very open about this so to that end you can either dismiss or employ their suggestions.

In the end I feel that is what you learn from using your product and sharing that will give you the best information. Office '08 is a pretty stable release, but like the Windows counter part, they make some really weird decisions that we just scratch our head at. Like when they took IE's HTML web rendering out of Outlook '07 and put in Word's HTML all hell broke loose. For Entourage they had to re-code everything if I remember correctly to work with the Universal binary so I look at this as a totally new, from the ground up product with the applicable amount of flaws.

I hope that when the next version comes out they will make the database more transparent and easier to edit offline but we will see what comes in the future.

 

[William Smith [MVP]]

No, I'm a Microsoft MVP and have been since Entourage was introduced. I'll
be sure to tell Andy Ruff who confirmed this for me (he's the lead program
manager for Entourage) that he should stop his work so he can verify this on
this newsgroup.

Sorry, if you guys wait for Microsoft to officially documents something you
might have to wait years. It took us over 4 years to get symbols used in
Entourage documented.

Just adding my two cents.

The Entourage developers have probably never extensively tested backups
with Entourage. Backups and backup solutions are not within the scope of
their development of the product.

The MVPs get direct access sometimes to the developers and we take the
questions you're asking directly to them when we don't know the answers
ourselves. How to safely back up Entourage has definitely been one of
those questions.

The answers we receive from them in cases like this are most likely
judgment calls. Again, I'd imagine the developers have never extensively
tested different backup scenarios such as "what happens when you
send/receive a message during the backup" or "what happens if you quit
Entourage during the backup" or a hundred other what-ifs.

If their judgment call is to make sure the Database Daemon is quit prior
to a backup then they are suggesting the safest way to proceed based on
multiple scenarios that are probably running through their heads in a
few seconds. Not lab tests and not field tests.

This stuff will probably never be documented by them because they
couldn't support it without testing in a variety of situations, which
will probably never happen. If someone lost data using a procedure that
they did publicly endorse they'd have to provide support for that too.
The variety of methods to back up are simply too numerous to do so.

If what you find works for you then by all means proceed. However,
you're not performing hundreds of hours of automated tests under a
variety of situations on a variety of machines running a variety of
operating systems, which they would do during their testing.

I'd trust their judgment call simply because they understand the
Database far better than we do and their advice isn't difficult to follow.

--

bill

Entourage Help Page <http://entourage.mvps.org/>
Entourage Help Blog <http://blog.entourage.mvps.org/>
YouTalk <http://nine.pairlist.net/mailman/listinfo/youtalk>
Twitter: follow <http://twitter.com/meck>

 

[Kerry]

Just adding my two cents.

The Entourage developers have probably never extensively tested backups
with Entourage. Backups and backup solutions are not within the scope of
their development of the product.

The MVPs get direct access sometimes to the developers and we take the
questions you're asking directly to them when we don't know the answers
ourselves. How to safely back up Entourage has definitely been one of
those questions.

The answers we receive from them in cases like this are most likely
judgment calls. Again, I'd imagine the developers have never extensively
tested different backup scenarios such as "what happens when you
send/receive a message during the backup" or "what happens if you quit
Entourage during the backup" or a hundred other what-ifs.

If their judgment call is to make sure the Database Daemon is quit prior
to a backup then they are suggesting the safest way to proceed based on
multiple scenarios that are probably running through their heads in a
few seconds. Not lab tests and not field tests.

This stuff will probably never be documented by them because they
couldn't support it without testing in a variety of situations, which
will probably never happen. If someone lost data using a procedure that
they did publicly endorse they'd have to provide support for that too.
The variety of methods to back up are simply too numerous to do so.

If what you find works for you then by all means proceed. However,
you're not performing hundreds of hours of automated tests under a
variety of situations on a variety of machines running a variety of
operating systems, which they would do during their testing.

I'd trust their judgment call simply because they understand the
Database far better than we do and their advice isn't difficult to follow..

--

bill

Entourage Help Page <http://entourage.mvps.org/>
Entourage Help Blog <http://blog.entourage.mvps.org/>
YouTalk <http://nine.pairlist.net/mailman/listinfo/youtalk>
Twitter: follow <http://twitter.com/meck>

I'd say that's a very balanced view and prudent. I think its in
anyone's best interest to develop a good backup strategy that works
effectively for them. Reviewing information in the various forums can
help plus reading materials such as Joe Kissell's book "Take Control
of Mac OSX Backups" is a great way to develop your own personal backup
strategy.