Outlook 2003 - Message-ID Missing

[Brad Baker]

We've recently encountered a problem with our CRM software - the email
retreival service is unable to retrieve certain emails our customers are
sending us. Upon investigation we discovered the problem was that the
emails were missing the message-ID header.

Digging deeper we determined this was a problem with Outlook 2003:

http://pessoal.org/blog/?p=30
http://extelligence.ringlet.net/roam/archives/000018.html
http://www.terryfrazier.com/fullThread$msgNum=1526

Apparently the message-ID header no longer gets added by the Outlook 2003
but rather Microsoft expects the mail server to add the header. Perfect.
Except our mail server vender is claiming that the mail client is supposed
to add the header quoting RFC 2821:

"The following changes to a message being processed MAY be applied when
necessary by an originating SMTP server, or one used as the target of SMTP
as an initial posting protocol:
- Addition of a message-id field when none appears"

One of the Microsoft MVP's (Neo) quotes a separate RFC (rfc2476) taking the
exact opposite stance:

"The MSA MAY add or replace the 'Message-ID' field, if it lacks it, or it is
not valid syntax (as defined by [MESSAGE-FORMAT])."

So we have Microsoft claiming it's the servers fault and the mail server
vender claiming it's the mail clients fault. In the mean time our software
doesn't work and we're missing customer emails.

I'm posting here partly in frustration at the lack of maturity between
Microsoft and our mail server vender but partly because I'm hoping that
Microsoft will be addressing or has addressed this problem via a hotfix and
will not be repeating the same mistake in Outlook 2007.

Any further comments or information anyone else has on this problem would be
greatly appreciated.

Thank You,
Brad

 

[neo [mvp outlook]]

To set the record straight since you only partially quoted one of my
messages on this subject...

RFC 2821 is for server to server communications. RFC 2476 is for client to
server. Considering that Microsoft Outlook, Outlook Express, The Bat, Pine,
..etc are just that, mail clients they are within their right to submit
incomplete messages. If the mail server vendor would just read RFC 2476,
you can see that it is very clearly talking about message submissions from
"Mail User Agents" (read mail clients) and that they may not be complete.
It is the responsibility of the mail server to complete the message before
routing it to the next hop.

I would not expect a hotfix at this time for Outlook 2003.

As for Outlook 2007, I'm hoping one of the other MVPs will chime in and say
what the default behavior is and whether or not there is an option to turn
it on. Unfortunately I can't answer this for you as the beta for Office
2007 line hasn't generated any desire in me to test or use it.

We've recently encountered a problem with our CRM software - the email
retreival service is unable to retrieve certain emails our customers are
sending us. Upon investigation we discovered the problem was that the
emails were missing the message-ID header.

Digging deeper we determined this was a problem with Outlook 2003:

http://pessoal.org/blog/?p=30
http://extelligence.ringlet.net/roam/archives/000018.html
http://www.terryfrazier.com/fullThread$msgNum=1526

Apparently the message-ID header no longer gets added by the Outlook 2003
but rather Microsoft expects the mail server to add the header. Perfect.
Except our mail server vender is claiming that the mail client is supposed
to add the header quoting RFC 2821:

"The following changes to a message being processed MAY be applied when
necessary by an originating SMTP server, or one used as the target of SMTP
as an initial posting protocol:
- Addition of a message-id field when none appears"

One of the Microsoft MVP's (Neo) quotes a separate RFC (rfc2476) taking
the exact opposite stance:

"The MSA MAY add or replace the 'Message-ID' field, if it lacks it, or it
is not valid syntax (as defined by [MESSAGE-FORMAT])."

So we have Microsoft claiming it's the servers fault and the mail server
vender claiming it's the mail clients fault. In the mean time our software
doesn't work and we're missing customer emails.

I'm posting here partly in frustration at the lack of maturity between
Microsoft and our mail server vender but partly because I'm hoping that
Microsoft will be addressing or has addressed this problem via a hotfix
and will not be repeating the same mistake in Outlook 2007.

Any further comments or information anyone else has on this problem would
be greatly appreciated.

Thank You,
Brad

 

[Brad Baker]

Neo -

I apologize for not including your entire original post but I was trying to
keep my post concise.

I appreciate the follow-up and I've passed your comments onto our mail
server vender, however if I understand correctly, even if our mail server
vender strictly follows both RFCs, there are still gaps. Consider the
following scenario:

Client Computer (running Outlook 2003) -> Origin Mail Server (per RFC2476 -
should add message-ids but doesn't) -> Destination Mail Server (our mail
server - per RFC2821 - not required to add message-id) -> CRM System (cannot
process message due to missing message id)

In the situation outlined above couldn't our mail server vender follow both
RFC's and Outlook follow the RFC's but we still end up with messages that
don't contain message-ids due to non-compliant third party MTAs? What are we
supposed to do in that case?

Right now I'm trying to convince our mail server vender to add checking into
their software for other MTA's not adding message-ids and our CRM provider
to see if they can re-write their software not to rely on message-ids. No
one wants to address the problem though.

Even if all parties are technically right, that doesn't negate the fact that
there is still a problem. I'm disappointed that all three venders haven't
made a more concerted effort to address this problem rather than trying to
shift the blame.

Brad

To set the record straight since you only partially quoted one of my
messages on this subject...

RFC 2821 is for server to server communications. RFC 2476 is for client
to server. Considering that Microsoft Outlook, Outlook Express, The Bat,
Pine, .etc are just that, mail clients they are within their right to
submit incomplete messages. If the mail server vendor would just read RFC
2476, you can see that it is very clearly talking about message
submissions from "Mail User Agents" (read mail clients) and that they may
not be complete. It is the responsibility of the mail server to complete
the message before routing it to the next hop.

I would not expect a hotfix at this time for Outlook 2003.

As for Outlook 2007, I'm hoping one of the other MVPs will chime in and
say what the default behavior is and whether or not there is an option to
turn it on. Unfortunately I can't answer this for you as the beta for
Office 2007 line hasn't generated any desire in me to test or use it.

We've recently encountered a problem with our CRM software - the email
retreival service is unable to retrieve certain emails our customers are
sending us. Upon investigation we discovered the problem was that the
emails were missing the message-ID header.

Digging deeper we determined this was a problem with Outlook 2003:

http://pessoal.org/blog/?p=30
http://extelligence.ringlet.net/roam/archives/000018.html
http://www.terryfrazier.com/fullThread$msgNum=1526

Apparently the message-ID header no longer gets added by the Outlook 2003
but rather Microsoft expects the mail server to add the header. Perfect.
Except our mail server vender is claiming that the mail client is
supposed to add the header quoting RFC 2821:

"The following changes to a message being processed MAY be applied when
necessary by an originating SMTP server, or one used as the target of
SMTP as an initial posting protocol:
- Addition of a message-id field when none appears"

One of the Microsoft MVP's (Neo) quotes a separate RFC (rfc2476) taking
the exact opposite stance:

"The MSA MAY add or replace the 'Message-ID' field, if it lacks it, or it
is not valid syntax (as defined by [MESSAGE-FORMAT])."

So we have Microsoft claiming it's the servers fault and the mail server
vender claiming it's the mail clients fault. In the mean time our
software doesn't work and we're missing customer emails.

I'm posting here partly in frustration at the lack of maturity between
Microsoft and our mail server vender but partly because I'm hoping that
Microsoft will be addressing or has addressed this problem via a hotfix
and will not be repeating the same mistake in Outlook 2007.

Any further comments or information anyone else has on this problem would
be greatly appreciated.

Thank You,
Brad

 

[Roady [MVP]]

From this information I would say that the issue should be resolved by the
CRM vendor. It is relying on something that cannot be simply assumed to be
there when respecting the RFCs.

Another way to go would be to fight the RFCs of course so that all mail
clients are forced to add message-ids if they want to be RFC compliant ;-)

--
Robert Sparnaaij [MVP-Outlook]
Coauthor, Configuring Microsoft Outlook 2003


-----
Neo -

I apologize for not including your entire original post but I was trying to
keep my post concise.

I appreciate the follow-up and I've passed your comments onto our mail
server vender, however if I understand correctly, even if our mail server
vender strictly follows both RFCs, there are still gaps. Consider the
following scenario:

Client Computer (running Outlook 2003) -> Origin Mail Server (per RFC2476 -
should add message-ids but doesn't) -> Destination Mail Server (our mail
server - per RFC2821 - not required to add message-id) -> CRM System (cannot
process message due to missing message id)

In the situation outlined above couldn't our mail server vender follow both
RFC's and Outlook follow the RFC's but we still end up with messages that
don't contain message-ids due to non-compliant third party MTAs? What are we
supposed to do in that case?

Right now I'm trying to convince our mail server vender to add checking into
their software for other MTA's not adding message-ids and our CRM provider
to see if they can re-write their software not to rely on message-ids. No
one wants to address the problem though.

Even if all parties are technically right, that doesn't negate the fact that
there is still a problem. I'm disappointed that all three venders haven't
made a more concerted effort to address this problem rather than trying to
shift the blame.

Brad

To set the record straight since you only partially quoted one of my
messages on this subject...

RFC 2821 is for server to server communications. RFC 2476 is for client
to server. Considering that Microsoft Outlook, Outlook Express, The Bat,
Pine, .etc are just that, mail clients they are within their right to
submit incomplete messages. If the mail server vendor would just read RFC
2476, you can see that it is very clearly talking about message
submissions from "Mail User Agents" (read mail clients) and that they may
not be complete. It is the responsibility of the mail server to complete
the message before routing it to the next hop.

I would not expect a hotfix at this time for Outlook 2003.

As for Outlook 2007, I'm hoping one of the other MVPs will chime in and
say what the default behavior is and whether or not there is an option to
turn it on. Unfortunately I can't answer this for you as the beta for
Office 2007 line hasn't generated any desire in me to test or use it.

We've recently encountered a problem with our CRM software - the email
retreival service is unable to retrieve certain emails our customers are
sending us. Upon investigation we discovered the problem was that the
emails were missing the message-ID header.

Digging deeper we determined this was a problem with Outlook 2003:

http://pessoal.org/blog/?p=30
http://extelligence.ringlet.net/roam/archives/000018.html
http://www.terryfrazier.com/fullThread$msgNum=1526

Apparently the message-ID header no longer gets added by the Outlook 2003
but rather Microsoft expects the mail server to add the header. Perfect.
Except our mail server vender is claiming that the mail client is
supposed to add the header quoting RFC 2821:

"The following changes to a message being processed MAY be applied when
necessary by an originating SMTP server, or one used as the target of
SMTP as an initial posting protocol:
- Addition of a message-id field when none appears"

One of the Microsoft MVP's (Neo) quotes a separate RFC (rfc2476) taking
the exact opposite stance:

"The MSA MAY add or replace the 'Message-ID' field, if it lacks it, or it
is not valid syntax (as defined by [MESSAGE-FORMAT])."

So we have Microsoft claiming it's the servers fault and the mail server
vender claiming it's the mail clients fault. In the mean time our
software doesn't work and we're missing customer emails.

I'm posting here partly in frustration at the lack of maturity between
Microsoft and our mail server vender but partly because I'm hoping that
Microsoft will be addressing or has addressed this problem via a hotfix
and will not be repeating the same mistake in Outlook 2007.

Any further comments or information anyone else has on this problem would
be greatly appreciated.

Thank You,
Brad

 

[neo [mvp outlook]]

Q) In the situation outlined above couldn't our mail server vender follow
both RFC's and Outlook follow the RFC's but we still end up with messages
that don't contain message-ids due to non-compliant third party MTAs?

A) Really depends on the vendor. For example, I sit behind an Exchange
server. If my server receives a message from the internet that is missing a
message-id, it generates it. Apparently your solution is different where
they could have made logic decisions that anything that comes to port 25 is
considered complete, but anything submitted on the message submission port
(587) is not.



Q) What are we supposed to do in that case?

A) In the example given, the Origin mail server is clearly the one with a
software bug if it supports RFC 2476 and didn't attach a message id. If it
doesn't support RFC 2476, then they should be haggling it over with said
vendor to see if it is time to update/upgrade.



Outside of that and to add to Roady's feedback, an email gateway server (say
a Windows 2003 Web/Standard Edition) that is using windows smtp services
might fix the problem when injected in-between the "Origin Mail Server (per
RFC2476 - should add message-ids but doesn't) -> Destination Mail Server
(our mail server - per RFC2821 - not required to add message-id) stage.

Neo -

I apologize for not including your entire original post but I was trying
to keep my post concise.

I appreciate the follow-up and I've passed your comments onto our mail
server vender, however if I understand correctly, even if our mail server
vender strictly follows both RFCs, there are still gaps. Consider the
following scenario:

Client Computer (running Outlook 2003) -> Origin Mail Server (per
RFC2476 - should add message-ids but doesn't) -> Destination Mail Server
(our mail server - per RFC2821 - not required to add message-id) -> CRM
System (cannot process message due to missing message id)

In the situation outlined above couldn't our mail server vender follow
both RFC's and Outlook follow the RFC's but we still end up with messages
that don't contain message-ids due to non-compliant third party MTAs? What
are we supposed to do in that case?

Right now I'm trying to convince our mail server vender to add checking
into their software for other MTA's not adding message-ids and our CRM
provider to see if they can re-write their software not to rely on
message-ids. No one wants to address the problem though.

Even if all parties are technically right, that doesn't negate the fact
that there is still a problem. I'm disappointed that all three venders
haven't made a more concerted effort to address this problem rather than
trying to shift the blame.

Brad

To set the record straight since you only partially quoted one of my
messages on this subject...

RFC 2821 is for server to server communications. RFC 2476 is for client
to server. Considering that Microsoft Outlook, Outlook Express, The Bat,
Pine, .etc are just that, mail clients they are within their right to
submit incomplete messages. If the mail server vendor would just read
RFC 2476, you can see that it is very clearly talking about message
submissions from "Mail User Agents" (read mail clients) and that they may
not be complete. It is the responsibility of the mail server to complete
the message before routing it to the next hop.

I would not expect a hotfix at this time for Outlook 2003.

As for Outlook 2007, I'm hoping one of the other MVPs will chime in and
say what the default behavior is and whether or not there is an option to
turn it on. Unfortunately I can't answer this for you as the beta for
Office 2007 line hasn't generated any desire in me to test or use it.

We've recently encountered a problem with our CRM software - the email
retreival service is unable to retrieve certain emails our customers are
sending us. Upon investigation we discovered the problem was that the
emails were missing the message-ID header.

Digging deeper we determined this was a problem with Outlook 2003:

http://pessoal.org/blog/?p=30
http://extelligence.ringlet.net/roam/archives/000018.html
http://www.terryfrazier.com/fullThread$msgNum=1526

Apparently the message-ID header no longer gets added by the Outlook
2003 but rather Microsoft expects the mail server to add the header.
Perfect. Except our mail server vender is claiming that the mail client
is supposed to add the header quoting RFC 2821:

"The following changes to a message being processed MAY be applied when
necessary by an originating SMTP server, or one used as the target of
SMTP as an initial posting protocol:
- Addition of a message-id field when none appears"

One of the Microsoft MVP's (Neo) quotes a separate RFC (rfc2476) taking
the exact opposite stance:

"The MSA MAY add or replace the 'Message-ID' field, if it lacks it, or
it is not valid syntax (as defined by [MESSAGE-FORMAT])."

So we have Microsoft claiming it's the servers fault and the mail server
vender claiming it's the mail clients fault. In the mean time our
software doesn't work and we're missing customer emails.

I'm posting here partly in frustration at the lack of maturity between
Microsoft and our mail server vender but partly because I'm hoping that
Microsoft will be addressing or has addressed this problem via a hotfix
and will not be repeating the same mistake in Outlook 2007.

Any further comments or information anyone else has on this problem
would be greatly appreciated.

Thank You,
Brad