Update 3/31/2011: The updated Exchange 2007 SP3 RU3 has been released. See Announcing the Re-release of Exchange 2007 Service Pack 3 Update Rollup 3 (V2).
3/30/2011: We posted a status update for this issue. See Exchange 2007/2010 Rollup 3 Status Update.

Over the weekend, the Exchange Product Group was made aware of an issue which may lead to database corruption if you are running Exchange 2007 Service Pack 3 with Update Rollup 3 (Exchange 2007 SP3 RU3). Specifically, the issue was introduced in Exchange 2007 SP3 RU3 by a change in how the database is grown during transaction log replay when new data is written to the database file and there are no available free pages to be consumed.

This issue is of specific concern in two scenarios: 1) when transaction log replay is performed by the Replication Service as part of ensuring the passive database copy is up-to-date and/or 2) when a database is not cleanly shut down and recovery occurs.

While only a small number of customers have been affected to date, we believe the risk is significant enough that we are recommending all customers to uninstall Exchange 2007 SP3 RU3 on all Mailbox Servers and Transport servers. Uninstalling the rollup will revert the system back to the previously installed version. We have also removed the Exchange 2007 SP3 RU3 download from the Microsoft Download Center and from Microsoft Update until we are able to produce a new version of the rollup.

We are actively working this issue and based on test results plan to release an updated version of Exchange 2007 SP3 RU3 to the Download Center later this week. In addition, we are conducting an internal review of our processes to determine how to prevent issues such as this in the future.

When this issue occurs, the following similar events are logged in the Application Event log of the Mailbox server. Regardless of whether you see these types of events, you should review the recovery instructions and begin that process. If you are uncomfortable performing any of these steps please contact Microsoft Support for assistance.

• Event ID: 454
  Event Type: Error
  Event Source: ESE
  Event Category: Logging/Recovery
  Description: Microsoft.Exchange.Cluster.ReplayService (12716) Recovery E20 SG1\DB1: Database recovery/restore failed with unexpected error -4001.
• Event ID: 2095
  Event Type: Error
  Event Source: MSExchangeRepl
  Event Category: Service
  Description: Log file D:\logs\SG1\E200006AFAE.log in SG1\DB1 could not be replayed. Re-seeding the passive node is now required. Use the Update-StorageGroupCopy cmdlet in the Exchange Management Shell to perform a re-seed operation
• Event ID: 2097
  Event Type: Error
  Event Source: MSExchangeRepl
  Event Category: Service
  Description: The Microsoft Exchange Replication Service encountered an unexpected Extensible Storage Engine (ESE) exception in storage group 'SG1\DB1'. The ESE exception is a read was issued to a location beyond EOF (writes will expand the file) (-4001) ().

In addition, in environments utilizing Continuous Replication, comparison of the database file between the active and passive nodes will indicate that the database file has decreased in size.

Regardless of whether you are experiencing this issue, we strongly recommend taking the below actions to ensure that you do not experience any data loss or outage event associated with this issue.

For example:

• If you have deployed your Mailbox servers utilizing Cluster Continuous Replication (CCR), failure of the active copies may affect your service SLA as you may have no viable passive copies to activate. Hardware failures may result in you not having a means to recover up to the point of failure and thus may experience data loss.
• If you have deployed your Mailbox servers utilizing Single Copy Clusters (SCC), switchovers or failovers may result in this issue as there is only one copy of the database and recovery is performed during switchovers and failovers.

For environments leveraging CCR and/or Standby Continuous Replication (SCR)

If you note the listed events in your environment the following steps must be taken in order to restore your high-availability configuration:

1. Rollback the CCR Mailbox server hosting the passive database copies and any SCR target Mailbox servers to the previously installed version (e.g., Exchange 2007 SP3 RU2) by uninstalling RU3.
2. Re-seed all affected database copies on the CCR Mailbox server and any SCR target Mailbox servers hosting the passive database copies.
3. Verify the database copy status is healthy for all passive copies.
4. Perform a switchover and rollback the remaining CCR Mailbox server to the previously installed version (e.g., Exchange 2007 SP3 RU2).

If you are not seeing these events in your continuous replication enabled environment, we recommend the following steps:

1. Rollback the CCR Mailbox server hosting the passive database copies and any SCR target Mailbox servers to the previously installed version (e.g., Exchange 2007 SP3 RU2) by uninstalling RU3.
2. Perform a switchover and rollback the remaining CCR Mailbox server to the previously installed version (e.g., Exchange 2007 SP3 RU2).

For environments leveraging Single Copy Clusters (SCC)

1. Rollback passive nodes within the SCC environment to the previously installed version (e.g., Exchange 2007 SP3 RU2) by uninstalling RU3.
2. Perform a switchover and rollback the remaining SCC Mailbox server nodes to the previously installed version (e.g., Exchange 2007 SP3 RU2).
3. If you have any databases that will not mount as a result of the above issue, you can restore the damaged databases leveraging a last known good backup.

For environments leveraging standalone Mailbox (or Public Folder) servers

1. Rollback the standalone Mailbox servers to the previously installed version (e.g., Exchange 2007 SP3 RU2) by uninstalling RU3.
2. If you have any databases that will not mount as a result of the above issue, you can restore the damaged databases leveraging a last known good backup.

For Hub Transport and Edge Transport servers

1. Rollback the standalone transport servers to the previously installed version (e.g., Exchange 2007 SP3 RU2) by uninstalling RU3.
2. If any transport servers have mail.que databases which currently do not mount as a result of the above issue, you can recover them by following the steps in Working with the Queue Database on Transport Servers.

Kevin Allison
GM Exchange Customer Experience

Background

The Exchange Store Driver is a core transport component which lives both on the Mailbox server role (as the mail submission service) and the Hub server role. It is responsible for:

• Retrieving messages from the mailbox server that have been submitted by end-users and submitting those to the Hub transport role for categorization and routing.
• Delivering messages to the appropriate mailbox server based on the location of the recipients mailbox.
• Extensibility platform for both mail submission & delivery. Store Driver currently hosts a number of agents that extend the functionality of Exchange. Examples include such agents as Inbox Rules, Conversations, meeting forward notifications, etc.

Exchange 2010 is currently being utilized in Live@EDU, as well as the upcoming Office 365. As you can probably imagine, the Exchange servers that run in those datacenters are loaded and pushed harder than almost any other Exchange server imaginable. Prior to SP1, there were several problems that were encountered with mail delivery to the Exchange mailbox store. In particular, there was a need to make sure that a handful of recipients did not starve the rest of the mail delivery system.

While many of you may not have noticed this problem, Microsoft has seen many of these types of cases over the years; often isolated to a single event like an inadvertent public folder replication storm.

This was despite the message throttling that was already available. Transport roles have also had functionality to avoid resource starvation known as Back Pressure, but this was not designed to protect the system from messages that were already in the Local Delivery queue.

Changes in SP1

In order to further protect both the Mailbox servers and Hub servers from resource starvation, new thread limits were introduced in SP1:

Note: These keys are not present in the EdgeTransport.exe.config file by default.

Is it possible to have too much protection?

Unfortunately, there are two scenarios after applying SP1 where we are seeing customers with messages backing up in the queue. The temporary error message is:

432 4.3.2 STOREDRV.Deliver; recipient thread limit exceeded

As you can probably guess, the two scenarios are:

• Journaling
• Public Folders

In both cases, the deliveries are occurring to a single recipient (or very small number of recipients). This is likely to occur during heavy mail flow. The screen shot below was taken from a lab server while reproducing the issue:

Figure 1: Messages backed up in mailbox delivery queue due to recipient thread limits being exceeded (click here for larger screenshot)

You can see a historical history of 4.3.2 events in connectivity logs on your Hub Transport servers (in the \Program Files\Microsoft\Exchange Server\V14\TransportRoles\Logs\Connectivity\CONNECTLOGxxxxxxxx-x.LOG), like:

#Software: Microsoft Exchange Server
#Version: 14.0.0.0
#Log-type: Transport Connectivity Log
#Date: 2011-01-12T00:00:00.775Z
#Fields: date-time,session,source,Destination,direction,description
 
2011-01-12T00:00:00.775Z,08CD7F1200CBDBD0,MapiSubmission,5f24e416-c380-41b5-bfe0-37b6f1091f49,>,"Failed; HResult: 1140850693; DiagnosticInfo: Stage:LoadItem, SmtpResponse:432-4.3.2 STOREDRV; mailbox server is too busy"
 
2011-01-12T00:00:00.775Z,08CD7F1200CBDBD0,MapiSubmission,5f24e416-c380-41b5-bfe0-37b6f1091f49,-,RegularSubmissions: 0 ShadowSubmissions: 0 Bytes: 0 Recipients: 0 Failures: 1 ReachedLimit: False Idle: False
 
2011-01-12T00:00:05.792Z,08CD7F1200CBDBD1,MapiSubmission,5f24e416-c380-41b5-bfe0-37b6f1091f49,+,Win2k8R2Ex14.dom2k8r2ex14.lab

In some cases, simply leaving the servers alone should cause the queues to slowly drain. In other cases, it may be necessary to take further action because the problem has persisted for a while or because it isn’t part of a one-time event like a Public Folder replication storm.

Alright, I understand. Now how do I fix my situation?

Like every other throttling & performance related feature that has ever been in Exchange, the solution isn’t exactly straight forward. For starters, we’ve seen some level of success simply incrementing both values up one, as follows:

<add key="RecipientThreadLimit" value="2" />
<add key="MaxMailboxDeliveryPerMdbConnections" value="3" />

In fact, there has been enough success with this that we’re considering changing the defaults in a future rollup or service pack. Of course, when we do, the new defaults will only apply to those who haven’t already modified the values. Although it has not yet been released, KB 2491972 will discuss the change when the time comes.

So if +1 is better, then why not +2 or more?

Well, the problem is that all Exchange servers will ultimately be bound by some hardware resource. You don’t want to introduce thrashing or resource starvation due to a public folder or journaling server event that now impacts all users as well. In addition, there are other limits that control the maximum number of threads that can service these types of deliveries. In short, we are not recommending going above 4 for MaxMailboxDeliveryPerMdbConnections, and RecipientThreadLimit should always be at least one fewer.

In an extreme case, if you are in a very busy environment with dedicated journaling mailbox servers, it may be worth considering having dedicated hub transport servers to go along with that. Of course, they can be virtualized or multi-role boxes, but in order to be isolated, the whole bunch will need to be in a dedicated site. Then, you would be able to increase these limits without risking delivery to other mailboxes.

Of course, this is an extreme example. For the rest of us, it may be as simple as trying the +1 approach while carefully monitoring the hub & mailbox servers.

Scott Landry, Steve Schiemann, Frank Brown and Jason Nelson

Can't send or receive email? The Mail Flow Guided Walkthrough (GWT) can help.

Problems with mail delivery can be a very frustrating for both the user and the administrator. Unfortunately, mail flow issues aren't uncommon and will likely occur at some point in time whether your mailboxes are hosted on-premises, in Office 365, or a combination of both.

To help you troubleshoot mailflow issues, we've just released the Mail Flow Guided Walkthrough (GWT). You can use the walkthrough for troubleshooting issues sending and receiving mail or when mail is delayed in the following scenarios:

• The sender’s mailbox is hosted on Office 365 and the recipient’s mailbox is outside your organization.
• The sender’s mailbox is outside your organization and the recipient’s mailbox is hosted in Office 365.
• The sender’s and recipient’s mailboxes are both located in Office 365.
• The sender’s mailbox is hosted in Office 365 and the recipient’s mailbox is located on an on-premise Exchange server and both users are in the same organization.
• The sender’s mailbox is hosted on an on-premise Exchange server and the recipient’s mailbox is hosted in Office 365 and both users are in the same organization.

The goal of this walkthrough is to assist you in resolving these issues in a timely manner by focusing on the scoping and steps used to isolate and resolve problems.

I’d like to thank everyone who contributed to the development of this troubleshooter with special recognition going to the following folks: Tom Kern, Nitin Shukla, Sainath Vijayaraghavan, Jim Martin, Bruce Wilson, Stephen Gilbert, Scott Landry, Charlotte Raymundo, Mary Browning, Star Li, Chen Jiang, Victor Zhang.

Joe Turick

We’re happy to announce that we’ve added some tests to Microsoft Remote Connectivity Analyzer (RCA), to help you test your mail flow configuration. These tests are intended for customers who use the Exchange Online Protection service to perform spam and malware filtering before sending email messages to your email servers. You can run these tests when setting up the service, or whenever you want to verify that your mail flow is set up correctly.

The Verify MX Record and Outbound Connector Test, shown in the image below, verifies that the MX record for your domain points to EOP, so that mail is routed to EOP for filtering before it reaches your mailboxes. This test also verifies that you have an Outbound on-premises connector configured to deliver email to your on-premises environment.

Figure 1: Verify MX Record and Outbound Connector Test

The Verify Service Delivery Test verifies delivery from the service to your on-premises environment by testing the connection between the service and your IP addresses or fully qualified domain names (FQDNs) defined in your Outbound on-premises connector.

For more information about how to run these tests, see Test Mail Flow With the Remote Connectivity Analyzer.

We welcome any feedback; please send comments to the following email address: exrcafb@microsoft.com.

Exchange Online Protection (EOP) Team

You requested it... and we delivered it in Exchange 2010!

One of the most requested items in exchange 2007 was something like this...

...we have 5-12 external domains that we need to allow some users to send to, but prevent sending to all other domains...

Or like this...

...we need a way to allow everyone to send to the internet but restrict members of 'contract workers group' to just certain domains. 

This blog post is meant to show how easy it now is to accomplish this oft heard request in Exchange 2010. Transport rules, introduced with Exchange 2007, provided a lot of new options for administration of mail resulting in even more requests for additional functionality. The rules now have new predicates and actions extending the possibilities of what can be done.

In particular, the predicates for address matching that were previously only available on the Edge role are now available for Hub role as well!

For more information about the new predicate and actions see the following links below:

Exchange 2010 Transport Rule Predicates:
http://technet.microsoft.com/en-us/library/dd638183.aspx

Exchange 2010 Transport Rule Actions:
http://technet.microsoft.com/en-us/library/aa998315.aspx

So I will use the 2nd "request" above to demonstrate how to create a rule in 2010 to accomplish it.

For our example, the rule will restrict "Active Directory Mail enabled users" who have their 'Department' defined as 'Temp Employees' from sending mail to the internet, except they must be allowed to send to 2 external domains called: 'partnerdomain.com' and 'fourthcoffee.com'. Additionally, to reduce Helpdesk calls, you want to send an NDR when they violate the rule. For demonstration purposes I will use 2 Conditions, one Action and one Exception.

Creating a new rule

1. Conditions

a. First condition:

"Sent to users that are inside or outside the Organization, or partners"

Screenshot #1 above, set the dropdown to Outside the Organization option

b. Second condition:

"When the sender's properties match text patterns".

Now note the new options with this in the 3rd screenshot below allowing selection of Active Directory properties on the user object!

Here we will be using the 'Title' property to match the rule to a sender.

2.Actions

"Send rejection message to sender with enhanced status code". The text you add here is displayed in the "Diagnostic information for administrators:" section of the NDR and can say whatever you wish.  Originally I started out with "You may only send internet mail to @fourthcoffee.com and partnerdomain.com".

While the NDR provides the information, it is somewhat 'hidden' to be practical for your typical user, so I will create a customized DSN. At this point, all we need to do is specify the text and enhanced status code for our administrators.  The new text will be "Diagnostic information for System Administrators" and we specified a specific and unique error code 5.7.122 that is easy for administrators to associate with this rule, should troubleshooting be necessary.

3.Exceptions

"Except when a recipient's address matches text patterns". This is where we add domains that these senders are allowed to send mail to on the "Specify text patterns" dialog box.

And finally, this is the customized NDR that senders receive when violating the rule we created. This test was to two recipients where one is an allowed domain, Janer@fourthcoffee.com, and another is not an allowed domain: mthomas@e2k3.dom.

Notice how the NDR was only generated for the rejected recipient.  All other recipients were allowed through.

For more information:

- Understanding Transport Rules
http://technet.microsoft.com/en-us/library/dd351127.aspx

- Understanding How Transport Rules Are Applied
http://technet.microsoft.com/en-us/library/bb124703(EXCHG.140).aspx

- Create a Custom DSN
http://technet.microsoft.com/en-us/library/aa996803.aspx

- Associating a DSN Message with a Transport Rule
http://technet.microsoft.com/en-us/library/bb123506(EXCHG.80).aspx

- Dave Forrest
(Contributions by Scott Landry, Stephen Gilbert and Steve Clagg)

Early 2008 we have posted a blog entry with a VB script that generates some pre-canned reports that are based on message tracking logs. The script has proven to be useful in understanding Microsoft's Exchange work load and guide some design decision for Exchange 2010. This script was developed by Todd Luttinen, Principal Program Manager at Microsoft.

During the development of Exchange 2010, we needed to extended our log analysis beyond just message tracking and to answer a variety of questions that assist with design decisions. This exposed a bottle neck with having a single script that has all the parsing and analyzers bundled together.

This resulted in the creation of ExLogAnalyzer by Victor Boctor, Principal Architect at Microsoft. ExLogAnalyzer was developed in C# with the following goals:

• Separation of syntax and semantics.
• Multi-Server support (process log files that span multiple servers). Log events across servers are processed in chronological order.
• Multi-Log Type support (process / cross reference logs of different log types to produce a single report). Log events across log types are processed in chronological order.
• Provide an extensibility model to support rapid development and distribution of extensions (to support new log types) and analyzers (to encapsulate reporting logic).
• Ability for the community to develop their own analyzers or even extensions.
• Support for Exchange 2007 / 2010 log types.

The main shift in this model, compared to the previous script, is that ExLogAnalyzer is built as a framework that can be used to analyze Exchange as well as possibly any other log format. New log types are supported via plugins called "extensions". Extensions are responsible for doing all the parsing and converting of log lines into events, where each event triggers a method and passes all the pre-parsed information as the event arguments. The specific reports are also implemented as plugins known as "analyzers", where each analyzer handles the events it is interested in and does the appropriate accounting and report generation (typically in CSV format). Implementing each analyzer in isolation (rather than one script that answers multiple questions) makes it much simpler to develop, understand and distribute such analyzers. Such extensions and analyzers can also be easily shared given the plugin model. The following simple diagram summarizes the architecture of this tool:

The ExLogAnalyzer is now released to the community with the following extensions / analyzers available out of the box:

• Message Tracking Log
• MsgTrkTopSendersByDeliverLogAnalyzer - Generates the top 1000 senders based on mailbox deliveries. Messages to the internet are not counted.
• MsgTrkTopSendersBySubmitLogAnalyzer - Provides an analysis of the sender load distribution based on number of messages sent from their mailboxes.
• MsgTrkTopRecipientLogAnalyzer - Generates the top 1000 recipients based on mailbox deliveries. Messages to the internet are not counted.
• MsgTrkMessageSizeDistributionLogAnalyzer - Provides an understanding of the message size distribution.
• MsgTrkRecipientNotFoundLogAnalyzer - Discover and summarize recipients for which "Recipient Not Found" error was generated.
• MsgTrkMailflowVisualizerLogAnalyzer - Generates a directed graph showing the server being analyzed and all the inbound / outbound mail flow paths.
• MsgTrkComponentLatencyPercentileLogAnalyzer (E14) - Analyzes the latencies of the different components and determines the latencies experienced by the specified percentiles of messages.
• MsgTrkDuplicateDeliveryLogAnalyzer - Analyzes the sources for duplicate deliveries to Store. Note that end users don't see such duplicates.
• MsgTrkEventFrequencyLogAnalyzer - Provides an understanding of the distribution of the event + source combinations.
• MsgTrkEventTimeDistributionLogAnalyzer - Provides an understanding of the event distribution over time with a per hour resolution.
• MsgTrkExpandLogAnalyzer - Analyzes the distribution list expansion load on the system.
• MsgTrkReceiveLogAnalyzer - Analyzes the distribution of the sources for the messages received by a server or a set of servers.
• Smtp Receive Log
• SmtpReceiveWorkLoadLogAnalyzer - Analyzes the SMTP receive work load over time while tracking tarpitting, client time outs, etc.
• SmtpReceiveDelayedAckLogAnalyzer (E14) - Analysis of delayed ack performance over time. This report provides an overview of the redundancy that is achieved for legacy systems via delayed ack.
• SmtpReceiveFormatterLogAnalyzer - Re-writes the logs with each session in a separate file, it also reformats the log so that the common session information is included in the header, hence, making the session details more readable.
• SmtpReceiveSeparatorLogAnalyzer - Re-writes the logs with each session in a separate file while maintaining the exact log format.
• SmtpReceiveSessionIndexLogAnalyzer - Provides a summary of all sessions processed within the provided logs.
• Connectivity Log
• ConnectivityWorkLoadLogAnalyzer - An analyzer that samples the connections over time. This analyzer generates a CSV file per source (e.g. SMTP or MAPI).
• ConnectivityStatsLogAnalyzer - An analyzer that provides the frequency of sessions, failed and DNS failures per source + destination combination.
• ConnectivityFormatterLogAnalyzer - Re-writes the sessions as a file per session, moved all the common session information to the header to make the sessions more readable.

Sample Reports

Following are some samples to provide a feel of the outputs of some of these analyzers.

Mail Flow Visualizer (demonstrated possible visualization using directed graphs):

Message Size Distribution:

SmtpReceiveFormatterLog (log re-writing for splitting sessions and making them more readable):

# Session Id: 08CBDCECE3DDF231
# Start Time (local): 2009-07-28T11:07:46.922
# End Time (local): 2009-07-28T11:07:46.953
# Start Time (UTC): 2009-07-28T18:07:46.922Z
# End Time (UTC): 2009-07-28T18:07:46.953Z
# Disconnect Type: Local
# Connector Id: MyServer\MyServer_CrossForest
# Local End Point: 157.54.7.153:25
# Remote End Point: 157.54.71.39:4183

0000000,+,,
0000000,*,None,Set Session Permissions
0000000,*,SMTPSubmit SMTPAcceptAnyRecipient SMTPAcceptAuthenticationFlag SMTPAcceptAnySender SMTPAcceptAuthoritativeDomainSender BypassAntiSpam BypassMessageSizeLimit SMTPAcceptEXCH50 AcceptRoutingHeaders AcceptForestHeaders AcceptOrganizationHeaders SMTPAcceptXShadow,Set Session Permissions
0000000,>,220 MyServer E14 Cross Forest,
0000000,<,EHLO otherhost.otherforest.microsoft.com,
0000000,>,250-MyServer.redmond.corp.contoso.com Hello [157.54.71.39],
0000000,>,250-SIZE 10485760,
0000000,>,250-PIPELINING,
0000000,>,250-DSN,
0000000,>,250-ENHANCEDSTATUSCODES,
0000000,>,250-AUTH,
0000000,>,250-8BITMIME,
0000000,>,250-BINARYMIME,
0000000,>,250-CHUNKING,
0000000,>,250-XEXCH50,
0000000,>,250 XSHADOW,
0000000,<,XSHADOW 3333YTkxYjEtYzE1OC00NDcxLWI4OTktMDA2NDI5YmVmZWRlQFRLNUVYMTRNTFRXNjUxLndpbmdyb3VwLndpbmRlcGxveS5udGRldi5taWNyb3NvZnQuY39t,
0000000,>,250 q7rdaFIdKk3NNRTbjRsjrQ==,
0000000,<,MAIL FROM:<sender@contoso.com> SIZE=25477 XSHADOW=70136df4-c89b-4700-9654-b642c4eb78bb,
0000000,*,08CBDCECE3DDF231;2009-07-28T18:07:46.922Z;1,receiving message
0000000,<,RCPT TO:<receiver@contoso.com> ORCPT=rfc822;receiver2@contoso.com,
0000000,>,250 2.1.0 Sender OK,
0000000,>,250 2.1.5 Recipient OK,
0000000,<,XEXCH50 1136 2,
0000000,>,354 Send binary data,
0000015,>,250 2.0.0 XEXCH50 OK,
0000015,<,BDAT 25477 LAST,
0000031,>,250 2.6.0 <DB82FD8C490D4F43ACE766C04B23A7050F0F12@someserver.otherforest.contoso.com> [InternalId=16796908] Queued mail for delivery,
0000031,<,XQDISCARD 50,
0000031,>,251 OK, no discard events,
0000031,<,QUIT,
0000031,>,221 2.0.0 Service closing transmission channel,
0000031,-,,Local

Top Senders by Submit (analysis yielding CSV - full report has top 1000):

Sender Distribution by Submit (analysis yielding CSV):

Distribution Group Expansion Analyzer (analysis yielding CSV):

Getting started

• Download ExLogAnalyzer from here.
• Checkout the Power Point slide deck included in the download for more details about ExLogAnalyzer.
• Use ExLogAnalyzer and its distributed sample analyzers to analyze your logs.
• Develop your own analyzers. Visual Studio and Visual C# Express Edition are the recommended tools. However, you can use Notepad or your favorite editor, given that ExLogAnalyzer detects and compiles the analyzer CSharp files at runtime.
• Provide us with feedback about ExLogAnalyzer, sample analyzers or the development process.
• Share your analyzers or ideas for useful new analyzers with the Exchange community.

- Victor Boctor

Get-TransportRulePredicate BetweenMemberOf | fl

The output:

Addresses :
Addresses2 :
LinkedDisplayTextException : except when the message is sent between members of
distribution list and distribution list
Name : BetweenMemberOf
Rank : 6
LinkedDisplayText : between members of <a id="Addresses">distribution
list</a> and <a id="Addresses2">distribution list</a>

The BetweenMemberOf predicate requires you to specify two values— Addresses, and Addresses2.

• Step 1: Instantiate the BetweenMemberOf predicate
  

  $condition = Get-TransportRulePredicate BetweenMemberOf

   

• Step 2: Add the values to the new condition we instantiated. As seen in step 1, the BetweenMemberOf predicate requires two values— the identity of two distribution groups.

  $condition.Addresses = @(Get-DistributionGroup Brokers)
  $condition.Addresses2 = @(Get-DistributionGroup Bankers)

• Step 3: Instantiate the RejectMessage action

  $action = Get-TransportRuleAction RejectMessage

• Step 4: Add value to the action we instantiated
  

  $action.RejectReason = "Members of Bankers and Brokers distribution groups are not allowed to send email to each other."

• Step 5: Create the new rule and add the instantiated condition and action to it.

  New-TransportRule -Name "EthicalWall" -Condition @($condition) -Action @($action)

Creating a Transport Rule in Exchange 2010

In Exchange 2010, we’ve made the following changes to the transport rule cmdlet experience based on your feedback.

• You’re no longer required to instantiate predicates and actions
• All predicates and actions are available in-line as parameters of the New-TransportRule and Set-TransportRule cmdlets, resulting in a much improved experience.
• You can now create and modify transport rules using a single command!

Find The Parameter

Before we start creating rules using the shell, we must understand the relationship between predicates, predicate properties, and parameters. Remember, all properties are now parameters for the New-TransportRule and Set-TransportRule cmdlets! So, Properties = Parameters. The key is to determine the predicate or action you want to use and find the parameters it requires.

Exchange 2010 has many new predicates and actions. The Get-TransportRulePredicate and Get-TransportRuleAction cmdlets let you easily list them, and also determine the properties required for each. You can also look up Exchange 2010 predicates and actions in Transport Rule Predicates and Transport Rule Actions in Exchange 2010 documentation.

Using the Get-TransportRulePredicate cmdlet provides the following output:

Once you've determined the predicate you need to use in a rule, you can list all the properties it requires. Some predicates require a single property, so the predicate name and the parameter name (used with New/Set-TransportRule cmdlets) are the same. For example, the From predicate takes a single property— an array of senders. The parameter is also called From. Some predicates require multiple properties. For example, the ADAttributeComparison predicate requires you to specify the name of the Active Directory attribute you want to compare, and a comparision operator (equal / not equal). The parameters are called ADComparisonAttribute and ADComparisonOperator.

You can simply cycle through (hit -, followed by the Tab key— known as "tab-complete" or "tab-through" in shell-speak... ) all available parameters of the New-TransportRule and Set-TransportRule cmdlets and chances are you would recognize the ones you need to use when they show up. If you remember the parameter name starts with the letter B, you can type that first letter and tab through all parameters starting with the letter B (New-TransportRule "Rule Name" -B + the Tab key to tab-through). You've just narrowed down the number of parameters to 3 - that's a lot less tabbing through.

This example lists properties of the the BetweenMemberOf predicate.

Get-TransportRulePredicate BetweenmemberOf | fl

The output (redundant information removed):

Addresses :
Addresses2 :
Name : BetweenMemberOf
Rank : 6
LinkedDisplayText : between members of <a id="BetweenMemberOf1">distribution list</a> and <a id="BetweenMemberOf2">distribution list</a>

The LinkedDisplayText is the predicate description you see on the Conditions page in the EMC rules wizard (see Figure 1). The properties required are enclosed in <a> and </a> tags in the LinkedDisplayText. The predicate requires 2 properties – Addresses and Addresses2. The parameter names for these properties (shown as the link ids) are called BetweenMemberOf1 and BetweenMemberOf2. The link text shown in EMC is distribution list for both properties - indicating the type of values required.

Figure 1: The LinkedDisplayText is the predicate description displayed in the EMC rules wizard. The two properties required for this predicate are called BetweenMemberOf1 and BetweenMemberOf2.

Ditto for all properties of an action. This example lists properties of the the RejectMessage action.

Get-TransportRuleAction RejectMessage | fl

The output:

RejectReason : Delivery not authorized, message refused
EnhancedStatusCode : 5.7.1
Name : RejectMessage
Rank : 14
LinkedDisplayText : send <a id="RejectMessageReasonText">rejection message</a> to sender with <a id ="RejectMessageEnhancedStatusCode">enhanced status code</a>

The RejectMessage action requires two properties - RejectReason and EnhancedStatusCode. The parameters are named RejectMessageReasonText and RejectMessageEnhancedStatusCode. The LinkedDisplayText is the action description shown on the Actions page in the rules wizard in EMC (see Figure 2). If a value isn't specified for the EnhancedStatusCode, the default value is 5.7.1.

Figure 2: The LinkedDisplayText is the action description displayed in the EMC rules wizard. The two properties required for this action are called RejectMessageReasonText and RejectMessageEnhancedStatusCode.

Let's create the same transport rule in Exchange 2010.

New-TransportRule -Name EthicalWall -BetweenMemberOf1 Brokers -BetweenMemberOf2 Bankers -RejectMessageReasonText "Members of Bankers and Brokers distribution groups are not allowed to send email to each other."

Yes, that's it - it's a one-liner!

Common Errors

The more common error you may make when creating rules is not specifying all parameters required for a predicate. For example, you may specify the BetweenMemberOf1 parameter, but not BetweenMemberOf2. Luckily, the shell provides some great to-the-point feedback in such cases.

When the BetweenMemberOf1 parameter is specified, the following parameters must also be specified: BetweenMemberOf2.
   + CategoryInfo : InvalidArgument: (BetweenMemberOf1:String) [New-TransportRule], ArgumentException
   + FullyQualifiedErrorId : 760A4623,Microsoft.Exchange.MessagingPolicies.Rules.Tasks.NewTransportRule

Another common error is specifying an incorrect value for a parameter, when the cmdlet expects a value from a fixed list of values. In this example, we've provided an incorrect value ("foo") for the ADComparisonAttribute parameter. In such cases, the shell lists all the correct values, as shown in the following example.

Cannot process argument transformation on parameter 'ADComparisonAttribute'. Cannot convert value "foo" to type "Microsoft.Exchange.MessagingPolicies.Rules.Tasks.ADAttribute" due to invalid enumeration values. Specify one of the following enumeration values and try again. The possible enumeration values are "DisplayName, FirstName, Initials, LastName, Office, PhoneNumber, OtherPhoneNumber, Email, Street, POBox, City, State, ZipCode, Country, UserLogonName, HomePhoneNumber, OtherHomePhoneNumber, PagerNumber, MobileNumber, FaxNumber, OtherFaxNumber, Notes, Title, Department, Company, Manager, CustomAttribute1, CustomAttribute2, CustomAttribute3, CustomAttribute4, CustomAttribute5, CustomAttribute6, CustomAttribute7, CustomAttribute8, CustomAttribute9, CustomAttribute10, CustomAttribute11, CustomAttribute12, CustomAttribute13, CustomAttribute14, CustomAttribute15".
   + CategoryInfo : InvalidData: (:) [New-TransportRule], ParameterBindin...mationException
   + FullyQualifiedErrorId : ParameterArgumentTransformationError,New-TransportRule

Figure 3: Certain properties require a value from a fixed set of values. In EMC, these can be selected from a drop-down list. If you specify an unexpected value when using the shell, a list of all correct values is provided.

View Transport Rules

The Get-TransportRule cmdlet retrieves a list of all transport rules. As a result of having all predicate and action properties available as parameters, when you use the Get-TransportRule cmdlet to view the properties of a transport rule (Get-TransportRule <rule name> | fl, where fl is short for the format-list cmdlet), you’ll see the large number of parameters scroll by rather quickly, making it difficult to determine which parameters are actually used for the cmdlet. This is a tradeoff we had to make when redesigning the cmdlet experience for transport rules.

However, there’s an easy workaround. Each rule has the Conditions and Actions properties which list the predicates and actions used. This example retrieves the rule conditions.

(Get-TransportRule EthicalWall).Conditions | fl

The output:

You can use (Get-TransportRule <rule name>).Exceptions to list all exceptions and (Get-TransportRule <rule name>).Actions to get a list of all actions specified in the rule. This example does not use any exceptions. The following output shows the actions:

Having all predicate properties and actions available as parameters also allows you to filter the rules based on these parameters. For example, to get a list of all transport rules where the Brokers distribution group is used in the BetweenMemberOf predicate:

Get-TransportRule | Where {$_.BetweenMemberOf1 -like "*Brokers*" -or $_.BetweenMemberOf2 -like "*Brokers*"}

Or you can list all transport rules that perform a particular action. This example lists all transport rules that use the RejectMessage action.

Get-TransportRule | Where {$_.RejectMessageReasonText -ne $null}

Modify a transport rule

Now you can use the Set-TransportRule cmdlet and modify the –BetweenMemberOf and/or –BetweenMemberOf2 parameters, and specify different distribution groups. You can also add additional predicates to the condition, or add exceptions. This example adds the additional predicate SubjectOrBodyContains so only messages that are sent between the two distribution groups AND with the word "trade" in the message subject or body are blocked by the rule.

Set-TransportRule EthicalWall -SubjectOrBodyContainsWords "trade"

If you decide not to use the BetweenMemberOf predicate, you can remove it by setting both its properties to $null.

Set-TransportRule EthicalWall –BetweenMemberOf1 $null –BetweenMemberOf2 $null

Note: If you don’t use any predicates or exceptions in a rule, the rule applies to all messages.

Enable, Disable and Remove Transport Rules

You can use the Enable-TransportRule and Disable-TransportRule cmdlets to enable or disable transport rules.

Enable-TransportRule EthicalWall

When disabling a transport rule using the Disable-TransportRule cmdlet, you’re greeted with a confirmation prompt. You can override the prompt using the –Confirm parameter.

Disable-TransportRule EthicalWall –Confirm:$false

This example disables all transport rules and overrides the confirmtaion prompt.

Get-TransportRule -State Enabled | Disable-TransportRule –Confirm:$false

Similarly, you can remove transport rules using the Remove-TransportRule cmdlet.

Change Transport Rule Priority

Tranpsort rules are executed by the rules agents on the Hub and Edge Transport servers based on their priority, in ascending order. The lowest priority value is 0, assigned to the first transport rule you create. For each successive rule you create, the priority is incremented by 1. You can change the priority of a rule to change the rule execution order. The priority values for other transport rules is adjusted (increased or decreased) depending on the new priority you assign to the rule you’re modifying.

For example, if you have 4 transport rules called EthicalWall1, EthicalWall2, EthicalWall3 and EthicalWall4, with the default priority assigned at creation (0 for EthicalWall1 and 3 for EthicalWall4). If you want to ensure the EthicalWall2 rule is executed last, you can use Set-TransportRule cmdlet to modify rule priority

Set-TransportRule EthicalWall2 -Priority 3

Remember, the highest priority is n-1, where n is the total number of rules. This also allows you to easily determine the total number of transport rules. Use Get-TransportRule | Select -Last 1. The total number of rules = priority of last rule + 1. Alternatively, you can use (Get-TransportRule).count.

Getting all parameters for a cmdlet

Here's a tip that many shell jockeys find useful. During Exchange 2007 development, I found myself frequently asking the same question— How can I list all the parameters for a cmdlet? Cmdlet help (Using Get-Help <Cmdlet Name> -Full or -Detail) would show me a lot of details, including the parameter name, type, description, examples, etc. — but I just need the list of all parameters! Cmdlet reference documentation has all this information, well-formatted and perhaps more readable than Shell output, but it requires firing up a browser window. Most shell jockeys would consider that inefficient! It wasn’t quite possible in Exchange 2007.

In Exchange 2010, you can get a list of parameters for any cmdlet using the following syntax.

(Get-Command <Cmdlet Name>).Parameters

For example, to get a list of all parameters for the New-TransportRule cmdlet:

(Get-Command New-TransportRule).Parameters

So, how many parameters does New-TransportRules cmdlet have? Use the following command to find out!

(Get-Command New-TransportRule).Parameters.Count

Bharat Suneja

EDIT 10/18/2010: Removed the reference to upcoming fix (no fix current planned).

Some customers are finding that when they try to replicate their public folder content to Exchange 2010, it will not replicate. Looking at the application log, you may find an event like this:

Log Name: Application
Source: MSExchange Store Driver
Event ID: 1020
Level: Error
Description:

The store driver couldnt deliver the public folder replication message "Hierarchy (PublicFolder@contoso.com)" because the following error occurred: The Active Directory user wasn't found.

This happens because we assume that if a Servers container exists, there will be a System Attendant object somewhere inside it. However, in an environment where Exchange 2000 or 2003 previously existed, and all those servers have been removed, you probably have a First Administrative Group or some other Administrative Group still hanging around with a Servers container, but no servers inside it. When the Exchange 2010 Store Driver stumbles across this empty Servers container, this error is the result.

To work around the issue, get rid of that empty Servers container. Exchange System Manager won't let you delete it, but you can use the ADSI Edit tool to remove it. Just be sure that the Servers container you are deleting is one that has nothing in it!

This workaround invariably brings up the question: Can't we just delete the old Administrative Group since we don't need it anymore? After all, Exchange System Manager will let you gracefully delete an Administrative Group. The answer to that is a little more involved.

The current documentation on TechNet strongly recommends against deleting old admin groups, and it states that if you do, and you see mail backing up in the System Attendant mailbox, you must then use ADModify to change the legacyExchangeDN on all your users from the old admin group. However, this is not entirely true, and that documentation is being changed.

Items stuck in the System Attendant mailbox

The System Attendant is responsible for publishing free/busy messages to public folders for certain clients such as OWA. When it can't publish these items to the appropriate free/busy public folder, the items stay in the System Attendant mailbox until the problem is corrected.

The most common cause of this problem is that there is simply no replica of the free/busy folder for a given admin group, which usually happens when the public folder stores for that admin group have been ungracefully removed (such as deleted through ADSI Edit) without properly replicating all the content off. This actually has nothing to do with the removal of the admin group itself, but rather with the removal of the public stores in that admin group.

Fortunately, this problem is easy to correct. Forcibly deleting all the public stores in an admin group does not delete the free/busy folder from the hierarchy - it just means all the replicas are gone. To correct this, you can use your public folder admin tool of choice to navigate to the System Folders and find the free/busy folder in question. Then, just add a replica of that folder to a public store in some other admin group. After allowing time for replication, the System Attendant mailbox should clear.

The important point here is that mail backing up in the System Attendant mailbox is not a reason to change the legacyExchangeDN on the users from that old admin group. Just add a replica, and you should be good to go.

A missing free/busy folder

Removing a legacy admin group does not typically cause a problem. However, there is a possible situation to be aware of.

Every admin group has a siteFolderServer attribute. This attribute points to the public folder store that is responsible for the free/busy folder for that admin group. Most of the time, that public folder store doesn't have to do anything, but it is responsible for making sure that the free/busy folder exists. If the free/busy folder for that admin group is missing, it's up to that public folder store to create it.

You can't delete the free/busy folder through any admin tool (ESM, the cmdlets, etc, won't let you), or even through something like ADSI Edit - it's an object in the store, not in the Active Directory. However, it is theoretically possible that somehow, that folder could go missing. If you deleted all your public folder databases and started over with clean ones, or something else unusual happened, you could end up in a situation where there is no free/busy folder for a particular legacy admin group. If that legacy admin group no longer exists in the directory, and thus has no siteFolderServer specified, then the free/busy folder will not get recreated, and you'll see items backing up in the System Attendant.

Even in this situation, there's a fairly easy way to fix the problem. If you still have Exchange System Manager, you can use it to recreate the legacy admin group. Alternately, you can use ADSI Edit to do the same. The important thing is to make sure the legacyExchangeDN is correct - make sure it matches the legacyExchangeDN on the users that were created in that old admin group. On the new admin group object, make sure you have a siteFolderServer that points to an existing public store in some other admin group. Within 24 hours, the free/busy folder for that admin group will get recreated.

If you don't want to recreate the admin group, then your other option at this point is to change the legacyExchangeDN on the users from that admin group. The steps for this are still documented in TechNet.

Our recommendation

We recommend you leave the old admin groups around, simply because there's no reason to remove them. Also, it's possible your free/busy folder could go missing at some point, and then you either have to recreate the admin group or change the legacyExchangeDN on the users.

If you decide to remove an admin group anyway, you should always do it through Exchange System Manager, which will prevent you from deleting it if it still contains objects you need - like the public folder hierarchy object. Deleting the admin group while it still contains the hierarchy object will completely break public folder replication and your ability to administer the folders, among other things.

For these reasons, our recommended workaround for the public folder replication issue is to delete the empty Servers container using ADSI Edit. But technically, yes, you could delete the admin group - gracefully from ESM - to achieve the same end. This doesn't usually cause a problem, and situations where you have to change the legacyExchangeDN of the users should be pretty rare.

- Bill Long

One of the things about Exchange 2010 that many of my customers find very attractive (and I have to agree with them) is the idea of the multi-role or “all-in-one” DAG server. This means having all three of the core Exchange 2010 roles installed on all of the servers in the DAG – Mailbox, Hub Transport and Client Access. There are a lot of reasons why this is an attractive solution, but here are a few of the main ones:

• All servers in the Exchange environment (discounting Unified Messaging and Edge) are exactly the same – same hardware, same configuration, etc. This simplifies ordering the hardware, maintenance and management of the servers.
• In many cases you end up with less Exchange servers in the environment. This is interesting because usually the operational expenditure is higher than the capital expenditure, meaning that it costs more to run a server than it did to purchase it in the first place.
• Less servers means that you also have less of some other things – a sort of “trickle down” effect. For instance, you might have a lower number of network switch ports needed, and for large customers this could be significant. Or maybe you need less physical space for these servers, and for some customers, space is a significant issue.
• The idea of “all roles on one server” allows you to take full advantage of the new hardware capabilities out there today. As part of the design, you want to ensure you utilize as much of the hardware as possible to drive down the cost/mailbox; the latest hardware is quite fast from a megacycle perspective, which means you are either increasing the scale on the server in terms of the number of mailboxes, virtualizing or deploying multiple roles.  For the multi-role server, Microsoft supports up to 24-cores – that is a 4-socket, 6-cores per socket server. Granted, this is only useful for larger customers, but for those larger customers it can be quite important!

But, when talking about these multi-role solutions, I always make sure to let my customers know that this is a starting point. There are no technical reasons why having the roles combined like this is “better” than having the roles separated. Exchange 2007 didn’t support CAS and HT on clustered servers, and we took a lot of customer feedback to help us decide that we needed to support that in Exchange 2010. That doesn’t mean that it is better, it means that it is another option, and if it is right for your environment, then great! If not, well, having separate CAS/HT servers (or separate CAS and separate HT servers) is fully supported and is still a valid solution model.

In fact, there are a couple of things that you need to carefully consider before deciding that the multi-role server is right for you. First is the idea that by putting your CAS role on the DAG member servers, you are forcing yourself to require a hardware load balancer (or third party software load balancer). The DAG still leverages Windows Failover Clustering as a container that defines the boundaries of the DAG and to help the DAG determine whether quorum can be met (amongst other things). When you combine that fact with the idea that you cannot have Windows Load Balancing Services loaded on a server that also has Windows Failover Clustering loaded, you are forced to find another solution for load balancing. For smaller customers, or for branch office scenarios, this might be a deal-breaker. There are some less expensive hardware load-balancing solutions out there, but for some customers, even those might be too expensive, which means that this multi-role server idea won’t work.

Another thing to consider as you determine the architecture you want to utilize for your Exchange 2010 environment is how you will patch your highly available servers. You don’t deploy a DAG unless you really need mailbox resiliency within the datacenter and/or between physical locations – it is a very expensive way to deploy Exchange 2010 if your requirements don’t drive you to need mailbox resiliency! So, if you are spending that money, you need to make sure you understand how to patch these servers and provide the highest level of availability possible.

As you think about these multi-role solutions, also remember the fact that all of the Client Access role servers will be identified as part of the CAS array in a given Active Directory site (this is automatic). When you configure your hardware load balancers, you will need to add all of these Client Access servers into the load-balanced array to allow them to actually be utilized for client access. But, if you’ve done that, how do you patch your servers? If you patch one of the servers (for ease of writing, let’s assume RTM to RU1) and add it back into the array, you now have the possibility a Client Access server at RTM (one of the un-patched servers) fronting a mailbox on RU1 (the newly patched server)! Not good – we all know that you patch these servers in alphabetical order – CAS, HT, MBX – and that means you aren’t supposed to have the mailbox at a newer build than the Client Access server!

So, what do you do about that? We’ll look at some scenarios in the rest of this article, and talk at a high level about the process you’ll take to ensure that you don’t get into a situation where your mailbox is at a newer build than the CAS in front of it.

All of the scenarios below have the patching impact that you will have to manipulate your load-balanced array every time you patch your servers. This possibly means coordinating with another team and putting some management load on them. Of course, any time you patch a CAS array, you’ll probably need to interface with the load-balancer team anyway – you need to “drain stop” each individual CAS server as you patch it anyway to keep the client disconnects and reconnects as low as possible. So, really, this might add a little management overhead to the load-balancer team, but it is possible that it isn’t a significantly high amount of additional work.

The only way you can balance this “additional work” for the load-balancer owners is the fact that HA costs money. You have to remember that. The higher you want your availability to be, the more it costs. Whether it is you or one of your customers, this core tenet of HA must be kept in mind. The other option is to just say that your maintenance window is a time when taking email services completely down is acceptable. But, then again, if you’re spending all this money on HA, is that really an option?

Patching Scenario – Small Office / Branch Office

This is the simplest DAG architecture out there. We’re talking about a single DAG in a single location with only 2 or 3 servers. This is for HA only – no site resilience. For our example here, we’ll look at a 3-member DAG – see this simple diagram:

How do we patch this DAG? Here are a few steps:

1. As is always necessary when moving databases around on your DAG, ensure that your replication is healthy and all copies are up to date.  For the purposes of this example, we’ll start the upgrade process with Server 3.
2. Activation block Server 3 so that a failure at this time won’t activate copies of the mailbox databases on that server.
3. Perform a switchover of all databases away from Server 3 to Server 1 and Server 2.
4. Drain-stop all connections to the CAS on Server 3, and then remove it from the load-balanced array (note that we do not remove the server from the CAS Array) and patch it.
5. Add Server 3 back into the load-balanced array, drain-stop Server 1 and Server 2 and remove them from the load-balanced array.
   1. Notice that for a short period, we have both upgraded and not-upgraded servers in the load-balanced array. This is not an issue, because we still have all mailboxes on the not-upgraded mailbox servers.
6. Remove the activation block on Server 3, activation block Server 2, perform the switchover of all databases from Server 2 to Server 1 and Server 3, patch Server 2 and add it back into the load-balanced array.
7. Remove the activation block on Server 2, activation block Server 1, perform the switchover of all databases from Server 1 to Server 2 and Server 3, patch Server 1 and add it back into the load-balanced array.
8. Remove the activation block on Server 1, and redistribute your databases evenly across all three servers (Note that there will be a pretty sweet script in Exchange 2010 Service Pack 1 to do this for you!).

Impact(s) of this process:

• At that time when you have patched a single server and you have the entire load-balanced array pointing at that one server (in our example above, when Server 3 has been added into the load-balanced array and Servers 1 and 2 have been removed from the load-balanced array), you have no HA on your CAS services. This makes the assumption that at the time of patching, that one CAS server can handle the load sufficiently. Until you have patched your second server and added it into the load-balanced array, you could have a service interruption by the failure of that one server.
  • Remember that this is only for the time period that you have not patched that second server and added it back into the load-balanced array.
  • Also remember that under normal conditions, this process will cause less interruption of service than taking the entire system offline for the time that we need to patch the 2 or 3 servers – so in most instances you will be at a higher level of availability than you would be if you didn’t follow this procedure.
• You should also keep in mind that during this patching process, you cannot take two servers offline at the same time – a three member DAG (as shown in our example) will lose quorum when you take two members down, and all email services will be unavailable in that case.

Patching Scenario – Medium Sized Deployment

This is a slightly larger environment – a single DAG with 4 or more servers in the DAG, all located in one location. Let’s use a 6-member DAG for our example this time. Refer to this diagram for this example:

Also note that for this example, we’re going to say that we have designed this DAG to support two concurrent failures. This means that if we take two servers out of actively hosting mailboxes for patching, by having three copies of all databases, we are assured that we can continue to provide email services. It is possible to modify this solution to only take a single server out of service at a given time, and that is a perfectly acceptable solution – this is just an example presented here for discussion.

1. Ensure that your replication is healthy and all copies are up to date.
2. In our example we are going to patch two servers at a time, starting with Server 5 and Server 6. 
3. Activation block Server 5 and Server 6 so that a failure at this time won’t activate copies of the mailbox databases on those servers.
4. Perform a switchover of all databases away from Server 5 and Server 6 to the other four servers in the DAG.
5. Drain-stop all connections to the CAS on Server 5 and Server 6, and then remove them from the load-balanced array
6. Patch Server 5 and Server 6.
7. Add Server 5 and Server 6 back into the load-balanced array, drain-stop all other servers and remove them from the load-balanced array.
   1. Notice that for a short period, we have both upgraded and not-upgraded servers in the load-balanced array. This is not an issue, because we still have all mailboxes on the not-upgraded mailbox servers.
8. Remove activation block from Server 5 and Server 6, activation block Server 3 and Server 4, perform the switchover of all databases from Server 3 and Server 4 to the other four servers, patch Server 3 and Server 4 and add them back into the load-balanced array.
9. Remove activation block from Server 3 and Server 4, activation block Server 1 and Server 2, perform the switchover of all databases from Server 1 and Server 2 to the other four servers, patch Server 1 and server 2 and add them back into the load-balanced array.
10. Remove activation block from Server 1 and Server 2, and redistribute your databases evenly across all three servers.

Impact(s) of this process:

• For a period of time, you will be running with a possibly lower availability stance than normal operating conditions. You only have 2 servers providing CAS services until you have patched those other servers and added them back into the load-balanced array. (If you only have 4 servers in the DAG, this might not be the case.)
• In the case where you have very high numbers of users in this physical location, it is possible that you would introduce a performance impact on CAS services, because of the reduced number of Client Access servers in service.
  • Think about the situation where you have 8 or 10 servers in the DAG in this physical location, and you have only patched 2 servers. In that case, those 2 servers could probably not handle the load of all users under full production load. But, you typically won’t be patching during a “full production” time of the day – you’ll have a maintenance window that you will be working in, and users will know to have a lower expectation of availability and such. As long as you understand this and are willing to accept this risk, this is fine, but you should almost certainly make sure that you document this case and make sure that it really is acceptable!
  • The other way to think about this is to make sure that the two servers you bring back into the load-balanced array have enough processing power and memory to support the entire load. This is probably the best engineering solution for a highly available Exchange 2010 environment, but it does have a cost associated with it just like everything else in HA. If I were going to recommend a solution, this is how I would recommend it – make sure you have enough processing that if you end up in a production environment on two upgraded servers, that they can handle your full production peak load.

Patching Scenario – Large Deployment

Multiple DAGs, multiple servers in each DAG, DAGs spread across multiple locations. Think about the scenario where you have two datacenters with two DAGs of 12 servers each, and users active in both datacenters. At any given time, you have 6 servers in a passive mode in each of the two datacenters. This is for big customers – a lot of my customers are very large, and I’m working with three customers right now: 120K mailboxes, 250K mailboxes and 600K mailboxes.

To help define this environment, here is a relatively simple diagram of two DAGs showing the replication data flow direction.

Now, how to patch this beast… This example will discuss patching the West Datacenter servers – just repeat this process for the East Datacenter after completing the West Datacenter upgrades.

1. Ensure that all replication is healthy and all copies are up to date on both DAGs.
2. Activation block all of the DAG2 servers in the West Datacenter to keep databases from failing over to these servers.
3. Ensure no databases from DAG2 are active in the West Datacenter. If so, perform the switchovers necessary to move those databases back to the East Datacenter and ensure that all replication comes back healthy and all copies come up to date.
4. In the West Datacenter only, drain-stop all of the servers in DAG2 and remove those servers from the West Datacenter load-balanced array.
5. Patch all DAG2 servers in the West Datacenter.
6. Add all DAG2 servers in the West Datacenter back into the load-balanced array and remove the activation block on those servers.
7. Drain-stop all DAG1 servers in the West Datacenter and remove them from the West Datacenter load-balanced array.
8. Work through the DAG1 servers patching them “two by two”. This means to move active mailboxes off the servers two at a time, patch those two servers, move databases off of two more servers, and patch. This would probably look like this (assuming you had 6 servers from DAG1 in the West Datacenter):
   1. In our example we are going to patch two servers at a time, starting with Server 5 and Server 6 (of DAG1 in the West Datacenter).  Activation block Server 5 and Server 6 so that a failure at this time won’t activate copies of the mailbox databases on those servers.
   2. Perform a switchover of all databases away from Server 5 and Server 6 to the other four West Datacenter servers in DAG1.
   3. Patch Server 5 and Server 6.
   4. Remove activation block from Server 5 and Server 6, activation block Server 3 and Server 4, perform the switchover of all databases from Server 3 and Server 4 to the other four servers in the West Datacenter, and patch Server 3 and Server 4.
   5. Remove activation block from Server 3 and Server 4, activation block Server 1 and Server 2, perform the switchover of all databases from Server 1 and Server 2 to the other four servers in the West Datacenter, and patch Server 1 and server 2.
   6. Remove activation block from Server 1 and Server 2, and redistribute your databases evenly across all of the DAG1 West Datacenter servers.
9. Once you have patched all servers from DAG1 in the West Datacenter and evenly distributed your databases, you can then add all of the West Datacenter DAG1 servers back into the West Datacenter load-balanced array, completing the patch of the West Datacenter Exchange servers.

Impact(s) of this process:

• While you are patching, it is probable that your site resilience stance will be lower. Think about the scenario where you have a datacenter failure while the servers in the other datacenter are in the process of being patched. If half of your servers have been patched, then this could cause a performance issue in the case where the “passive” datacenter needs to be activated, or it could add time to the activation process while other server patches are completed.

Conclusion

Most of this isn’t “rocket science” – it is just something to think about. We have to be aware that in some instances, especially in those very small environments (small orgs or branch offices), we might want to look at another solution such as virtualizing the whole thing and using Windows NLB instead of using the multi-role servers. This goes back to one of the first things I said – it is all driven by the requirements. If you don’t need mailbox resiliency, don’t deploy a DAG. If your requirements drive you away from the multi-role server, don’t hesitate to go with roles broken out onto separate servers. Just make sure that you make these decisions with your eyes open – understand the implications of everything right down to how you will patch these servers once they have been deployed!

- Robert Gillies

Update 3/31/2011: The updated Exchange 2007 SP3 RU3 has been released. See Announcing the Re-release of Exchange 2007 Service Pack 3 Update Rollup 3 (V2).
3/30/2011: We posted a status update for this issue. See Exchange 2007/2010 Rollup 3 Status Update.

Over the weekend, the Exchange Product Group was made aware of an issue which may lead to database corruption if you are running Exchange 2007 Service Pack 3 with Update Rollup 3 (Exchange 2007 SP3 RU3). Specifically, the issue was introduced in Exchange 2007 SP3 RU3 by a change in how the database is grown during transaction log replay when new data is written to the database file and there are no available free pages to be consumed.

This issue is of specific concern in two scenarios: 1) when transaction log replay is performed by the Replication Service as part of ensuring the passive database copy is up-to-date and/or 2) when a database is not cleanly shut down and recovery occurs.

While only a small number of customers have been affected to date, we believe the risk is significant enough that we are recommending all customers to uninstall Exchange 2007 SP3 RU3 on all Mailbox Servers and Transport servers. Uninstalling the rollup will revert the system back to the previously installed version. We have also removed the Exchange 2007 SP3 RU3 download from the Microsoft Download Center and from Microsoft Update until we are able to produce a new version of the rollup.

We are actively working this issue and based on test results plan to release an updated version of Exchange 2007 SP3 RU3 to the Download Center later this week. In addition, we are conducting an internal review of our processes to determine how to prevent issues such as this in the future.

When this issue occurs, the following similar events are logged in the Application Event log of the Mailbox server. Regardless of whether you see these types of events, you should review the recovery instructions and begin that process. If you are uncomfortable performing any of these steps please contact Microsoft Support for assistance.

• Event ID: 454
  Event Type: Error
  Event Source: ESE
  Event Category: Logging/Recovery
  Description: Microsoft.Exchange.Cluster.ReplayService (12716) Recovery E20 SG1\DB1: Database recovery/restore failed with unexpected error -4001.
• Event ID: 2095
  Event Type: Error
  Event Source: MSExchangeRepl
  Event Category: Service
  Description: Log file D:\logs\SG1\E200006AFAE.log in SG1\DB1 could not be replayed. Re-seeding the passive node is now required. Use the Update-StorageGroupCopy cmdlet in the Exchange Management Shell to perform a re-seed operation
• Event ID: 2097
  Event Type: Error
  Event Source: MSExchangeRepl
  Event Category: Service
  Description: The Microsoft Exchange Replication Service encountered an unexpected Extensible Storage Engine (ESE) exception in storage group 'SG1\DB1'. The ESE exception is a read was issued to a location beyond EOF (writes will expand the file) (-4001) ().

In addition, in environments utilizing Continuous Replication, comparison of the database file between the active and passive nodes will indicate that the database file has decreased in size.

Regardless of whether you are experiencing this issue, we strongly recommend taking the below actions to ensure that you do not experience any data loss or outage event associated with this issue.

For example:

• If you have deployed your Mailbox servers utilizing Cluster Continuous Replication (CCR), failure of the active copies may affect your service SLA as you may have no viable passive copies to activate. Hardware failures may result in you not having a means to recover up to the point of failure and thus may experience data loss.
• If you have deployed your Mailbox servers utilizing Single Copy Clusters (SCC), switchovers or failovers may result in this issue as there is only one copy of the database and recovery is performed during switchovers and failovers.

For environments leveraging CCR and/or Standby Continuous Replication (SCR)

If you note the listed events in your environment the following steps must be taken in order to restore your high-availability configuration:

1. Rollback the CCR Mailbox server hosting the passive database copies and any SCR target Mailbox servers to the previously installed version (e.g., Exchange 2007 SP3 RU2) by uninstalling RU3.
2. Re-seed all affected database copies on the CCR Mailbox server and any SCR target Mailbox servers hosting the passive database copies.
3. Verify the database copy status is healthy for all passive copies.
4. Perform a switchover and rollback the remaining CCR Mailbox server to the previously installed version (e.g., Exchange 2007 SP3 RU2).

If you are not seeing these events in your continuous replication enabled environment, we recommend the following steps:

1. Rollback the CCR Mailbox server hosting the passive database copies and any SCR target Mailbox servers to the previously installed version (e.g., Exchange 2007 SP3 RU2) by uninstalling RU3.
2. Perform a switchover and rollback the remaining CCR Mailbox server to the previously installed version (e.g., Exchange 2007 SP3 RU2).

For environments leveraging Single Copy Clusters (SCC)

1. Rollback passive nodes within the SCC environment to the previously installed version (e.g., Exchange 2007 SP3 RU2) by uninstalling RU3.
2. Perform a switchover and rollback the remaining SCC Mailbox server nodes to the previously installed version (e.g., Exchange 2007 SP3 RU2).
3. If you have any databases that will not mount as a result of the above issue, you can restore the damaged databases leveraging a last known good backup.

For environments leveraging standalone Mailbox (or Public Folder) servers

1. Rollback the standalone Mailbox servers to the previously installed version (e.g., Exchange 2007 SP3 RU2) by uninstalling RU3.
2. If you have any databases that will not mount as a result of the above issue, you can restore the damaged databases leveraging a last known good backup.

For Hub Transport and Edge Transport servers

1. Rollback the standalone transport servers to the previously installed version (e.g., Exchange 2007 SP3 RU2) by uninstalling RU3.
2. If any transport servers have mail.que databases which currently do not mount as a result of the above issue, you can recover them by following the steps in Working with the Queue Database on Transport Servers.

Kevin Allison
GM Exchange Customer Experience

Background

The Exchange Store Driver is a core transport component which lives both on the Mailbox server role (as the mail submission service) and the Hub server role. It is responsible for:

• Retrieving messages from the mailbox server that have been submitted by end-users and submitting those to the Hub transport role for categorization and routing.
• Delivering messages to the appropriate mailbox server based on the location of the recipients mailbox.
• Extensibility platform for both mail submission & delivery. Store Driver currently hosts a number of agents that extend the functionality of Exchange. Examples include such agents as Inbox Rules, Conversations, meeting forward notifications, etc.

Exchange 2010 is currently being utilized in Live@EDU, as well as the upcoming Office 365. As you can probably imagine, the Exchange servers that run in those datacenters are loaded and pushed harder than almost any other Exchange server imaginable. Prior to SP1, there were several problems that were encountered with mail delivery to the Exchange mailbox store. In particular, there was a need to make sure that a handful of recipients did not starve the rest of the mail delivery system.

While many of you may not have noticed this problem, Microsoft has seen many of these types of cases over the years; often isolated to a single event like an inadvertent public folder replication storm.

This was despite the message throttling that was already available. Transport roles have also had functionality to avoid resource starvation known as Back Pressure, but this was not designed to protect the system from messages that were already in the Local Delivery queue.

Changes in SP1

In order to further protect both the Mailbox servers and Hub servers from resource starvation, new thread limits were introduced in SP1:

Note: These keys are not present in the EdgeTransport.exe.config file by default.

Is it possible to have too much protection?

Unfortunately, there are two scenarios after applying SP1 where we are seeing customers with messages backing up in the queue. The temporary error message is:

432 4.3.2 STOREDRV.Deliver; recipient thread limit exceeded

As you can probably guess, the two scenarios are:

• Journaling
• Public Folders

In both cases, the deliveries are occurring to a single recipient (or very small number of recipients). This is likely to occur during heavy mail flow. The screen shot below was taken from a lab server while reproducing the issue:

Figure 1: Messages backed up in mailbox delivery queue due to recipient thread limits being exceeded (click here for larger screenshot)

You can see a historical history of 4.3.2 events in connectivity logs on your Hub Transport servers (in the \Program Files\Microsoft\Exchange Server\V14\TransportRoles\Logs\Connectivity\CONNECTLOGxxxxxxxx-x.LOG), like:

#Software: Microsoft Exchange Server
#Version: 14.0.0.0
#Log-type: Transport Connectivity Log
#Date: 2011-01-12T00:00:00.775Z
#Fields: date-time,session,source,Destination,direction,description
 
2011-01-12T00:00:00.775Z,08CD7F1200CBDBD0,MapiSubmission,5f24e416-c380-41b5-bfe0-37b6f1091f49,>,"Failed; HResult: 1140850693; DiagnosticInfo: Stage:LoadItem, SmtpResponse:432-4.3.2 STOREDRV; mailbox server is too busy"
 
2011-01-12T00:00:00.775Z,08CD7F1200CBDBD0,MapiSubmission,5f24e416-c380-41b5-bfe0-37b6f1091f49,-,RegularSubmissions: 0 ShadowSubmissions: 0 Bytes: 0 Recipients: 0 Failures: 1 ReachedLimit: False Idle: False
 
2011-01-12T00:00:05.792Z,08CD7F1200CBDBD1,MapiSubmission,5f24e416-c380-41b5-bfe0-37b6f1091f49,+,Win2k8R2Ex14.dom2k8r2ex14.lab

In some cases, simply leaving the servers alone should cause the queues to slowly drain. In other cases, it may be necessary to take further action because the problem has persisted for a while or because it isn’t part of a one-time event like a Public Folder replication storm.

Alright, I understand. Now how do I fix my situation?

Like every other throttling & performance related feature that has ever been in Exchange, the solution isn’t exactly straight forward. For starters, we’ve seen some level of success simply incrementing both values up one, as follows:

<add key="RecipientThreadLimit" value="2" />
<add key="MaxMailboxDeliveryPerMdbConnections" value="3" />

In fact, there has been enough success with this that we’re considering changing the defaults in a future rollup or service pack. Of course, when we do, the new defaults will only apply to those who haven’t already modified the values. Although it has not yet been released, KB 2491972 will discuss the change when the time comes.

So if +1 is better, then why not +2 or more?

Well, the problem is that all Exchange servers will ultimately be bound by some hardware resource. You don’t want to introduce thrashing or resource starvation due to a public folder or journaling server event that now impacts all users as well. In addition, there are other limits that control the maximum number of threads that can service these types of deliveries. In short, we are not recommending going above 4 for MaxMailboxDeliveryPerMdbConnections, and RecipientThreadLimit should always be at least one fewer.

In an extreme case, if you are in a very busy environment with dedicated journaling mailbox servers, it may be worth considering having dedicated hub transport servers to go along with that. Of course, they can be virtualized or multi-role boxes, but in order to be isolated, the whole bunch will need to be in a dedicated site. Then, you would be able to increase these limits without risking delivery to other mailboxes.

Of course, this is an extreme example. For the rest of us, it may be as simple as trying the +1 approach while carefully monitoring the hub & mailbox servers.

Scott Landry, Steve Schiemann, Frank Brown and Jason Nelson

You may have noticed a change in the behavior of the Safe Senders list within Outlook starting in Exchange 2010. Users can no longer add accepted domains to Outlook’s Safe Senders list.

This was done as an anti-spam deterrent as we have all seen cases where Joe The Spammer spoofs the mail from your own domain. Adding your own domain to the Safe Senders list would bypass all Outlook client-side anti-spam checks, dumping that message from the Nigerian prince (spoofed using your own domain) into your users’ Inboxes. Not so good unless you were really waiting for that business opportunity.

A valid SPF record and our anti-spam agents (specifically the SenderID agent) would go a long way to block these types of spam. However, many customers out there have not exactly jumped on the SPF bandwagon.

You can learn more about SenderID filtering in Sender ID and Understanding Sender ID. Use the Sender ID Framework SPF Record Wizard to create an SPF record for your domain.

With Exchange 2010, you CAN still add individual email addresses from your own accepted domains to the Safe Senders list - you just can’t add the entire domain, as shown in the screenshot above.

What happens if you DO decide to add the whole domain?

If you try to do this for a user via the Shell, you will get the very helpful error below:

We tell you exactly why we are throwing an error.

How about when a user does this via Outlook? First, Outlook lets the user add a domain.

However after a few minutes the entry will magically disappear.

Allowing app servers to send as your own domain

Many customers have various applications that submit mail anonymously to Exchange where the messages come from email addresses from your accepted domains.

Some of you have apps submitting from so many accepted domain addresses that it wouldn’t be feasible (let alone fun) attempting to add all of these addresses individually to the safe senders list in Outlook to ensure these messages do not end up in junk mail.

Now that we can’t add the whole domain, what are our options?

• We know that adding all the addresses manually is an available albeit painful option
• A second option is to disable Outlook’s client side filtering (yeah... not a good idea, so do not seriously consider it. Spam checks out the window!)
• A third and best option is to install the anti-spam agents on your relay hub(s) and add the IPs of your app servers to the IP Allow list of the connection filtering agent as documented here.

When the sending SMTP host’s IP address is on the IP Allow List in Exchange, it bypasses all anti-spam checks (except sender/recipient filtering) and the Content Filter agent stamps an SCL of -1 on the message which Outlook will honor.

X-MS-Exchange-Organization-AuthAs: Anonymous
X-MS-Exchange-Organization-Antispam-Report: IPOnAllowList
X-MS-Exchange-Organization-SCL: -1

So, go ahead and run the Install-AntispamAgents.ps1 from the Scripts folder on your Hub Transport server, and then add IP addresses or subnets of our application servers to the IP Allow List.

If using the Shell, use this command to add an IP address to the IP Allow List:

Add-IPAllowListEntry –IPAddress 192.168.10.120

What Not To Do: Using Externally Secured Authentication

Now I know what you’re thinking: Why don’t we just add Externally Secured Authentication as an authentication type on a Receive Connector, scope the connector’s remote IP range to the sending application servers and call it a day?

Well, not so fast... you see, while Externally Secured gives the sending IP the ms-Exch-Bypass-Anti-Spam extended right, this only circumvents the Exchange Anti-Spam checks, not Outlook’s. And it is Outlook that’s moving the message into junk mail in this case.

Also note that Externally Secured does not stamp any SCL X-headers on the message as an SCL of -1 would’ve bypassed Outlook’s checks. The only header this authentication type creates is the one below:

X-MS-Exchange-Organization-AuthAs: Internal

If you're still confused about Exchange and Outlook anti-spam checks, take a look at Exchange anti-spam myths revealed.

Big thanks to Tak Chow for tech reviewing this post.

Tom Kern

Starting with Exchange 2007, installation of the current version of Office Filter Packs is a prerequisite. For Exchange 2007 and Exchange 2010 RTM, the pre-req was 2007 Office System Converter: Microsoft Filter Pack. In Exchange 2010 SP1, this was updated to Office 2010 Filter Packs. The filters are used by Exchange Search to index email attachments on Mailbox servers and by the transport rules agent to scan attachments on transport servers. The filters are also important for discovery searches in Exchange 2010 – Multi-Mailbox Search uses content indexes generated by Exchange Search.

In Exchange 2010 RTM and Exchange 2007, you need to register the IFilters with Exchange Search (for Exchange 2007, see 'KB 944516 How to register Filter Pack IFilters with Exchange Server 2007'). In Exchange 2010 SP1, IFilters included in the Office 2010 Filter Pack are automatically registered by Exchange Setup. You must register any third-party filters you install.

An updated filter pack, Service Pack 1 for Microsoft Office Filter Pack 2010 (KB2460041) 64-bit Edition is now available on Download Center. The Filter Pack has been tested with Exchange 2010 and Exchange 2007. The updated filter pack doesn’t have any fixes or updates that impact Exchange scenarios or functionality. If you have the RTM version of Office 2010 filter packs installed, you can update it but it’s not required. For new installs, we recommend going with the current versions.

Note: The SP1 package updates the RTM version of Office 2010 Filter Packs.

Bharat Suneja