The customer called to indicate one of his user’s Inbox had been moved to his Deleted Items folder. I had never before seen this, as Outlook prevents users from making such “mistakes” with special folders. I’d heard rumors that previous versions of OWA would allow users to do this, however he insisted that it just appeared this way after the migration. What was really strange was that new email continued to be delivered to this seemingly “deleted” inbox. No attempt to move the folder back to the root of his mailbox would work, and Google turned up little helpful information this time. Deleting the Outlook profile and all of its offline cache goodies proved futile, and I was on the verge exporting his mailbox to a PST file and nuking it, when The Google finally answered.

Turns out some random bloke on a Technet message board had this problem when moving users from Exchange 2003 to 2007, and after some of the same steps I had taken, he had found the solution that worked equally well for me. Moving the user back to the old server, then back AGAIN to the new server put the Inbox back where it belonged. While I don’t understand the root cause of this problem, I’m glad to have solved it without the pains of nuking a mailbox. Just further proves that keeping an old Exchange server around for a few weeks after its migration is a Smart Move™.

All the clients were running Outlook 2003. Some users were seeing duplicates of many of their system-level folders (Inbox, Calendar, etc). All users were unable to access any folder but their Inbox. Trying to view the calendar, contacts, or even a user-created mail folder would cause Outlook to crash. I suspected it had something to do with offline folder files, although deleting the Outlook profile and it’s associated OST files had no affect. A bit of Googling finally turned up this post at joeware, which pointed to this post in the Microsoft newsgroups, which contained the answer.

After some work, we were able to determine why Outlook 2003 crashes after moving mailboxes off of Exchange 2000 onto Exchange 2003. The fix is to add a registry value “Guid-Replid Caching” under HKLM\System\CurrentControlSet\Services\MSExchangeIS\SERVERNAME. Under each mailbox store we added a REG_DWORD of “Guid-Replid Caching” with a value of 0.

Taking their advice, I made the change, restarted the Exchange IS service, and damn if that didn’t solve the problem.

The background

We run a Microsoft ISA Server 2004 firewall to provide Internet access and security for all computers on campus. ISA Server is known for its strong per-user authentication features, and we make heavy use of them. All outbound connections from our workstations must be authenticated, which is handled by the Firewall Client that is installed through group policy. We also use the web caching feature of ISA, so all of our browsers are configured to use the ISA Server as a proxy through the auto-configuration script.

The symptoms

Initial site access was fine, and load times were quick. Subsequent page loads from the same site would begin to slow down, and some page elements (read: images) would fail to load. Once that happened, future attempts to access the site would fail with timeout errors.

The investigation

When testing this problem, I noticed the problem was much more prevalent in Internet Explorer, so I switched this particular user to Firefox for this one site. Things seemed to work better, but reports of page failures still would come in. Other attempts at resolving the problem, including removing proxy settings from Firefox and disabling caching for the site, provided initial but ultimately false hope. Finally, as a desperation attempt, I added a rule to our ISA server to allow any request to the site without authentication or HTTP content checks. I even bumped this rule to the top of the list, to reduce the amount of time the firewall was involved in the transaction. In short, these blind fix attempts proved fruitless.

It was time to take it up a notch, so I busted out network monitor. Traffic between the ISA server and the site appeared to be normal, with no unusual packets or headers to throw things off. Of course, much of the traffic was encrypted with SSL, so network monitor was next to useless. I then began to monitor the log files from the ISA console while troubleshooting the issue, and initially everything seemed to be normal. Then I saw something.

The realization

I noticed that the connections to the site were staying open, even after all the data for that connection was transferred. After ISA’s connection timeout limit was reached, the connections would then close. Strange. I also began to notice that once I got locked out of this site, I was locked out of any type of Internet access for about two minutes. Two minutes, the exact amount of ISA’s timeout limit. Then the clincher, connection attempts to port 8080 and 1745 on the firewall were being denied from my machine. For those of you not familiar with ISA Server, port 8080 is used for the proxy/cache portion of ISA, and port 1745 is the Microsoft Firewall Client communication port. Every single new connection my machine was making to the ISA server was being denied. I began to pore over my ISA settings looking for something that could explain this. I soon found it.

The solution

ISA ships with a pre-defined per-machine connection limit of 40. Once a machine makes 40 simultaneous connections to the ISA server, any new connections are denied. Because the HTTP and SSL connections were being left open, jumping around various pages on this site quickly brought the number of connections to 40. Raising the per-machine connection limit to 160, 4x the default, as well as reducing the timeout to 60 seconds has solved these problems. I’m not quite certain why connections to this site are being left open after all the data for the connection has been transferred, nor have I determined if this problem is specific to this site. I’ll have to break out my TCP/IP book and do a little RFC reading to get that fully figured out. I’m just glad to have finally solved this problem.