The customer called to indicate one of his user’s Inbox had been moved to his Deleted Items folder. I had never before seen this, as Outlook prevents users from making such “mistakes” with special folders. I’d heard rumors that previous versions of OWA would allow users to do this, however he insisted that it just appeared this way after the migration. What was really strange was that new email continued to be delivered to this seemingly “deleted” inbox. No attempt to move the folder back to the root of his mailbox would work, and Google turned up little helpful information this time. Deleting the Outlook profile and all of its offline cache goodies proved futile, and I was on the verge exporting his mailbox to a PST file and nuking it, when The Google finally answered.

Turns out some random bloke on a Technet message board had this problem when moving users from Exchange 2003 to 2007, and after some of the same steps I had taken, he had found the solution that worked equally well for me. Moving the user back to the old server, then back AGAIN to the new server put the Inbox back where it belonged. While I don’t understand the root cause of this problem, I’m glad to have solved it without the pains of nuking a mailbox. Just further proves that keeping an old Exchange server around for a few weeks after its migration is a Smart Move™.

A quick Google search turned up numerous hits about a VMware specific RPM available from APC for v2.21. A quick search of APC’s website turned up no such thing, and downloading the newest release for Linux didn’t get me very far. During the installation it through an error about VMware not being supported. After some further Google digging, I finally found a direct link to the RPM buried on APC’s FTP site. Installing the RPM worked like a champ, and once I opened up the requisite firewall ports in ESX I was able to access the web interface and get it configured.

To save others the same headaches I encountered, I’ve preserved the RPM file on my site until APC decides to support VMware in new releases again. The file is available below.

APC Network Shutdown v2.21 for VMware

All the clients were running Outlook 2003. Some users were seeing duplicates of many of their system-level folders (Inbox, Calendar, etc). All users were unable to access any folder but their Inbox. Trying to view the calendar, contacts, or even a user-created mail folder would cause Outlook to crash. I suspected it had something to do with offline folder files, although deleting the Outlook profile and it’s associated OST files had no affect. A bit of Googling finally turned up this post at joeware, which pointed to this post in the Microsoft newsgroups, which contained the answer.

After some work, we were able to determine why Outlook 2003 crashes after moving mailboxes off of Exchange 2000 onto Exchange 2003. The fix is to add a registry value “Guid-Replid Caching” under HKLM\System\CurrentControlSet\Services\MSExchangeIS\SERVERNAME. Under each mailbox store we added a REG_DWORD of “Guid-Replid Caching” with a value of 0.

Taking their advice, I made the change, restarted the Exchange IS service, and damn if that didn’t solve the problem.

This whole project changed when I happened to check-up on pfSense, a firewall distribution based on FreeBSD. Lo and behold, they had added multiple WAN support over the summer. A quick download and test run later, and I had my winner. It had the raw support for the features that I need, with the polish coming down the pike in the coming months. It was free, since I already had a spare server to put it on. It was configured completely through a web interface, making for easy administration. It was… a done deal.

We went live with the setup before Christmas, and it’s been running flawlessly. Policy-based routing allows me to control which packets go where, and strong NAT/firewall rules make it a breeze to publish services out to the world. I’ve even got it running a fourth interface for a guest VLAN. More on that later…

With the easy part out of the way, I embarked on a quest to manage two WAN links. Our firewall/gateway is a Microsoft ISA Server, which doesn’t support multiple WAN links. The only ISA add-on that does support multiple WAN links has just been deemed end of life by EMC. Just as well, as it was $3000. So, I began looking for hardware solutions. Thus began the hard part.

I found many SOHO solutions that can load balance multiple WAN connections. The list includes the NetGear FVX538, Xincom DPG602, Linksys RV082, and a few others. All of these are low-cost, feature rich solutions, but are obviously built on low-cost hardware. They don’t seem capable of handling the throughput of a large network, as evidenced by their 253 user limit. I’m not even certain what this user limit is, or how it’s enforced. In any event, these seem best suited to small offices of 50 PCs or less. Certainly not ideal for our 300+ node environment.

My next foray led me to the Cisco 1841 router. There are a few users on the ISA Server message boards that seem to be raving about this router. It has many features and options, but my initial impression is that it cannot handle the routing speeds of a broadband cable connection. My CDWG rep is looking into this, so I should have some more information shortly. If it can meet my needs without breaking the bank, I think it’s a sure bet. Enterprise class features and support are always a plus.

Finally, an intriguing option would be to setup a Linux-based router using this article as a guide. This is option has the appeal of low cost, many possibilities with traffic control and QOS, and of course all the smell of a fun project. However, support and service are realistically nil, so the long term outlook isn’t good.

Hopefully I’ll have some more options and answers in the coming days, and I’ll post an update then.

The background

We run a Microsoft ISA Server 2004 firewall to provide Internet access and security for all computers on campus. ISA Server is known for its strong per-user authentication features, and we make heavy use of them. All outbound connections from our workstations must be authenticated, which is handled by the Firewall Client that is installed through group policy. We also use the web caching feature of ISA, so all of our browsers are configured to use the ISA Server as a proxy through the auto-configuration script.

The symptoms

Initial site access was fine, and load times were quick. Subsequent page loads from the same site would begin to slow down, and some page elements (read: images) would fail to load. Once that happened, future attempts to access the site would fail with timeout errors.

The investigation

When testing this problem, I noticed the problem was much more prevalent in Internet Explorer, so I switched this particular user to Firefox for this one site. Things seemed to work better, but reports of page failures still would come in. Other attempts at resolving the problem, including removing proxy settings from Firefox and disabling caching for the site, provided initial but ultimately false hope. Finally, as a desperation attempt, I added a rule to our ISA server to allow any request to the site without authentication or HTTP content checks. I even bumped this rule to the top of the list, to reduce the amount of time the firewall was involved in the transaction. In short, these blind fix attempts proved fruitless.

It was time to take it up a notch, so I busted out network monitor. Traffic between the ISA server and the site appeared to be normal, with no unusual packets or headers to throw things off. Of course, much of the traffic was encrypted with SSL, so network monitor was next to useless. I then began to monitor the log files from the ISA console while troubleshooting the issue, and initially everything seemed to be normal. Then I saw something.

The realization

I noticed that the connections to the site were staying open, even after all the data for that connection was transferred. After ISA’s connection timeout limit was reached, the connections would then close. Strange. I also began to notice that once I got locked out of this site, I was locked out of any type of Internet access for about two minutes. Two minutes, the exact amount of ISA’s timeout limit. Then the clincher, connection attempts to port 8080 and 1745 on the firewall were being denied from my machine. For those of you not familiar with ISA Server, port 8080 is used for the proxy/cache portion of ISA, and port 1745 is the Microsoft Firewall Client communication port. Every single new connection my machine was making to the ISA server was being denied. I began to pore over my ISA settings looking for something that could explain this. I soon found it.

The solution

ISA ships with a pre-defined per-machine connection limit of 40. Once a machine makes 40 simultaneous connections to the ISA server, any new connections are denied. Because the HTTP and SSL connections were being left open, jumping around various pages on this site quickly brought the number of connections to 40. Raising the per-machine connection limit to 160, 4x the default, as well as reducing the timeout to 60 seconds has solved these problems. I’m not quite certain why connections to this site are being left open after all the data for the connection has been transferred, nor have I determined if this problem is specific to this site. I’ll have to break out my TCP/IP book and do a little RFC reading to get that fully figured out. I’m just glad to have finally solved this problem.