The customer called to indicate one of his user’s Inbox had been moved to his Deleted Items folder. I had never before seen this, as Outlook prevents users from making such “mistakes” with special folders. I’d heard rumors that previous versions of OWA would allow users to do this, however he insisted that it just appeared this way after the migration. What was really strange was that new email continued to be delivered to this seemingly “deleted” inbox. No attempt to move the folder back to the root of his mailbox would work, and Google turned up little helpful information this time. Deleting the Outlook profile and all of its offline cache goodies proved futile, and I was on the verge exporting his mailbox to a PST file and nuking it, when The Google finally answered.

Turns out some random bloke on a Technet message board had this problem when moving users from Exchange 2003 to 2007, and after some of the same steps I had taken, he had found the solution that worked equally well for me. Moving the user back to the old server, then back AGAIN to the new server put the Inbox back where it belonged. While I don’t understand the root cause of this problem, I’m glad to have solved it without the pains of nuking a mailbox. Just further proves that keeping an old Exchange server around for a few weeks after its migration is a Smart Move™.

A quick Google search turned up numerous hits about a VMware specific RPM available from APC for v2.21. A quick search of APC’s website turned up no such thing, and downloading the newest release for Linux didn’t get me very far. During the installation it through an error about VMware not being supported. After some further Google digging, I finally found a direct link to the RPM buried on APC’s FTP site. Installing the RPM worked like a champ, and once I opened up the requisite firewall ports in ESX I was able to access the web interface and get it configured.

To save others the same headaches I encountered, I’ve preserved the RPM file on my site until APC decides to support VMware in new releases again. The file is available below.

APC Network Shutdown v2.21 for VMware

Generally, the upgrade process was smooth, but not without some hiccups, plus I had to do a fair amount of work to get the much-sought-after Today screen working. Just to help others that may experience the same pain, here are the tips and gotcha’s I encountered:

• Before running the upgrade I made a full backup using Blackberry Desktop Manager, and restored it post-upgrade (although the OS installer does that for you). Regardless, I found myself re-doing a number of my settings, likely because of the newer themes. I also had to re-activate it against the BES at my office.
• The new OS is significantly slower in certain areas, particularly the Profile switcher and on the today screen, however I’ve gotten used to it.
• From some unknown reason, the new OS dumps the Notifier_BikeHorn sound that I loved. All attempts to reinstall it by hand from files found on BlackberryForums and PinStack failed, leaving me somewhat sad. Now I’m using the Notifier_Eager sound, which reminds me of the DC Metro’s “Please stand clear of the doors!” bell. Kiersten seems to think about Bewitched every time she hears it, but I think that’s daft.
• No matter what I tried, I couldn’t get any of the “Today” style themes from around the ‘net to work, despite that fact that 4.2 introduces this feature. After some Googleing, multiple forum posts led me to this post at BlackberryForums.com. In short, download this theme and JavaLoader.exe. Unzip the theme, and move it along with JavaLoader.exe to your Blackberry Desktop installation directory, and execute the following command while your Blackberry is docked.

  JavaLoader.exe -u load net_rim_theme_bb_dimension_today_320x240.cod
  
  Now, every Today-style theme you load will work. I rather prefer the Dimension Today theme and use it, but there are others you may want to play with.

• I had to update a fair number of installed applications in order for them to use the phone’s data connection, in particular Google Maps and GMail. As it turns out, I was woefully out of date on both of those programs, so in addition to fixing the data problems, they also brought some new features and improved speeds with them.

As I’ve said, the new OS slowed down the previously nimble phone to more pedestrian speeds, however I feel the trade-off with added features and broader application compatibility was worth it. If anything, it will keep me tided over until I finally decide to upgrade to the 8830!

Despite this great bounty for only $120, they do have a major weakness – they tend to lock up after about 2 weeks of normal use, which requires a hard power-cycle to resolve. After some Googling, I recently stumbled across a work-around on Netgear’s forums. It seems by setting an SNMP OID to a certain value, you can cause the access point to do a soft reboot. The trick is to schedule such an event on a weekly, or even daily, basis, so that it occurs before the AP has a chance to lock up. The command below works quite well using the Windows task scheduler and the Net-SNMP tool set.

snmpset.exe -v 1 -c private 10.10.10.10 1.3.6.1.4.1.4526.4.3.9.1 integer 1

Just change the community string (in this instance, private) to your R/W community, and of course the IP address to match your AP. I have this running at two locations each rebooting 5 of these AP’s on a weekly basis and so far no lockups.

All the clients were running Outlook 2003. Some users were seeing duplicates of many of their system-level folders (Inbox, Calendar, etc). All users were unable to access any folder but their Inbox. Trying to view the calendar, contacts, or even a user-created mail folder would cause Outlook to crash. I suspected it had something to do with offline folder files, although deleting the Outlook profile and it’s associated OST files had no affect. A bit of Googling finally turned up this post at joeware, which pointed to this post in the Microsoft newsgroups, which contained the answer.

After some work, we were able to determine why Outlook 2003 crashes after moving mailboxes off of Exchange 2000 onto Exchange 2003. The fix is to add a registry value “Guid-Replid Caching” under HKLM\System\CurrentControlSet\Services\MSExchangeIS\SERVERNAME. Under each mailbox store we added a REG_DWORD of “Guid-Replid Caching” with a value of 0.

Taking their advice, I made the change, restarted the Exchange IS service, and damn if that didn’t solve the problem.

I decided to make the jump from Comcast after debating prices vs. features and picture quality, finally settling on a FIOS package relatively equivalent in both price and features to my current Comcast package. I placed a call to Verizon, and ordered the 5/2 Internet service, the premier package television service, the “movies” add-on, 1 DVR box, and 2 standard boxes. We have three TV’s in our house, and each television requires a box with Verizon. With my order complete, and installation date scheduled (about 2 weeks out), the wait began.

Installation

Two days before my installation, I received an automated call from Verizon confirming my installation date and reminding me that someone over the age of 18 must be present. The day before my installation I received a second call, this time from an actual person who basically regurgitated what the automated message said.

The day of the installation, I received a call from the installer informing me he was on his way to start the Internet install, and that the television installer would be arriving shortly afterwards. Both installers arrived at the same time, within about 30 minutes, and introduced themselves and explained what they were going to be doing. I showed them my existing wiring (home-run RG6 and CAT 5e), my network closet, and the location of the three televisions. We consulted on where the cables from the optical network terminal (ONT) should enter the house and where the battery backup unit (BBU) should be installed. I was one of their “new-style” installs that used a single coax cable from the ONT for both television and Internet. This allowed them to simply “jack in” to my existing coax and kept their wiring job to a minimum. They split a television run that went right by the BBU and used my existing wiring as is. Very easy.

They replaced my cable modem with an Actiontech router, and plugged the coax cable right into the back. The router provides Internet access for the computers, upstream access for the cable boxes (for on-demand, the guide, and the like), and an internal network for the cable boxes to communicate on. This later option is used mainly by the home media DVR feature, something I may check out in the coming months.

In all, the installers were on-site for around 3 hours, including “training” me on the use of the cable television and configuring my router. The entire time both techs were friendly, happy to answer questions, and very professional. I couldn’t be more pleased. I thanked them for their time, they thanked me for making their work easier (pre-wiring is a big deal), and they were off. While I don’t expect every installation to be as smooth as mine, I have no doubts that Verizon goes much further for their customers than Comcast.

Internet

I decided to tackle the Internet connection first, as I needed to integrate it into my existing network configuration to get back online. In the past I used a Motorola Surfboard cable modem to connect my Comcast cable Internet to a Linux router/server. With this new install method of a single coax, however, I was forced to use their router. I didn’t feel like trying to reconfigure the router as a bride, so I simply changed the IP range of the router to a different class C, and set my Linux server as a DMZ host. This essentially gives me a double-NAT’ed Internet connection, but everything works just fine. I was also pleased to see that I was a DHCP customer, rather than PPPoE, which makes for a much easier configuration to deal with.

Speed levels so far have been consistent, with very low latency and no packet loss. The 2Mb upload speed (in reality, about 1800Kb) is fantastic, and makes for slightly snappier VPN or remote desktop sessions over the ‘net. Verizon blocks in-bound port 80, but not 443, so my SSL enabled web mail server runs just fine. No other ports seem to be blocked in either direction, including outbound SMTP. I’m able to relay mail through my web-host’s mail servers straight over port 25, something Verizon DSL customers have been prevented from doing in the past. All in all, I’m very happy. I’m considering bumping up to the 15/2 plan, although I may wait until they upgrade it to 20/5.

Television

While the Internet connection is very nice, it’s nothing special. I never had problems with my Comcast connection, so I didn’t really gain or loose anything with FIOS Internet. Television, on the other hand, seems like the FIOS killer app.

Picture quality is excellent. High-definition channels look just as good as Comcast, which was always excellent. Standard definition (SD) channels are hands down the best I’ve ever seen. Nothing brings out a crappy SD picture like an HDTV, and on my set SD Comcast looked particularly crappy, with compression artifacts all over the place. Thanks to the bandwidth and on-demand architecture of FIOS TV, there is no
noticeable compression on any of the SD channels. The picture and sound are crystal clear, and playback is smooth and instantaneous.

The sheer number of channels is overwhelming. With their “Premier Package” (in reality their most basic standard plan), I have over 200 english-language channels, plus the HD channels, plus 40+ digital music channels, for less than $35 a month. Wading through all of these channels was a little frustrating at first, but it’s gotten easier over the last week as we get used to where our favorite channels are.
The DVR box is the same basic crappy Motorola set Comcast uses, with slightly different software. While better than the Comcast solution, it’s no TiVo. Still, there are little things about the FIOS DVR software that are a lot nicer than the Comcast DVR software. Recorded shows are grouped together, the cursor jumps to the next logical selection, and the box jumps back 8 seconds or so when you finish fast-forwarding, obviously trying the imitate TiVo’s jumback feature but failing miserably. For downsides, the Verizon guide is a little more jumbled than Comcast’s, their provided remote does not have a page down button, and there series-recordings are as flexible. However, until the Series 3 TiVo drops below $200, I’ll stick with the Verizon DVR.

Conclusion

All in all, I’m very pleased with the Verizon FIOS service and highly recommend it. Comcast can probably make you better deals right now, as their sales reps have more freedom with package pricing, but Verizon will catch up to this. Also, if you have a lot of televisions, Comcast may be cheaper since Verizon requires a box for every TV. However, if you care more about picture quality and features than about saving $10 a month on your bill, this service is definitely for you.

This whole project changed when I happened to check-up on pfSense, a firewall distribution based on FreeBSD. Lo and behold, they had added multiple WAN support over the summer. A quick download and test run later, and I had my winner. It had the raw support for the features that I need, with the polish coming down the pike in the coming months. It was free, since I already had a spare server to put it on. It was configured completely through a web interface, making for easy administration. It was… a done deal.

We went live with the setup before Christmas, and it’s been running flawlessly. Policy-based routing allows me to control which packets go where, and strong NAT/firewall rules make it a breeze to publish services out to the world. I’ve even got it running a fourth interface for a guest VLAN. More on that later…

With the easy part out of the way, I embarked on a quest to manage two WAN links. Our firewall/gateway is a Microsoft ISA Server, which doesn’t support multiple WAN links. The only ISA add-on that does support multiple WAN links has just been deemed end of life by EMC. Just as well, as it was $3000. So, I began looking for hardware solutions. Thus began the hard part.

I found many SOHO solutions that can load balance multiple WAN connections. The list includes the NetGear FVX538, Xincom DPG602, Linksys RV082, and a few others. All of these are low-cost, feature rich solutions, but are obviously built on low-cost hardware. They don’t seem capable of handling the throughput of a large network, as evidenced by their 253 user limit. I’m not even certain what this user limit is, or how it’s enforced. In any event, these seem best suited to small offices of 50 PCs or less. Certainly not ideal for our 300+ node environment.

My next foray led me to the Cisco 1841 router. There are a few users on the ISA Server message boards that seem to be raving about this router. It has many features and options, but my initial impression is that it cannot handle the routing speeds of a broadband cable connection. My CDWG rep is looking into this, so I should have some more information shortly. If it can meet my needs without breaking the bank, I think it’s a sure bet. Enterprise class features and support are always a plus.

Finally, an intriguing option would be to setup a Linux-based router using this article as a guide. This is option has the appeal of low cost, many possibilities with traffic control and QOS, and of course all the smell of a fun project. However, support and service are realistically nil, so the long term outlook isn’t good.

Hopefully I’ll have some more options and answers in the coming days, and I’ll post an update then.

The background

We run a Microsoft ISA Server 2004 firewall to provide Internet access and security for all computers on campus. ISA Server is known for its strong per-user authentication features, and we make heavy use of them. All outbound connections from our workstations must be authenticated, which is handled by the Firewall Client that is installed through group policy. We also use the web caching feature of ISA, so all of our browsers are configured to use the ISA Server as a proxy through the auto-configuration script.

The symptoms

Initial site access was fine, and load times were quick. Subsequent page loads from the same site would begin to slow down, and some page elements (read: images) would fail to load. Once that happened, future attempts to access the site would fail with timeout errors.

The investigation

When testing this problem, I noticed the problem was much more prevalent in Internet Explorer, so I switched this particular user to Firefox for this one site. Things seemed to work better, but reports of page failures still would come in. Other attempts at resolving the problem, including removing proxy settings from Firefox and disabling caching for the site, provided initial but ultimately false hope. Finally, as a desperation attempt, I added a rule to our ISA server to allow any request to the site without authentication or HTTP content checks. I even bumped this rule to the top of the list, to reduce the amount of time the firewall was involved in the transaction. In short, these blind fix attempts proved fruitless.

It was time to take it up a notch, so I busted out network monitor. Traffic between the ISA server and the site appeared to be normal, with no unusual packets or headers to throw things off. Of course, much of the traffic was encrypted with SSL, so network monitor was next to useless. I then began to monitor the log files from the ISA console while troubleshooting the issue, and initially everything seemed to be normal. Then I saw something.

The realization

I noticed that the connections to the site were staying open, even after all the data for that connection was transferred. After ISA’s connection timeout limit was reached, the connections would then close. Strange. I also began to notice that once I got locked out of this site, I was locked out of any type of Internet access for about two minutes. Two minutes, the exact amount of ISA’s timeout limit. Then the clincher, connection attempts to port 8080 and 1745 on the firewall were being denied from my machine. For those of you not familiar with ISA Server, port 8080 is used for the proxy/cache portion of ISA, and port 1745 is the Microsoft Firewall Client communication port. Every single new connection my machine was making to the ISA server was being denied. I began to pore over my ISA settings looking for something that could explain this. I soon found it.

The solution

ISA ships with a pre-defined per-machine connection limit of 40. Once a machine makes 40 simultaneous connections to the ISA server, any new connections are denied. Because the HTTP and SSL connections were being left open, jumping around various pages on this site quickly brought the number of connections to 40. Raising the per-machine connection limit to 160, 4x the default, as well as reducing the timeout to 60 seconds has solved these problems. I’m not quite certain why connections to this site are being left open after all the data for the connection has been transferred, nor have I determined if this problem is specific to this site. I’ll have to break out my TCP/IP book and do a little RFC reading to get that fully figured out. I’m just glad to have finally solved this problem.

In an effort to further school myself in this program, I stumbled upon an excellent guide called Efficient Editing With vim over at Jonathan McPherson’s home page. This is an excellent guide to for someone familiar with vim’s basic editing tasks, who is looking to increase their efficient in moving around a file.