The background

We run a Microsoft ISA Server 2004 firewall to provide Internet access and security for all computers on campus. ISA Server is known for its strong per-user authentication features, and we make heavy use of them. All outbound connections from our workstations must be authenticated, which is handled by the Firewall Client that is installed through group policy. We also use the web caching feature of ISA, so all of our browsers are configured to use the ISA Server as a proxy through the auto-configuration script.

The symptoms

Initial site access was fine, and load times were quick. Subsequent page loads from the same site would begin to slow down, and some page elements (read: images) would fail to load. Once that happened, future attempts to access the site would fail with timeout errors.

The investigation

When testing this problem, I noticed the problem was much more prevalent in Internet Explorer, so I switched this particular user to Firefox for this one site. Things seemed to work better, but reports of page failures still would come in. Other attempts at resolving the problem, including removing proxy settings from Firefox and disabling caching for the site, provided initial but ultimately false hope. Finally, as a desperation attempt, I added a rule to our ISA server to allow any request to the site without authentication or HTTP content checks. I even bumped this rule to the top of the list, to reduce the amount of time the firewall was involved in the transaction. In short, these blind fix attempts proved fruitless.

It was time to take it up a notch, so I busted out network monitor. Traffic between the ISA server and the site appeared to be normal, with no unusual packets or headers to throw things off. Of course, much of the traffic was encrypted with SSL, so network monitor was next to useless. I then began to monitor the log files from the ISA console while troubleshooting the issue, and initially everything seemed to be normal. Then I saw something.

The realization

I noticed that the connections to the site were staying open, even after all the data for that connection was transferred. After ISA’s connection timeout limit was reached, the connections would then close. Strange. I also began to notice that once I got locked out of this site, I was locked out of any type of Internet access for about two minutes. Two minutes, the exact amount of ISA’s timeout limit. Then the clincher, connection attempts to port 8080 and 1745 on the firewall were being denied from my machine. For those of you not familiar with ISA Server, port 8080 is used for the proxy/cache portion of ISA, and port 1745 is the Microsoft Firewall Client communication port. Every single new connection my machine was making to the ISA server was being denied. I began to pore over my ISA settings looking for something that could explain this. I soon found it.

The solution

ISA ships with a pre-defined per-machine connection limit of 40. Once a machine makes 40 simultaneous connections to the ISA server, any new connections are denied. Because the HTTP and SSL connections were being left open, jumping around various pages on this site quickly brought the number of connections to 40. Raising the per-machine connection limit to 160, 4x the default, as well as reducing the timeout to 60 seconds has solved these problems. I’m not quite certain why connections to this site are being left open after all the data for the connection has been transferred, nor have I determined if this problem is specific to this site. I’ll have to break out my TCP/IP book and do a little RFC reading to get that fully figured out. I’m just glad to have finally solved this problem.

In an effort to further school myself in this program, I stumbled upon an excellent guide called Efficient Editing With vim over at Jonathan McPherson’s home page. This is an excellent guide to for someone familiar with vim’s basic editing tasks, who is looking to increase their efficient in moving around a file.