When good security kills performance

Continuing on my recent theme of security pain points, I’m finding that many companies suffer horrible log-on delays because of their computer security defenses. I’m not talking about a minor inconvenience. I’m documenting 8- to 10-minute boot-ups and log-ons versus 1.5 minutes without the host-based firewall or anti-virus software that’s getting in the way. It doesn’t matter which operating system the end-user is running. The problem affects both Windows and Linux/BSD machines — mostly during log-on, but often again when the user starts a browser session or an e-mail program.

I think we all rightfully expect a performance trade-off when running PC-based computer security defenses. But I’m seeing cases where popular security products are causing absolutely unacceptable delays. Even worse, administrators and users alike are accepting the poor performance as a necessary cost of doing business with computers.

[ Whose endpoint security products stay out of your way? See the InfoWorld Test Center’s endpoint security shoot-out. ]

They shouldn’t give in. It’s common and expected to trade some level of performance for greater security. But extremely bad performance shouldn’t be the cost of implementing a computer security product. So what can you do about it?

Performance checklist

First, get a baseline to measure against before starting to troubleshoot. Using a stopwatch or watch that increments by the second, measure the time it takes the computer to go from a cold boot to a fully usable desktop. “Usable” means the user can begin operating programs without significant delay. CPU utilization should be sustained under 5 percent. Further, I normally break the boot-up process into two phases: from cold boot until the user is prompted for their log-on credentials, and from the time the user successfully enters their log-on credentials until a usable desktop. If possible, configure the computer to automatically log on the user to prevent mistyped credentials from distorting your time measurements. Restart from a cold boot at least three times or until you have a fairly consistent boot-up time, differing by maybe 5 or 10 seconds.

If the boot-up time is horribly long, try temporarily disabling your host-based firewall or anti-virus product, then redo the boot timing tests. In some of the worst cases, I’ve found various popular firewall or anti-virus products to be responsible for the most significant parts of the delay. I’ve seen host-based firewalls slow network packet performance by a factor of 6 or even 10. Sometimes it’s just the first packet to any new host that is delayed, but this is enough to cause local resources to respond too slowly, prompting even slower remote or backup resources to serve the request.

Sometimes the security software can be tweaked to improve performance, in which case you may need the help of the vendor. In a few cases I’ve seen, a competing product, just as secure but a swifter performer, was chosen as a replacement.

If disabling the host-based firewall or anti-virus client doesn’t significantly improve performance, re-enable them and try something else. I’ve often resorted to disabling various, noncritical services or daemons one by one, and rebooting in between, until I’ve located the culprit.

If it isn’t a service or autostarting daemon, I move on to the user mode programs. After the user logs on, you can use many different programs to determine what is auto-loading in user mode. In Linux/BSD, there are many different locations where auto-starting programs can be launched. It’s best to do a little Internet searching on the subject if you are unsure about which folders and text files to interrogate.

In Windows, my go-to analysis software for finding autoloading programs is Autoruns, which will show you the dozens to hundreds of programs that automatically launch whenever you start Windows. It’s easy to work with, and it can be used to disable various suspect programs.

Try, try again

Once Autoruns lists your suspects, you’ll still need to do the painstaking troubleshooting work: Disable each suspect program one at a time, reboot, measure the performance, and re-enable the program if it isn’t the causative agent. Autoruns is a good way to identify malware, spyware, and adware, too. Process Explorer and Process Monitor are also great tools for analyzing running programs and the resources they consume on Windows computers.

Other common performance headaches include low physical memory (256MB is not enough to efficiently run most Linux/BSD versions, much less Windows XP or Vista), persistent drive mappings to nonexistent drive shares, inordinately paused log-on scripts, and corrupted configuration files. Just this week, I saw a popular scripting add-on program cause a 1.5-minute boot-up delay, when the exact same commands and if-then decisions could have been implemented in Group Policy Object in milliseconds. Sometimes all it takes is a little education.

I often use network traffic sniffers, such as Microsoft Network Monitor or Wireshark, in promiscuous mode to capture all the inbound and outbound traffic, looking for high error counts, small MTUs (Maximum Transmission Units), high retransmission rates, and long-delayed packets. Using packet sniffing a few weeks ago, I was able to determine that one client’s computers were randomly, but frequently, connecting to domain controllers located halfway around the world and not the domain controllers located 10 feet away.

On Windows computers, the Windows Performance Toolkit, with xperf.exe, is an excellent technical troubleshooting tool. It will measure every running program started or stopped during boot-up, shutdown, sleep, or suspend, and it will show you in graphical detail which program was taking up which computer resource and when. It’s an advanced troubleshooting utility and a goldmine of information.

Good security always incurs a performance penalty. The question is how much of a performance hit is acceptable? While every environment is different, log-ons exceeding 3 minutes are certainly outside the norm and probably excessive. If your users are used to even longer log-on experiences, I encourage you to take a day or two and isolate the causative agent. Often the blame can be laid at the feet of misbehaving computer security defense tools or an inordinate, even accidental combination of programs and scripts that no single owner or administrator expected to cause a problem, but resulted in “death by a thousand cuts.” A good security administrator appropriately weights security against performance and adjusts the strategy to the needs of the business and the environment.