No admin looks forward to a network outage, but add the national power grid and a blame-happy CIO to the mix, and you may find yourself undergoing an impromptu test. Not only are your technical skills under examination, but also your ability to keep your cool as accusatory eyes watch your progress in resolving the crisis.
I was thrown into this very situation over a decade ago when I worked for a network provider and was dispatched to fix a problem when a customer called about a large-scale disruption. The problem occurred at the head office of the national power grid operator in my country. The CIO there had a long-established track record of rapidly attributing blame elsewhere whenever things went wrong, so getting a call from him immediately put us on guard. I was not looking forward to what interactions I'd have with him while fixing the mess.
[ Ditch the slackers, take on dirty work, do it with data: 12 effective habits of indispensable IT pros. | Follow InfoWorld's Off the Record on Twitter for tech's war stories, career takes, and off-the-wall news. | Subscribe to the InfoWorld Off the Record newsletter for your weekly dose of workplace shenanigans. ]
The CIO met me when I arrived and explained the situation. He was very tense and said his IT staff had determined that some component of the network -- that is, my company's responbility -- must be causing the problems. He didn't blame us outright, but his tone was accusatory. I mentally geared myself up to deal with what was very probably going to be a long and stressful day.
The mood at the site was panicked to say the least. The office had about 500 employees, and by the time I arrived about half of them had been unable to work for almost two hours. The IT staff were investigating to see if other sites or the national power grid were affected. The CIO was under extreme pressure from other managers to resolve the problem -- yesterday.
The corporate network used two ranges of IP addresses with a large core router providing the link between the two halves of the network. About half of the on-site PCs were intermittently losing access to servers in the other subnet.
The internal IT support personnel had been working on the issue but had hit a roadblock. They'd come to the conclusion that a major fault had arisen in the core router. They had started an investigation but found that the network breaks were so severe they couldn't log on to most of the network equipment to begin troubleshooting.
The CIO was frantic and hovered over my shoulder through every troubleshooting step, demanding updates every minute. I eventually asked him to wait outside the room so that I could get on with the job. To his credit, he complied.