Emergency Preparedness for IT Departments

http://gigaom.com/2013/10/17/what-superstorm-sandy-taught-us-about-protecting-it-infrastructure/

Excellent article for when your office looks like this:

Good article for the folks working to make the plans etc. I've found that the IT folks generally are more aware of the need to deal with planning and preparing (read spending money on) these types of rare but catastrophic circumstances.

In my experience, the first problem (and in some ways the hardest) is to get the "decision makers" and "budget people" to be willing to spend present day dollars on something that probably will not happen on their watch.

If you take out the few weather-specific references, this article sounds like it could've been written right after 9/11. I don't know, maybe these "IT" people or companies in this apparently Manhattan-centric article weren't even around then.

I remember many of the big banks and financial companies scrambled to set up pretty elaborate alternate offices outside Manhattan, mostly in New Jersey, after 9/11. I wonder how long those lasted until some bean counter decided it was an unjustifiable expense? Must cost a pretty penny to keep a ghost office mothballed, just in case.

And one big change since 9/11 is the explosion in cloud computing. Of course, the term "IT" encompasses lots of different functions beyond what cloud computing provides, but the ability to instantly and seamlessly transfer your computing needs to various data centers around the world allows even small companies now to have a high level of reliability and redundancy that only the big boys could afford ten years ago. You don't even have to own your own data centers. Just pay for what you need from third party cloud computing companies like Amazon Web Services.

The human element will always be the most unpredictable and irreplacable element to plan for, I think, and I agree that making it #1 is probably a good call.

I'm glad that I don't see any rats swimming around in that flooded Verizon lobby...

Amazing how we have to keep relearning the same lessons. When I was a clinical perfusion student in Houston in 1985, the chief perfusionist of Texas Heart Institute told my class of his experience during Hurricane Carla in 1961. An emergency heart surgery was under way when Carla hit Houston. The power failed and the emergency generators smoothly kept the power running to the cardio-pulmonary bypass pump; unfortunately the generators were in the basement of Texas Children's Hospital and when the flood waters filled the basement, the generators died. The perfusionists (one staff, one student) finished the case turning the pumps by hand. When I was a student there, the generators were located on the roof.

I spent many years as an IT consultant, with many projects surrounding business continuity planning and disaster recovery.

Frequently, IT staff would ask me what to do if and when decision-makers refused to invest sufficiently in infrastructure or redundancy. My response was always the same: "Tell them, in writing, what it is you must do to provide the redundancy that they ask for. Tell them what you must do to provide the PROPER redundancy. When you have an outage, point to the document you wrote." This resulted in a number of consulting engagements where I would be paid to review or write such documents.

Folks it seems have become a little too dependent on Information Technology and don't know or cannot prepare themselves to go just back even 20 years ago before all this Interweb and communications dependency came into existence. I didn't even have a telephone in the house until the early 1990s and I had only 4 TV channels to watch in the 1980s.

http://www.youtube.com/watch?v=iJOdusfDCb0

Data backup and security is really just common sense such as not putting Nuclear Reactors on active fault lines subject to Tsunamis, the real problem of course is having a 70 year power infrastructure with little resilience to storm perturbations whether it is Solar or even EMP, Wind or ice. There is not much that can be done about that for SMEs and even multinationals to keep data comms open and working even if your IT database centre is just peachy.

Try getting a Bank Account with a Bank Statement Book or ask for your Employer to give wages in cash every week. They were able to do this 20 years ago. The Post Office was even more reliable back then.

IT dependency really just adds to a lack of resilience due to the psychological panic it induces in the 'I have to have it done now' Society.

To be fair, Passbook savings accounts are still easily available in the U.S. They're mostly used for accounts directed towards youth, the elderly, migrant workers ("illegals"), Christmas clubs, and such. However, they're still offered to regular adult customers who simply want them.

I know down south a lot of church groups still use them to manage church funds, which means in some communities, even today, they can still actually make up 10%-20% of a banks total deposits.

Famous aphorisms of adam2

"do not put servers or other crucial IT equipment in basements. Basements fill with water, no matter what assurances may be given

Do not put servers or other crucial It equipment on the top floor, roofs leak no matter what assurances may be given

Do not put servers etc below toilets, washrooms, kitchens, showers, or plant rooms, as these facilities leak, no matter what assurances may be given.

Do not put servers or other crucial equipment in rooms with external windows. These will be broken by thieves, vandals, rioters, extreme weather, or terrorist bombs.

Test standby generators on FULL LOAD at least monthly

Replace the batteries in UPS systems at least as frequently as recomended"

Any datacenter with wet pipes in it will be compromised by leaks. If the fire code requires sprinklers rather than FM-200, see if your fire marshal will allow dry pipe systems. If that isn't possible, seek a better location for your datacenter.

The contract you have with your fuel supplier may be flammable but it won't keep your generators running. Your onsite fuel storage must be adequate for your business requirements.

If your backup power systems are inadequate to handle your cooling systems, your backup power systems will keep you up long enough for an orderly shutdown of your datacenter, if you move fast.

Do not build your backup datacenter in a convenient location. Build it instead in a location that's outside of any likely natural disaster that can encompass your primary datacenter.

My experience in the USA is different; mail delivery has become far more consistent and reliable than it was in the 90's.

I agree. While it is not perfect it has improved a lot over the last few decades.

One of the problems with IT people is that they are rarely well versed in the systems that keep their stuff running.

It is amazing to me how many servers are plugged into a UPS bought at Radio Shack that is in turn plugged into the wall outlet.

generators that run on diesel need to have the diesel fuel tested and/or filtered periodically. it may even need to be replaced occasionally.

some data centers don't routinely test the automatic transfer switch that actually switches the power from the utility to their backup generator.

I read once of a case where the data center security system had a UPS backup but when the UPS battery died during a power failure, people were no longer able to get in because the card reader stopped working.

The flaws with halon suppression systems caused a lot of data centers to go with sprinklers. These days dry sprinklers are pretty common and very reliable. they are very common in unheated spaces like warehouses.

BTW, the reason sprinklers leak is because they use the thinnest pipe they can find to save money. It is specially made and only used by the sprinkler industry.

the thing is if your sprinkler goes off it does not matter much if it is water or not. you have a huge problem that is only solved by having a redundant data center several hours away.

unless you are a really large IT center, it is often very cost effective to just rent capacity from a commercial data center, especially for your backup needs.

I don't think that FM-200 has the same issues with long-term damage to equipment that Halon had. Most of the datacenters I go into on a day to day basis use FM-200; seeing one with dry-pipe sprinklers is unusual.

Once upon a time I had a customer with wet-pipe sprinklers...and a baking tray underneath one that was leaking, with a little tube taking the water out of the baking tray out of the datacenter and into a drain in the bathroom. It was an impressive bit of engineering. I was told that the one server that the leak destroyed cost them $50k to replace, without regard to the downtime cost.

There's a reason my nom-de-Internet is what it is -- if anything can go wrong, it will go wrong in my presence. I've managed to avoid being present for an inadvertent fire suppressant discharge, however.