Does your company have a disaster recovery plan?

The last few weeks, and just this weekend, have seen the startup of a very unpleasant part of spring in the Midwest – dangerous tornadoes.   These terrible natural disasters can be devastating to life, property and personal belongings.  But they can also ruin an otherwise successful businesses.

Back in 2005, we wrote about disaster planning in this YP Talk article.  Many of the comments we made at that time are still true.  For example, in just the area of IT related disasters such as a major loss of business data, 43% of companies never reopen, 51% close within two years, and only 6% will survive long-term (source).

How much should you spend to develop and implement your plans?  Estimates vary but by example,  most large companies spend between 2% and 4% of their IT budget on disaster recovery planning, with the aim of avoiding larger losses in the event that the business cannot continue to function due to loss of IT infrastructure and data (source).

Disaster recovery is a broad topic covering more than just IT and data.  Considerations need to include buildings, people, customers, financial, etc.   Here is a partial list of events that could affect your company:

• Data security (virus, denial of service, unauthorized access)
• Telecom failure
• Power outage
• Data center hardware/software failure
• Structural fire (internal)
• Water pipe break
• Gas/chemical spill or leak
• Physical security (workplace violence, terrorism, etc.)
• Natural disaster (earthquake, tornado, hurricane, wildfire,      etc.)
• Labor disputes

Given the human tendency to look more on the bright side, many business executives are subject to ignoring “disaster recovery” because disasters seem like such an unlikely event, and will never happen to their business.  If you are in the camp, let me offer you one example of how naïve that thinking can be.

While I was working for a large publisher, our operations were based in the lower level of their secure data center, on its own security controlled property, with all of the latest innovations to prevent most disasters.  I’m sure we thought we were totally safe.  But then we arrived one morning to 5” of standing water in the lower level of our building from a blocked sewer connection just 50 yards outside the building property. Our phone systems were fried, anything on the floor (such as pc’s) was ruined, and we were displaced from the building for over a week while cleanup was done.  And our plan ….. didn’t exist.  It took months before things got back on track.

Here are some general steps we offered in our 2005 article that on developing a disaster recovery plan:

1) Determine the impact of being out of business for X amount of time.
In the middle of a major book campaigns, the answer to this question can be expressed in monetary terms: “We would lose $xxx,000 in sales/potential revenue in a day if our sales effort was stopped.” But you should also consider customers who won’t be served, print windows that may slip, customers already served that have now gone out of business, employees that are still hoping (expecting) to get paid, etc. etc. etc.

The purpose of this step is two fold. First, it provides you with a benchmark against which you can access the costs of varying levels of redundancy and backup (in some ways-not all-more protection means more money). Second, it will help position each part of the business within the context of the organization’s priorities (e.g., which function must get restored first).

2) Identify potential threats.
A disaster recovery plan, like an insurance policy, is most effective if all the risks and threats are realistically identified. While hurricanes and earthquakes do happen, most threats do not arrive in dramatic, news-making fashion. You will need to prepare for water damage (from broken pipes, backed up drains, failed condensation pumps, roof leaks, ground or flood water, discharging fire sprinklers or the fireman’s hose), fire and smoke damage, component and network failures, cable cuts, power losses from blackouts and brownouts, sabotage and lightning strikes. Given the integrated information world most mid to larger size company’s operate in, you will also need to identify how your systems will behave if a key component goes down-e.g., what happens to calls when/if a major telecom link fails at a remote site?

3) Take Preventative Measures.
As you identify potential threats and areas of vulnerability, preventative countermeasures will emerge. Hardware and networks are protected primarily through redundancy and diversity in equipment and services. Specific steps usually include subscribing to services from multiple carriers, deploying fire detection and suppression equipment, working with suppliers to identify critical system components you should keep on site, equipping your system for power backup and ensuring you have good wiring and adequate power line protection against lightning strikes and voltage surges.

Regular record keeping and off-site backup is critical to prevention. Key information and database files should be regularly backed up and stored both onsite and offsite.

Let’s not forget about your most valuable resource – your employees. Home and mobile contact numbers for key people should be collected. Do you have a plan if you need to totally relocate the whole operation (yep, moving the whole shebang to an alternative site)?.

4) Develop an Escalation Plan.
An effective escalation plan outlines appropriate responses to each potential disaster and specifies the thresholds at which they should be deployed. It should address the following:

1. What constitutes a disaster?
2. Who in the organization declares  a disaster and puts the disaster recovery plan into motion? How can they be reached?
3. How will key people inside and  outside the organization be notified of a disaster, and what roles will they fill in the recovery effort?
4. What’s the appropriate escalation plan for the disaster, given its type and magnitude?

The plan should be simple to understand, easy to follow and up to date. For example — plans and vendor references in the disaster impacted area should contact the ABC-based Disaster Recovery Team at 123-456-7890 or at http://www.<ourhomepage.com>/disaster-plan.

5) Practice and Update the Plan.
Your carefully constructed plan will be of no value if it sits on the shelf during a disaster. Reviewing and practicing recovery plans may be reminiscent of school days, but these drills are worth a lot more than nostalgia. Many disasters happen quickly and without warning. People have to know what to do!   We just had our tornado shelter drill last week.  When have you scheduled yours??

These are just some general thoughts. We’d like to hear what your company is doing in this area. Drop us a note at ken@yptalk.com.