Now what do we do?? Lessons learned from Sandy

The images on the news following Hurricane Sandy of the suffering and living struggles people are going through in the New York/New Jersey area have been horrific to watch.  It brings back many memories of the aftermath of Katrina, but on a different scale.  I still have family in the area and many were without electric and heat for 9+ days.

What you will not hear as much about is the impact on businesses from this serious, destructive act of nature.  It has been estimated that more than 3 million businesses – large and small, schools, government agencies, and hospitals – are still affected to a point of being partially or completely shut down.  Even YP Talk was not immune as the serves that host our system are based in the northern New Jersey area and lost power for a couple of days.

Once again, it reminds all of us of the need for disaster recovery plans.  It’s easy to talk disaster recovery;  it’s a whole other thing to actually implement it.  And we’re not only talking about hurricanes, flooding, or blizzards:  recall the earthquakes of the early 1990s in California, the recent fires this summer in Colorado, record size tornados in the southeast (not the mid-west where it is expected), and the nuclear reactor issues in Japan. Sandy’s total economic damage from this super storm could be as high as $50 billion.

Back in September, 2005, post Katrina, we covered the need for every business to have a disaster recovery plan in place (link to full article).  In that article we suggested 5 general steps for implementing a plan.   What’s different now, some seven years after Katrina and that original YP Talk article (has it really been that long?), is that technology and communications are even more critical in today’s businesses and business operations. Sandy brought down a lot of the core infrastructure:  power, Internet, and even cellular communications were wiped out and in many areas, have still not been fully restored.

While we are not suggesting that we are disaster recovery experts at YP Talk, but the two key questions you should consider in your organization is:

• How long can our business remain down with no data, no computer access, no internet, or no telephone/communication operations before we will have to shut down completely?
• How much data can we afford to lose before our business will suffer irreparable damage going forward?

Obviously, the answers are different for every industry and business.  But here’s some food for thought:  according to the Insurance Information Institute, up to 40% of businesses fail after a disaster and only 43% of all businesses feel prepared to handle an emergency. Other statistics indicate that 61% of businesses that where without communication for seven days or more eventually shut down permanently within a year.   Further, a loss of more than 30 days of data is proven to be catastrophic to 78% of businesses.

As an extension to our earlier article list, here are a couple of things to consider for your disaster recovery plans:

Step back before going forward:  Have you determined your organizations vulnerabilities and capabilities?  Expect the unexpected.  And no plan is worth the paper it is printed on if you don’t test your plan often — practice does make perfect

Backup Power is Key:  Today, EVERYTHING requires power.  If you house your own systems, do you have redundancy plan that may even include being able to run on generators for up to three days without refueling?

 Location, location, location:  Is all of your key physical infrastructure in the basement or other susceptible location??  One harrowing story from an IT manager indicated that their building’s block-long basement was filled with water and it actually made it up about four feet into the lobby above.  Hence, no surprise that the damage to the building was enough that they couldn’t get into the building at all, and then didn’t have any power for five days.  But even then, all that infrastructure was lost.

Communications, Phone, and Email:  Maintaining communications with staff, with your customers is essential after a storm.  Many firms have employees in different parts of their operation that can work from anywhere as long as they have access to servers and technology. While cellular communication if often the default provider after a disaster, as Sandy showed us, it’s not totally infallible.  Have a Plan C.   Social or business networks such as Linkedin.com or Facebook can be extremely useful for group communications, news distribution to groups, and just staying in touch.

The cloud is available:  One thing that is different from the Katrina days is that a number of cloud-based services are now available to both backup data and help companies get operational again. Think about using a service that can provide needed back up and support during disaster events.

Risk Management and Insurance – Does your company carry business interruption insurance? Flood insurance is also good because most insurance policies do not cover “rising water” from floods. The government sells affordable flood insurance to many Americans, but you must buy a contract well before a storm is viable to have coverage in force. Companies such as: Met Life or Lincoln Financial as well as others provide this service.

As Sandy showed us, here in the twenty-first century, our businesses rely heavily on critical infrastructure – and it’s essential to have a workable business continuity plan and disaster recovery plan to keep your business going, because you never know what’s next…..

 

 

 

Does your company have a disaster recovery plan?

The last few weeks, and just this weekend, have seen the startup of a very unpleasant part of spring in the Midwest – dangerous tornadoes.   These terrible natural disasters can be devastating to life, property and personal belongings.  But they can also ruin an otherwise successful businesses.

Back in 2005, we wrote about disaster planning in this YP Talk article.  Many of the comments we made at that time are still true.  For example, in just the area of IT related disasters such as a major loss of business data, 43% of companies never reopen, 51% close within two years, and only 6% will survive long-term (source).

How much should you spend to develop and implement your plans?  Estimates vary but by example,  most large companies spend between 2% and 4% of their IT budget on disaster recovery planning, with the aim of avoiding larger losses in the event that the business cannot continue to function due to loss of IT infrastructure and data (source).

Disaster recovery is a broad topic covering more than just IT and data.  Considerations need to include buildings, people, customers, financial, etc.   Here is a partial list of events that could affect your company:

• Data security (virus, denial of service, unauthorized access)
• Telecom failure
• Power outage
• Data center hardware/software failure
• Structural fire (internal)
• Water pipe break
• Gas/chemical spill or leak
• Physical security (workplace violence, terrorism, etc.)
• Natural disaster (earthquake, tornado, hurricane, wildfire,      etc.)
• Labor disputes

Given the human tendency to look more on the bright side, many business executives are subject to ignoring “disaster recovery” because disasters seem like such an unlikely event, and will never happen to their business.  If you are in the camp, let me offer you one example of how naïve that thinking can be.

While I was working for a large publisher, our operations were based in the lower level of their secure data center, on its own security controlled property, with all of the latest innovations to prevent most disasters.  I’m sure we thought we were totally safe.  But then we arrived one morning to 5” of standing water in the lower level of our building from a blocked sewer connection just 50 yards outside the building property. Our phone systems were fried, anything on the floor (such as pc’s) was ruined, and we were displaced from the building for over a week while cleanup was done.  And our plan ….. didn’t exist.  It took months before things got back on track.

Here are some general steps we offered in our 2005 article that on developing a disaster recovery plan:

1) Determine the impact of being out of business for X amount of time.
In the middle of a major book campaigns, the answer to this question can be expressed in monetary terms: “We would lose $xxx,000 in sales/potential revenue in a day if our sales effort was stopped.” But you should also consider customers who won’t be served, print windows that may slip, customers already served that have now gone out of business, employees that are still hoping (expecting) to get paid, etc. etc. etc.

The purpose of this step is two fold. First, it provides you with a benchmark against which you can access the costs of varying levels of redundancy and backup (in some ways-not all-more protection means more money). Second, it will help position each part of the business within the context of the organization’s priorities (e.g., which function must get restored first).

2) Identify potential threats.
A disaster recovery plan, like an insurance policy, is most effective if all the risks and threats are realistically identified. While hurricanes and earthquakes do happen, most threats do not arrive in dramatic, news-making fashion. You will need to prepare for water damage (from broken pipes, backed up drains, failed condensation pumps, roof leaks, ground or flood water, discharging fire sprinklers or the fireman’s hose), fire and smoke damage, component and network failures, cable cuts, power losses from blackouts and brownouts, sabotage and lightning strikes. Given the integrated information world most mid to larger size company’s operate in, you will also need to identify how your systems will behave if a key component goes down-e.g., what happens to calls when/if a major telecom link fails at a remote site?

3) Take Preventative Measures.
As you identify potential threats and areas of vulnerability, preventative countermeasures will emerge. Hardware and networks are protected primarily through redundancy and diversity in equipment and services. Specific steps usually include subscribing to services from multiple carriers, deploying fire detection and suppression equipment, working with suppliers to identify critical system components you should keep on site, equipping your system for power backup and ensuring you have good wiring and adequate power line protection against lightning strikes and voltage surges.

Regular record keeping and off-site backup is critical to prevention. Key information and database files should be regularly backed up and stored both onsite and offsite.

Let’s not forget about your most valuable resource – your employees. Home and mobile contact numbers for key people should be collected. Do you have a plan if you need to totally relocate the whole operation (yep, moving the whole shebang to an alternative site)?.

4) Develop an Escalation Plan.
An effective escalation plan outlines appropriate responses to each potential disaster and specifies the thresholds at which they should be deployed. It should address the following:

1. What constitutes a disaster?
2. Who in the organization declares  a disaster and puts the disaster recovery plan into motion? How can they be reached?
3. How will key people inside and  outside the organization be notified of a disaster, and what roles will they fill in the recovery effort?
4. What’s the appropriate escalation plan for the disaster, given its type and magnitude?

The plan should be simple to understand, easy to follow and up to date. For example — plans and vendor references in the disaster impacted area should contact the ABC-based Disaster Recovery Team at 123-456-7890 or at http://www.<ourhomepage.com>/disaster-plan.

5) Practice and Update the Plan.
Your carefully constructed plan will be of no value if it sits on the shelf during a disaster. Reviewing and practicing recovery plans may be reminiscent of school days, but these drills are worth a lot more than nostalgia. Many disasters happen quickly and without warning. People have to know what to do!   We just had our tornado shelter drill last week.  When have you scheduled yours??

These are just some general thoughts. We’d like to hear what your company is doing in this area. Drop us a note at ken@yptalk.com.