Disaster Recovery Facilities in the making?

As promised, more on the subject of disaster recovery planning, with a twist.

A recent article in my local news paper, The Morning Call, got me thinking about the whole issue of disaster recovery facilities as an economic development project. The article chronicles the trials and tribulations of a group attempting to create an environment attractive enough to convince disaster planners from Wall Street's largest investment firms to locate their disaster facilities to the Pocono Mountain region. The group has amassed $15 Million in funding to be used to convince the firms that it would be a good idea to house such facilities outside of a "blast zone" that is Manhattan, and one that is on a separate power grid zone. I give a lot of credit to the local power authority for isolating my region from the last large power outage that affected most of the North East US. This alone does not make the Pocono mountain area an ideal place to put disaster backup facilities.

There are only two main routes that run to the west out of Manhattan, Interstate 80 which runs to the Pocono's and Interstate 78, which runs through New Jersey and eastern Pennsylvania. Both routes would likely be affected by any wide scale disaster in the area, making it harder to get people to these newly established facilities. Both regions discussed in the article, the Pocono's and the Lehigh Valley are about 2 hours away from Manhattan on a good day. As a result of this and other factors, most Wall Street firms have located their backup facilities in northern New Jersey. The bandwidth required to support synchronous data replication is already in place between Manhattan and say Jersey City. The same can not be said for the Pocono's or Lehigh Valley areas. Most disaster recovery plans call for generating your own power for a good amount of time, so being on a separate power grid just relieves you from getting fuel to your generators over a long term.

So what is the point? I am somewhat embarrassed to say that the leaders of the region in which I live would spend $15 Million on training and education and not have a single tangible thing to show for the investment. No enhancements to network bandwidth, facilities, or agreement with service firms to provide the essential support functions that would make the region a viable option for disaster recovery services. If they really wanted to gain the respect needed to be in the running for consideration as a viable disaster recovery location, then at least get the bandwidth in place. Until large firms sign up to use it, at least the local citizens could enjoy having great Internet service.

The opposite of "if you build it, they will come" seems to be "if you promise it, they might consider it" and in this case not one firm has taken the bait. This makes for a pretty expensive fishing trip.

Disaster recovery planning-The need

Having been around the network awhile I have to admit that I have been a party to some disaster recovery efforts. Thankfully most of them have been pretty minor. Some have been averted due to some prior planning and some have been recoverable with some ingenious use of technology, such as it was at the time. It was also not like we could not see it coming.

In some of my past positions I have done some strange things to get the information we needed in order to plan for a disaster scenario. The good news was we had forward thinking management at the time who saw the need to plan ahead. The most odd thing I ever did was buy the power company lineman coffee if he would allow me to see his site planning books with enough time to copy the relevant pages. It took a while but we got what we needed not only to prove we were single threaded at our major data center but that there was an alternate feed available for us to split our power feeds with little effort on the part of the utility. The comment on the part of the field engineering manager was "Boy you guys sure know a lot about our physical plant"! That "knowledge" cost me a cheap cup of coffee and came in handy on the day that a 150KV line went up in smoke without explanation, and no outage.

The other scenario was the recognition that we were single threaded in our main telecom feeds going along a well travelled road up north to the first "PoP" to a national network. I do not take credit for the discovery, but when the ultimate disaster scenario came true one day we were all standing in the data center looking at each other wondering what we could do about it. We were told that we were hours or longer away from having service restored to one of our primary service systems. Insult to injury was our "dial backup" service followed the same path as it turned out. I will take credit for using my calling card from an alternate provider to establish 14 alternate paths out of our data center and up to a backup site a thousand miles away. It turns out we were down on our primary service for over 13 hours.

If the message has not gotten through already, everyone and every organization needs to think about what they would do in the event of a disaster of some kind. More on this topic to follow based on current events.

Should Administrators be licensed?

We live in a world where there are licenses and certifications required to do just about anything. I carry around 5 or 6 every day, but none of them for IT.

While we have professional certifications in the IT business we have few requirements for having and using Admin rights on servers and end user workstations. With data leakage becoming more of an issue every day it would seem appropriate that Admin rights come with a requirement to be licensed to use them. While nine out of every 10 people you ask will tell you that data leakage is centered around non-malicious activities, it is the one in 10, the malicious one, that is the most damaging and costly. It is usually pulled off with help from inside the enterprise and it is usually someone that has admin access rights. Doesn't it make sense that we know who we are giving these rights to, and then require them to be bonded to do their job?

I am not into burdening people unnecessarily but we have gotten to the point in IT where it would make some sense to stand up and say to everyone who entrusts personal data to us "You can trust me to do the right thing." Don't want to be required to have your background checked and live up to a professional code of ethics? Give up your admin rights then.

Enterprises are being required to spend millions of dollars on software and appliance based security tools to prevent data leakage and other malicious activities. Those costs are being passed on to all of us as consumers with little being provided in return. Perhaps we can offer some additional value to the equation.

Social Networking

Some of the latest news from the security world about enterprises blocking social networking sites strikes me as disturbing. I have found that some of the social networking sites provide a valuable way for people to stay in touch with former colleges, perhaps for fun, and perhaps to use these resources to solve current problems. To think that enterprises are now blocking these sites with web filters boggles my mind.

The justifications for blocking access run the gamott from resource utilization to lost productivity to security concerns. For those sites with known security threats I can see why blocking them would be useful in protecting the enterprise network. As for productivity enhancements, there are any number of distractions that can rob staff productivity during the work day. I used to think that employees going to the "smoking lounge" was a big productivity loss. I suspect it still is but I have given up on that point. Networking resources is another issue alltogether. If streaming video is a big issue coming from social networking sites then assign it the lowest quality of service and let users download it to their smartphones instead.

Social networking has been going on for centuries and will likely continue for many more to come. Some of the new internet based applications make any excuses to not keep up with people virtually disappear. Enterprises should embrace the capability to network among people in as efficient a manner as possible.

IT in health care- First hand experience

My family had the good fortune (and bad) to experience the services of one of the best health care providers in the country recently. Children's Hospital of Philadelphia (CHOP) has been one of the top hospitals for kids for many years and has made significant strides using technology solutions as an enabler.

This latest visit afforded me another opportunity to see how far they have come with their technology implementation. The customer care there has always been great but our visit this time impressed me even more. We saw paperless EEG testing, collaboration of results with a wide range of medical professionals, constant contact among staff members, and even wireless access for patients and visitors. Our next visit will allow us to see more advanced medical monitoring and communications, including portable monitors and recording equipment, and software that analyzes results and assists in the development of patient care plans.

Anyone that really wants to know how to use IT solutions to serve their customers better needs to find a model organization to point to as a good example of how to get it done. CHOP is one of many such leaders.