Home » COOP » Planning » Unintended affects of failures
unintended 250

Unintended affects of failures

I have an Aussie friend who not long ago shared a tale about a station dog he came across when he was working repair calls for the phone company near Booligal Australia.  A farmer called in a repair using a neighbor’s phone saying that his phone had quit. So, into the queue he went for a repair visit. The station my friend was calling on was a fair distance out so he knew it was going to be an all day jaunt.border collie

As he made his way to the address he would stop and check the pair of copper assigned to the station. Each time he stopped he would ring the number and check the voltage on the line. In a few instances he had to remake the connections just to be sure there was a good strong ring voltage getting through.

Nearing mid-day he arrived at the station address and made his way through a couple gates up to the house. It was a fairly typical station with a large house surrounded by paddocks and sheep. He recounted that it was a nice quiet setting with laundry hanging, a slight breeze, and a border collie tethered with a chain to one of the paddock gates.

border collie 2My friend introduced himself to the farmer – a practical fellow with deep creases in his face and weathered hands. They chatted a bit, spoke about the sheep and the Booligal sheep races that were upcoming. All of a sudden the quiet lazy dog at the farmers feet erupted off the ground with a spine-tingling howl and started barking and running to the end of his tether frantically. This lasted for a minute or so and the dog suddenly stopped and lay back down quiet once again.

The farmer told my friend that he was sure the old dog was dying from a tumor of something. For the past few days the dog had been showing this erratic behavior. One minute he would be calm and the next minute he was a crazed animal. He was afraid to have him in the house so he kept him tethered outside most of the time. The farmer shared that the dog was worse today. It seemed every half hour or so the dog would go crazy and then lie down and sleep. A very sad situation.

My friend nodded and got to work. It seems the line was broken and needed to be reattached. This was done in short order and my friend tested the line with a call to the station address. phoneThe phone rang, but the voltage was low. There must be another problem in the line so he began to follow the line across the top of the fence. He wasn’t sure he wanted to get too close to the collie since it had begun barking again when he was testing the line, though not as loudly and frantic as before.

As he got closer to the gate he could see the glint of bare wire rubbing against the metal gate where the dog was tethered. Immediately, my friend started laughing and promptly went up to the dog gave him a good scratch. That old dog wasn’t dying, he was grounding the ring voltage from the wire rubbing against the paddock gate. He wasn’t getting worse today, he was reacting to the test calls my friend was making every half hour or so. Each time the phone rang that poor dog would provide the path to ground for the 100 volt ring voltage through his feet! Of course he was frantic, each time a paw hit the ground he was receiving a shock!

My friend patched the wire, gave the dog another scratch and made his way back to the shop. It wasn’t more than a few days later that he received a written phone message from the farmer thanking him for the repair and letting him know he didn’t have to put down the dog after all. The dog had recovered.


Un-intended consequences of telecommunication failures

Why share this story on a COOP blog? other than to have a good compassionate laugh. This story brings to light the problems associated with having communications service a distance away from the telecom provider, and one of the pitfalls of relying on analog service. The story also serves to demonstrate that there can be unintended events which can occur as a result of losing telecommunications service. Who would have thought that the dog would be bothered by a broken phone line?


Why is it a problem to have telecom service so far from a central office (CO)?

The distance factor impacts your cost of doing business in a few ways.

1. It costs you more in monthly fees to be located a long distance from the CO. All providers charge for mileage between the CO and your business address. That charge it added to the other service fees associated with the type of circuit you are using.

2. The probability of having a service related problem increases with the distance from the CO. With more wire exposed and more signal conditioners/repeaters between your business and the CO, the chances of having equipment failures and/or wire path disruptions is greatly increased.

3. All repairs that involve new wire being pulled/laid/strung require more human resources to complete the task. It may be that the telecom provider needs to free up more personnel, or it may be that wire or circuit boards have to be ordered. A service outage could also be as a result of poles being struck or blown down. Each of these failures could result in a prolonged outage depending on the preparedness of the telecom provider.

4. To implement multi-pathed homing / path redundancy in an effort to compensate for losing a telecom circuit may not be possible without considerable expense. If your business is in a rural setting, you may not have an alternate path possible. The telecom provider may simply say there is no path other than the path that is specified. Of course you can fund the creation of an alternate path back to a central office, but that may likely be prohibitive in terms of cost.

5. If you choose to implement a recovery site and you opt for real-time data replication to that site, be ready to at least triple your monthly telecommunications expense. Not only will you need to add new lines to the recovery site, but you will need to add primary telecom service from the recovery site to your service provider with telecom lines that are separate from your existing primary lines. Depending on the distance to your recovery site from the local CO, you may opt for a recovery site in a metropolitan market.

There are, of course, methods to circumnavigate telecom expense. These methods would include hosted services and wireless phones. while some telecom expenses could be avoided, new hosting fees would be introduced and should be evaluated before making any data architecture changes.

The best solution to the distance problem is to plan before committing to an office lease or other financial obligation. Examine the month-to-month telecom costs and see if its possible to locate your organization closer to a CO.


Analog service POTS v. digital

At one time there was only a single offering of Plain Old Telephone Service (POTS) available to business owners. When an organization wanted phones they had POTS lines installed. potsThese lines went from the CO directly to the business. The CO maintained large DC power supplies with batteries and generators to backup the power. The dial tone was generated in the CO when you picked up your phone receiver. This was the configuration that poor border collie was reacting to when the phone rang. A significant ring voltage was sent from the CO to the phone with enough current to cause the bell on the phone to react and ring.

For the past three decades telecom providers have been installing increasingly powerful CO switches that are all digital. These switches bring the CO closer to your business by placing active electronics in cabinets throughout communities. The availability of analog circuits is quickly disappearing.

Today all telecommunication is digital on the telecom providers network. If you really need an analog circuit, the telecom provider has the necessary cards to generate an analog signal, but it is only local and significantly less powerful.

digitalI can hear it now… the engineers are all pacing with their hands in the air shaking their heads and saying it is still all analog. They are 100 percent right. All digital circuits still rely upon an analog representation of a voltage pulse at a given value for a determined time interval. That pulse is then detected and represented as a ‘1’ if the voltage is high and a ‘0’ if the voltage is low. There, I said it. All the electrical engineers are calmer now and sitting.

Since digital electronics are all analog at their most basic level, they are still subject to the ‘IR Drop” over distance. Hence, the discussions of distance still applies. So what about fiber optics. Wing to our electrical engineers – fiber optics is still analog at its heart. What fiber optics allow us to do is to bring a signal farther with less loss due to the ‘IR Drop’. However, the optical signal is still converted to an analog signal of voltage pulses and interpreted as a digital signal.

This is all to say that there really is no analog vs. digital conversation occurring today. All telecom is presented as digital, and no border collies are frantically howling at the end of their tether.


The un-intended affect of systems failure.

The poor border collie was caught by surprise when the POTS circuit wire was rubbed bare and came into contact with the metal sheep gate. In like fashion, when a systems failure occurs with your business telecommunications, there may be unintended secondary failures that could potentially negatively affect your business.

credit-card-terminalsDetermining what these failures will be is sometimes difficult to quantify. Often there is a straightforward cause and effect relationship. For a restauranteur, the loss of the phone line that connects the credit card machine to the bank will mean that their batches for the day will not be sent to the bank and no money will be deposited into the restaurateur’s bank account. Other organizations might see a delay in synchronizing backups if their telecom connection is down, or may have a delay in paying an invoice, etc.

An unintended impact to a business is more complex and often not immediately apparent. Consider the loss of a telecommunications circuit that supports backup synchronization to a fail-over site. The immediate affect of a non-functioning telecom link is the halting of the backup synchronization.

Recovering from a synchronization delay is not unusual in most Information Technology divisions. It is well understood that backups will need to catch up when a telecom line is restored. Now consider the impact to an organization as the backup synchronization is recovering, and there is another telecom loss which delays an additional backup synchronization. Now there are two synchronizations pending and insufficient time to recover before the next daily synchronization is scheduled to start.

The backups are now out of sync and the recovery facility does not have the most current data upon which the organization would need to rely in the event of a failure. Inadvertently, the ability of the organization to successfully fail-over to a recovery site is impacted long after the telecom circuit is restored to normal operation. Hopefully there is a systems administrator who is diligent in their job to notice the situation.

To address the unintended consequences of failure it is helpful for an organization to develop an impact plan that speaks to operational activities and their related dependencies. The plan does not speak directly to technology issues, but does speak to work-flow and decision tree paths. By focusing on the dependency analyses contained in the plan as opposed to focusing on technologies specifically, the analysis tends to be more comprehensive and likely to identify critical paths that would otherwise be gleaned over.

Keep in mind that it is probably not possible to be 100% sure that all unintended consequences of failures are addressed. Use the 80:20 rule where there will always be 20% or issues not addressed by any plan. Armed with that knowledge always overestimate a bit to give your organization wiggle-room when the unexpected unintentional consequence of failure rears its head.

… and never tether a dog with a chain to a metal sheep paddock gate when in Booligal Australia. The sheep races are something to see!sheep


Add a Comment

Your email address will not be published. Required fields are marked *

More 2 posts in Planning
Recommended for you
text help
Foibles of inbound calls and texting

As much as we are influenced to think that texting has overtaken communications between a...