Thanks to their high level escalation team I’ve now received an official explanation from British Telecom about the events that led up to our broadband outage on October 3rd. Here it is in full:

Loss of BT Retail Business Broadband Service on 3rd October 2011

Like any major national organisation we experience power outages on occasions. In order to protect our systems and therefore customers’ services, we have warning systems and emergency generators that ensure customers are rarely aware of any power outage affecting our facilities.

 

On Monday 3rd October 2011 there was a loss of the external power supply to Birmingham Telephone House at 11:41 am. Internal backup maintained power until 12:29 pm at which point a total loss of power occurred. External power supply was restored ten minutes later at 12:39pm. The external power outage was caused by a main circuit breaker tripping, followed by a failure of the backup power supply after 48 minutes.

 

The total loss of power at 12:29 pm caused a loss of Broadband service for a total of 1.3 million BT Retail Consumer & Business broadband customers.

1.2 million BT Retail Consumer & Business Broadband customers had service restored in under one hour. 0.1 million Business Broadband customers who use a static IP address required more complex resolution and service was restored within 3 hours to more than 80% of these customers. 18 thousand Business Broadband customers took longer than 3 hours to restore service because of a static IP traffic routing error. Service was restored to all these customers within 24 hours and this routing error has been identified and fixed.

 

We provide priority repair options for businesses that apply to the vast majority of faults. Major incidents like this are an exception and we worked to restore systems as quickly as we could, whether Business or Consumer. A power management team was deployed instantly to Birmingham Telephone House and have implemented changes to prevent this incident from happening again. Whilst we do not guarantee that Broadband service will be uninterrupted we recognise how important it is for businesses to have a stable and reliable internet connection. We have 99.9% network reliability and incidents of this nature are extremely rare. We take every precaution to prevent incidents happening and to learn from incidents to prevent them happening in the future. Should anything go wrong we provide 24×7 technical support from business dedicated UK based teams.

That all goes some way towards explaining how an 8 minute power cut in the middle of Birmingham at 11:41 eventually caused our broadband connection to fail here in the middle of nowhere in South West England at 12:32 or thereabouts. However that explanation raises at least as many questions as it answers. For starters, here’s the one uppermost in my mind at the moment:

Who was responsible for the “main circuit breaker” that seems to be at the heart of the matter? Western Power Distribution say it wasn’t them, and BT Retail say it wasn’t them.

The BT high level escalation team have suggested to me that the answer to that question might possibly be BT Operate, and that I call the BT Corporate Press Office to see if they can help. I think I’ll give that a try next.