British Telecom Explain October 3rd Broadband Problem

Thanks to their high level escalation team I’ve now received an official explanation from British Telecom about the events that led up to our broadband outage on October 3rd. Here it is in full:

Loss of BT Retail Business Broadband Service on 3rd October 2011

Like any major national organisation we experience power outages on occasions. In order to protect our systems and therefore customers’ services, we have warning systems and emergency generators that ensure customers are rarely aware of any power outage affecting our facilities.

 

On Monday 3rd October 2011 there was a loss of the external power supply to Birmingham Telephone House at 11:41 am. Internal backup maintained power until 12:29 pm at which point a total loss of power occurred. External power supply was restored ten minutes later at 12:39pm. The external power outage was caused by a main circuit breaker tripping, followed by a failure of the backup power supply after 48 minutes.

 

The total loss of power at 12:29 pm caused a loss of Broadband service for a total of 1.3 million BT Retail Consumer & Business broadband customers.

1.2 million BT Retail Consumer & Business Broadband customers had service restored in under one hour. 0.1 million Business Broadband customers who use a static IP address required more complex resolution and service was restored within 3 hours to more than 80% of these customers. 18 thousand Business Broadband customers took longer than 3 hours to restore service because of a static IP traffic routing error. Service was restored to all these customers within 24 hours and this routing error has been identified and fixed.

 

We provide priority repair options for businesses that apply to the vast majority of faults. Major incidents like this are an exception and we worked to restore systems as quickly as we could, whether Business or Consumer. A power management team was deployed instantly to Birmingham Telephone House and have implemented changes to prevent this incident from happening again. Whilst we do not guarantee that Broadband service will be uninterrupted we recognise how important it is for businesses to have a stable and reliable internet connection. We have 99.9% network reliability and incidents of this nature are extremely rare. We take every precaution to prevent incidents happening and to learn from incidents to prevent them happening in the future. Should anything go wrong we provide 24×7 technical support from business dedicated UK based teams.

That all goes some way towards explaining how an 8 minute power cut in the middle of Birmingham at 11:41 eventually caused our broadband connection to fail here in the middle of nowhere in South West England at 12:32 or thereabouts. However that explanation raises at least as many questions as it answers. For starters, here’s the one uppermost in my mind at the moment:

Who was responsible for the “main circuit breaker” that seems to be at the heart of the matter? Western Power Distribution say it wasn’t them, and BT Retail say it wasn’t them.

The BT high level escalation team have suggested to me that the answer to that question might possibly be BT Operate, and that I call the BT Corporate Press Office to see if they can help. I think I’ll give that a try next.

Contemporary Art in the Community #2/n

The second video of the Contemporary Art in the Community project.

Kasia’s Studio (My summer studio)
The Haldon Hills, Devon, South West England
Saturday 1st October 2011

An interview with Doreen, in which she recognises Hell when she sees it, and feels herself drowning in a really stormy, turbulent sea.

 

Hell-Purgatory-Heaven - triptych by Kasia B.Turajczyk

BT Improve Broadband Network, and Ensure Prompt Payment?

My British Telecom Business Broadband Connection stopped working yet again this morning! The lights on our router would turn off as it made some loud clicking noises. The router would start itself up again and eventually reconnect us to the internet. We could browse the web for a minute or two then the whole process would repeat itself, apparently ad infinitum.  On one of those brief browsing sessions the following message from BT appeared on my screen:

BT request prompt payment for their business broadband "service"

BT request prompt payment for their business broadband "service"

As luck would have it I was already experiencing some disruption to my “service”. I certainly didn’t want things to get any worse, so I did my best to pay as quickly as humanly possible, and clicked the “Pay Online” (It’s fast and easy!) button. On the next screen BT invited me to part with my credit card details, which unfortunately I was unable to do, since the appropriate boxes were missing:

BT request my credit card details, but forget to provide the right boxes!

BT request my credit card details, but forget to provide the right boxes!

I clicked “Next” anyway, and BT invited me to part with my credit card details once more, which I dutifully did:

BT request my credit card details once more.

BT request my credit card details once more.

I clicked on “Pay Now”, only to discover that I couldn’t pay now:

BT helpfully block "Verified by VISA"

BT helpfully block "Verified by VISA"

That’s because BT were helpfully blocking access to any site other than their own, including VISA’s attempt to use 3-D Secure to prevent fraudulent transactions on my credit card account.

Since my router was already rebooting itself with monotonous regularity I figured that recommended course of action wouldn’t help a whole lot. I gave up battling BT’s excuse for a broadband “service” at this juncture, and phoned up BT’s telephone payment system instead, which did at least eventually manage to automatically take £126 pounds off me. The only trouble then was that my BT broadband router was still disconnecting me from the internet every couple of minutes.

Next I phoned my old friends on the BT Broadband helpline.  This time around I listened to an automated message that was brand spanking new, to me at least:

BT is currently doing work to improve the broadband network. This may cause some customers to experience a temporary loss of service.

I was certainly experiencing a continually repeating temporary loss of service, so maybe my tardy payment was merely a red herring?  I’d discovered I’d have to wait to find out since the “wait time is 5-10 minutes”. Unlike last time I was at least offered the option of receiving a call back from BT when someone was free to answer my questions. Twelve minutes later Mark from Dundee did indeed ring me back, but he was unable to answer my first question, since he and his computer system appeared to have no knowledge whatsoever about any ongoing “work to improve the broadband network”.  Neither had BT’s broadband status page:

BT's known broadband problems at 11:27 on November 7th 2011

BT's known broadband problems at 11:27 on November 7th 2011

Mark didn’t think my continuing disconnection problem was anything to do with BT’s methods for persuading people to pay promptly, but could offer no alternative suggestions. By this time our router seemed to have settled down, and it had stopped frantically clucking and clicking. Rather than waste any more valuable time of either Mark or myself I resigned myself to accepting just one more of life’s little mysteries.

P.S. I’ve discovered that if you lie to BT’s original message asking you if you’re the account holder, and instead tell them that you’re not, they will let you play around on the web long enough to successfully pay your account online using a credit card!

Western Power Confirm British Telecom Power Cut Lasted 8 Whole Minutes

I have made some more phone calls to British Telecom since my last blog post, and my persistence has been rewarded because I’m now in touch with one of their high level escalation specialists. However I still haven’t had a huge amount of joy getting any convincing answers concerning exactly what went wrong with BT’s broadband service on October 3rd.

However I have had rather more success with my enquiries at Western Power Distribution. They are responsible for keeping the lights on in Newhall Street in Birmingham, and I called them up to discover if the initial version of events I gleaned from the Birmingham Mail was accurate or not.  I spoke to their corporate communications department, who in turn spoke to the relevant district manager for the area in question.  It seems the problem was a “catastrophic failure” in a 132 kV underground electric cable at 11:41 on the 3rd. However WPD employ “triple redundancy” on their high voltage cables. 7,295 WPD customers including BT, were initially affected, but by judicious switching to alternate supplies power was restored to every customer within 8 minutes.  Western Power assured me when we spoke that original fault was in the process of being repaired, and that all 3 HV cables would be back in action soon.

This leaves a number of unanswered questions, foremost amongst which in my mind at present is the following:

If electric power was restored to British Telecom’s facilities in Birmingham by 11:49 on the morning of October 3rd, what precisely was it that caused my broadband connection to suddenly stop working at 12:32 or thereabouts?

How to Complain About British Telecom Broadband – Free of Charge!

Kasia’s been moaning at me for spending too much time on the telephone to BT about broadband outages, but since a week has passed since my last such conversation I thought now might be a good time to update my loyal reader(s) on yet another telephone conversation I had last Friday afternoon.

Following Anthony’s suggestion I phoned up Ofcom at 16:40, and given my recent experiences with British Telecom I was pleased to discover that according to their automated operator I was first in the queue to speak to a real human being. Shortly thereafter I found myself speaking to Alex, who wanted to know if I wished to register a complaint.  I told him that whilst I was certainly not exactly a happy bunny, at this juncture I was only seeking information and advice.  Having briefly outlined the contents of this blog for him, Alex suggested that my next telephone call should be to the Ombudsman Services for Communications to find out more about something called the “Alternative Dispute Resolution Scheme” (or ADR for short”.

According to the Ofcom website:

Under Ofcom’s regulations, phone companies must be a member of a recognised Alternative Dispute Resolution Scheme.
The schemes provide an impartial alternative if you and your phone company can’t agree about a complaint.
It is free, and is open to residential customers as well as small businesses with up to 10 employees.

Here at The Dreamers we’re certainly a small business, and we most definitely have less than 10 employees, so maybe I’m getting somewhere useful at long last?

Also according to Ofcom:

If you haven’t reached an agreement with your provider after eight weeks – or earlier, if they agree you’re at a stalemate – you can then ask an ADR scheme to consider the case. The complaint must also be less than 9 months old.

It looks as though I still have plenty of time left to continue my fact finding mission before registering an official complaint, so I still haven’t got around to clicking the big blue button on the Ombudsman’s website:

How to make a complaint to the Communications Ombudsman

How to make a complaint to the Communications Ombudsman

BT Apologise to Another 100,000 Broadband Customers

Having finally received a reply from British Telecom to my written “complaint” about their (lack of) broadband service, I called the BT Broadband Technical Helpline once again at 15:50 BST this afternoon. A recorded voice assured me that the waiting time was currently over half an hour. Unlike my previous calls, this time around I wasn’t offered a call back, so I waited on the line for over half an hour. BT’s helpline helpfully kept repeating the same message:

  • Thank you for holding
  • We are very busy at the moment and apologise for the delay
  • Your call will be answered as soon as possible

“As soon as possible” turned out to be 16:27, when Anthony picked up at BT’s end. I enquired about the reason for the long delay, and he told me that BT had been receiving lots of calls from the Sheffield area that day, because at one point around 100,000 people had been without a broadband connection. The BT broadband service status page seemed to confirm as much:

A long list of BT broadbandless dialling codes on October 14th 2011

A long list of BT broadbandless dialling codes on October 14th 2011

I’m slightly curious about how a problem reported at 11:46 can be fixed by 11:40, but I digress. Anthony couldn’t give me any details about the causes of today’s problem, so I asked him instead about the causes of the October 3rd broadband outage. He told me that one of BT’s authorisation nodes in Birmingham had suffered a power cut, and that as a result 25% of BT’s broadband customers had lost their internet connection, around 1.2 million customers in total. I told Anthony I was more than slightly curious, because last week William had told me that 3 of BT’s authorisation nodes had gone down, and 2 million customers were affected. I gave Anthony some more details of my requests for more information about the underlying causes of that outage.  I idly wondered whether BT Wholesale possessed such things as batteries and/or diesel generators to prevent brief power cuts in Brum from inconveniencing millions of their customers nationwide. I idly enquired whether BT could reveal the area in which power was lost, so that I might pursue that line of enquiry. Anthony went away to speak to his supervisor. When he eventually returned he passed on to me his supervisor’s suggestion, which was that if I was still unhappy I should give Ofcom a call. I assured Anthony that I was indeed still unhappy, and would follow that course of action.

Contemporary Art in the Community Launched at Last

The average person visiting contemporary art galleries or modern art centres at the present time doesn’t understand most of the art works presented. She or he wants to understand the “art works”. Because they can’t understand it or explain it, or find any sense in it, they decide they don’t possess the institutionalised knowledge of contemporary art. They decide they just don’t like it! The secret is: nobody understands it; not the art historian, not the art curator, not even the people who are writing the articles in the art magazines or the introductions in the books. Very often even the artist her/himself doesn’t understand it either. And it is OK! Art doesn’t exist anymore. Most of modern day ‘Art’ isn’t art any more. Nothing is art and everything is art. Art is about personal and emotional engagement. You like it or you do not, you feel it or you do not feel it; it makes you laugh, it makes you cry, it makes you angry, it makes you disgusted. It is OK!  Art is about feeling….or should I say NON-ART is about sensation. Never think about what an artist is trying to convey, concentrate on the piece and what it does with you, how it makes you feel. After all it is your personal experience!

Pablo Picasso said:

Everyone wants to understand art. Why not try to understand the song of a bird? .. People who try to explain pictures are usually barking up the wrong tree.

Contemporary Art in the Community is a joint project between Kasia Turajczyk and econnexus.org. 

The primary aim of the project is to take contemporary art out of galleries and museums, to take it out of artist’s studios, and to show it to all sorts of different people in a variety of environments and locations around South West England.

We would like to discover from the ordinary people of South West England what kind of art they like and what they don’t like. What contemporary art means to them?  Are they interested in art at all? Is there a place for art in their life, living room, kitchen, mind? Or maybe they are not interested in art at all? We will let them see diverse art works and ask them about their feelings, their thoughts, and about them. These are the questions and the items we are going to be asking people.

Maybe we will ask you too!

Contemporary Art in the Community Nr. 1/n

Widemouth Bay, North Cornwall, South West England –  Thursday 29th September 2011

BT Apologise Again, but Offer Only Their Regrets

At long last I’ve received a response from BT to the “complaint” I sent them on October 6th about last weeks massive broadband outage. It seems nobody looked at it until today, whereupon they responded swiftly! Here’s the meat of it:

Thank you for your enquiry, received on 12-10-11

First of all, please accept our apologies for any delay in this email reaching you, as we have been extremely busy lately.

I am sorry to hear you’ve been adversely affected by our recent outage.

Unfortunately we do not have any further details as to the cause or nature of the outage, and following discussions with my supervisors regarding this, I regret we are unlikely to receive any further information from BT Wholesale on this matter.

I do apologise for again any inconvenience caused during this time.

Here at Marketing Dreams we have certainly suffered from some inconvenience, and we will continue to seek  ”further information from BT Wholesale on the recent outage”. Surely somebody somewhere must have some idea about what went wrong?

Would Artificially Intelligent Agents Assist British Telecom?

A new week has dawned, and our BT Business Broadband connection has had a lot of use today.  Fortunately it stood up to the strain much better than it did last Monday. Following that fiasco BT Wholesale still haven’t got back to us with an explanation of what went wrong, after almost a week of asking.

On a (wholly unrelated?) topic, the Stanford University online courses in both Artificial Intelligence and Machine Learning started today. I’ve signed up for both of them, whilst Kasia is doing just the AI one. In something of a departure from the usual syllabus, we’ve also simultaneously signed up to become involved in an “Artistic Intelligence” project!

Take a quick look at the Stanford AI Class “Intro to AI” video 1.4:

As you can see “financial trading agents” are the first example that Stanford offer of a “real world” use of artificial intelligence. According to Sebastian Thun, one of the joint course leaders:

There’s a huge number of applications of artificial intelligence in finance, very often in the shape of making trading decisions, in which case the agent is called a “trading agent”.

As luck would have it I’ve been prattling on about such things for a while now over at the Trading Gurus.  If you take a look at the video in that link you will note that at around 2:30 Professor Dave Cliff of the University of Bristol states that:

Managing the hot air in a data centre is a huge problem. When a Computer Room Air Conditioning Unit fails everything gets very hot, very quickly. You have about 3 minutes before the motherboards physically melt.

One can’t help but wonder at this juncture if British Telecom might not be able to benefit from using similar technology to prevent their broadband network from suffering meltdown every once in a while?

 

Mintaka’s Message for Xmas Shoppers

I went to the supermarket yesterday. It is now obvious that Christmas is coming. Mintaka’s message bears repeating. Here it is once more, for Christmas shoppers everywhere.

Merry Christmas from Mintaka
Merry Christmas from Mintaka

My name is Mintaka. I am a stranger here on Earth; I was born in the Nebula of Orion.

Your species is dominant, but weird. The only time in the long calendar of the year
when you are kind, forgiving, charitable and pleasant to each other is at Christmas time.
It is the only time when “men and women open their shut-up hearts freely”.
(One of your famous writers pointed this out a long time ago).

I am not sure I understand you. Why don’t you behave in such a way on all the other days of the year? I am watching you and waiting to see you beautiful and brave.

Maybe one day….