Technical problems
It's not often we get a message from the ´óÏó´«Ã½'s technical support teams saying, "Total outage of all ´óÏó´«Ã½ websites".
But for getting on for an hour this evening, until just before midnight, that's what happened. We haven't yet had a full technical debrief, but it's clear it was a major network problem.
We'd like to apologise to everyone who couldn't get onto the ´óÏó´«Ã½ News website during that time.
Steve Herrmann is editor of the .
Comment number 1.
At 30th Mar 2011, mym wrote:This comment was removed because the moderators found it broke the house rules. Explain.
Complain about this comment (Comment number 1)
Comment number 2.
At 30th Mar 2011, Guv-nor wrote:If the internet has an equivalent to the ticking metronome of Radio Leningrad for many bbc.co.uk is that reassurance.
Thankfully Radio 4 Long Wave did not go off the air at the same time.
Complain about this comment (Comment number 2)
Comment number 3.
At 30th Mar 2011, asif wrote:Greatest support ticket title ever.
Complain about this comment (Comment number 3)
Comment number 4.
At 30th Mar 2011, magnificentpolarbear wrote:It's only a website!
Usefeul to have but the world turned nonetheless - there was no juddering halt.
There are people in Japan still homeless and searching for loved ones.
There are people in Libya (and other countries) fighting for their basic human rights.
Those put the loss of the ´óÏó´«Ã½ website(s) for a couple of hours into perspective.
No need for anyone (and I bet there will be some) to go overboard demanding public enquiries and the head of the DG.
Complain about this comment (Comment number 4)
Comment number 5.
At 30th Mar 2011, I am the Editor wrote:I demand a full public enquiry and the head of the DG.
Thank you.
Complain about this comment (Comment number 5)
Comment number 6.
At 30th Mar 2011, Stephen Turner wrote:Whenever the ´óÏó´«Ã½ News website is unreachable, the first thing I do is head to CNN's website to see if something terrible has happened in the world.
Complain about this comment (Comment number 6)
Comment number 7.
At 30th Mar 2011, Andrew Betts wrote:Stephen: Like relying on CNN for news?
For me it's quite refreshing to know that internet gremlins even catch Auntie unawares occasionally.
Complain about this comment (Comment number 7)
Comment number 8.
At 30th Mar 2011, BenitoB wrote:Not sure what your techies told you, but your leading story about what happened contains a big error: "Users trying to get on to the ´óÏó´«Ã½ website on Tuesday evening from around 11pm to midnight were confronted with an error message, showing the ´óÏó´«Ã½'s iconic test card."
Your bbc.co.uk domain wouldn't resolve at all, which means it wouldn't display a custom ´óÏó´«Ã½ 'iconic test card' just as much as it wouldn't display any other page from your server. We just got a browser error page that the website couldn't be found.
Complain about this comment (Comment number 8)
Comment number 9.
At 30th Mar 2011, Macin Tosh wrote:I understand the solution was to switch the ´óÏó´«Ã½ off, wait 10 seconds, and then switch back on again?
Complain about this comment (Comment number 9)
Comment number 10.
At 30th Mar 2011, Macin Tosh wrote:@BenitoB
Yup, I was surprised to read that about the test card - I certainly didn't see one.
Complain about this comment (Comment number 10)
Comment number 11.
At 30th Mar 2011, Alex wrote:It was a DNS problem. All 4 of the bbc hosted name servers were unreachable. As a result there was no way to get an IP address for any ´óÏó´«Ã½ domain.
I was quite surprised as the IP addresses were in distributed net blocks so I assumed they were physically separated.
Complain about this comment (Comment number 11)
Comment number 12.
At 30th Mar 2011, Justin wrote:Is an hours outage really a problem? I've been impressed with the availability of web services from the ´óÏó´«Ã½ and it's that classic issue of people always complain about things when they are down and never congratulate when they are up and running. I'll bet it if you look at the availability of services they are better than 99.9% which is no mean feat given the complexity of the infrastructure that will support these web services. Give the lads a break, in IT, things break every so often with varying effects, as it was only an hour I'm sure there was huge dedication to get it sorted asap...
Complain about this comment (Comment number 12)
Comment number 13.
At 30th Mar 2011, RobKi wrote:My hunch is that the outage was caused by thousands of former 648 kHz listeners in Europe trying to access the ´óÏó´«Ã½ player in order to listen to the World Service.
Complain about this comment (Comment number 13)
Comment number 14.
At 30th Mar 2011, Peter Galbavy wrote:"oops"
Complain about this comment (Comment number 14)
Comment number 15.
At 30th Mar 2011, JunkkMale wrote:This comment was removed because the moderators found it broke the house rules. Explain.
Complain about this comment (Comment number 15)
Comment number 16.
At 30th Mar 2011, Steven wrote:more wikileak hacker attacks anyone
Complain about this comment (Comment number 16)
Comment number 17.
At 30th Mar 2011, olly183 wrote:Oh dear did it happen again? approx 19:00 on the 30th? I'm in Barcelona and had nothing for about on hour, same on Iphone and laptop.
Complain about this comment (Comment number 17)
Comment number 18.
At 31st Mar 2011, Tio Terry wrote:This comment was removed because the moderators found it broke the house rules. Explain.
Complain about this comment (Comment number 18)
Comment number 19.
At 1st Apr 2011, GeoffWard wrote:How we are conditioned by our expectations:
I read: Total outage of ´óÏó´«Ã½ websites
I saw : Total outrage of ´óÏó´«Ã½ websites
Complain about this comment (Comment number 19)
Comment number 20.
At 1st Apr 2011, GeoffWard wrote:This comment was removed because the moderators found it broke the house rules. Explain.
Complain about this comment (Comment number 20)
Comment number 21.
At 1st Apr 2011, MrWonderfulReality wrote:These things happen, nothing is infallible, just as well ´óÏó´«Ã½ is not a nuclear powerstation.
===============================
4. At 02:58am on 30th Mar 2011, magnificentpolarbear wrote:
It's only a website!
Usefeul to have but the world turned nonetheless - there was no juddering halt.
There are people in Japan still homeless and searching for loved ones.
There are people in Libya (and other countries) fighting for their basic human rights.
Those put the loss of the ´óÏó´«Ã½ website(s) for a couple of hours into perspective.
No need for anyone (and I bet there will be some) to go overboard demanding public enquiries and the head of the DG.
=======================================
How about a revolution then, to make sure that ´óÏó´«Ã½ has enough back up systems to never go down, but nothing overboard, just tanks & planes & not the full monty!!!
Complain about this comment (Comment number 21)
Comment number 22.
At 1st Apr 2011, Debbie Rockford wrote:Oops... Perhaps I shouldn't have been so quick to blame my South African internet service provider for the poor service - thought it was one of the usual local incidences where we can't access international websites for a few hours at a time.. didn't even cross my mind that the ´óÏó´«Ã½ sites themselves may be at fault!
Debbie
Complain about this comment (Comment number 22)
Comment number 23.
At 1st Apr 2011, This is a colleague announcement wrote:I'm sure it's not for technical reasons, but why was the Fukushima thread closed to entries after a short, but busy time? (The one on climate change has been open, but dormant, for about a year).
I don't know why I ask really: the answer's fairly self evident to most people.
Complain about this comment (Comment number 23)
Comment number 24.
At 1st Apr 2011, JunkkMale wrote:Have to say you are excelling yourselves.
Excising a post linking to a ´óÏó´«Ã½ online article about Government problems with internet delivery on a blog about ´óÏó´«Ã½ problems with, er, internet delivery as 'off topic', takes some rare doing.
I'd say the problems go beyond technical.
Complain about this comment (Comment number 24)
Comment number 25.
At 1st Apr 2011, GeoffWard wrote:This comment was removed because the moderators found it broke the house rules. Explain.
Complain about this comment (Comment number 25)
Comment number 26.
At 1st Apr 2011, David0944 wrote:I have been using the ´óÏó´«Ã½/Kent webpage as my homepage for some time now (at least a coupe of years) and had notice quite a few good updates during that time.
However, I had been away from home for a few days and came back to the abomination that tries to pass itself off as the ´óÏó´«Ã½ Kent homepage.
What made you decide to wreck a wonderful webpage? I doesn't even refresh properly of update itself in any reasonable way. For instance, if I want an update on local weather, I have to go to the weather page.
Please reinstate the excellent homepage that came before this current mess. If not, then it's goodbye ´óÏó´«Ã½ website for ever.
Complain about this comment (Comment number 26)
Comment number 27.
At 1st Apr 2011, GeoffWard wrote:This comment was removed because the moderators found it broke the house rules. Explain.
Complain about this comment (Comment number 27)
Comment number 28.
At 2nd Apr 2011, GeoffWard wrote:IShouldBeTheEditor wrote @ #5: "I demand a full public enquiry and the head of the DG."
I ask for the similar removal of the same bit of Steve Herrmann for the messy **** that has become HYS/Editors. (#20, #27 Censored).
here I introduced the word for prematurely removing an unwanted foetus - you know, a literary analogy.
Apparently if you ask for the ´óÏó´«Ã½'s Director General to have his head severed, this is OK, see #5.
´óÏó´«Ã½ Moderation/Censorship is a bit like the Spanish Inquisition - nobody expects them, nobody knows what the real rules of punishment involve, some escape scot-free, others are censored out of existence.
I guess serial censorship shows that Steve has friends in inquisitorial high places but it's OK for posters to request the head of the DG ( a bit biblical, though).
For God's sake, moderators, lighten up - it's only a blog!
Complain about this comment (Comment number 28)
Comment number 29.
At 2nd Apr 2011, eva888 wrote:This comment was removed because the moderators found it broke the house rules. Explain.
Complain about this comment (Comment number 29)
Comment number 30.
At 3rd Apr 2011, Traffic Man wrote:This comment was removed because the moderators found it broke the house rules. Explain.
Complain about this comment (Comment number 30)
Comment number 31.
At 3rd Apr 2011, raaj wrote:There are words like mad population about Indians when mentioning the victory of ICC World Cup in the ´óÏó´«Ã½ Scrooling News.Can we Say you English people are mad About foot ball so that mad English poulation (!).It is one of your So called mad English men Started playing Cricket You Know?!.A global News Channel LIke ´óÏó´«Ã½ Shouldnot use these kinds of words.Regards.
Raaj
India
Complain about this comment (Comment number 31)
Comment number 32.
At 3rd Apr 2011, This is a colleague announcement wrote:What Ecclefechan good is it having a blog on technical problems if we can't post, as I find?
Complain about this comment (Comment number 32)
Comment number 33.
At 4th Apr 2011, GeoffWard wrote:.
You may have Total Technical Breakdown, but you have a much more FUNDIMENTAL PROBLEM.
Why have a Have Your Say that nobody can contribute to? Like today, 4 April, 2011.
How can you call it "Have Your Say" when that’s exactly what we cannot do?
You have destroyed this site, it was better in the past, by no means perfect - but now it is totally worthless.
You have HYSs running well past their end-dates,
whilst *most* popular subjects are not covered at all.
What you have done to it is a total disaster.
From something that was the envy of many other media-related, similar systems, you have reduced it to total rubbish.
You should be ashamed of yourselves and you should have your monopoly in terms of Licence Fees removed from you, to allow competition in this area.
I know others could do much, much better than you are doing now.
I've always been a supporter of the ´óÏó´«Ã½ but this is the last straw for me.
Your HYS is a total, abysmal failure.
I will support further withdrawal of (our) public funding because of the unforgivably poor service you are now providing.
Why have you not listened to the acres of criticism and freely-given counselling on your key areas of weakness (many)?
When we give you on-line guidance, the least you can do is acknowledge it and incorporate such guidance as is sensible – and there are hundreds of contributors with high levels of professional knowledge who KNOW what sort of service they prefer.
Whether you like it or not, you are held to account- you, Steve, and all the other ‘Editors’ that are responsible for this signal and continuing failure.
I really think the time has come for those of you that are responsible for this mess that ´óÏó´«Ã½ Online has become, to consider your positions.
[*** These comments are a synthesis of many posted comments - THEY ARE IMPORTANT.]
And for God's sake, Moderators, try to do something more than just brain-dead censoring - COLLECT the file of comments on this DISASTER and put it in the hands of the Senior Executive.
.
Complain about this comment (Comment number 33)
Comment number 34.
At 5th Apr 2011, moriaeencomium wrote:Let me share, I've experienced this difficulty and it made me turn to twitter for a very first time, in real time. I've thought, look I've just read how ´óÏó´«Ã½ wants some sort of spearheading with its @´óÏó´«Ã½Breaking Twitter account and now I'm looking at it as it explains ´óÏó´«Ã½'s breakdown. I've thought, by the tweeting nightingales, even if it isn't some wacko's commercial stunt its marvelous happening.
It also made me examine twitter more closely… stream of thoughts. It would be fun to have one of those state of the art analytical algorithms to analyze all that.
'Full technical debrief', that should be interesting too.
Complain about this comment (Comment number 34)
Comment number 35.
At 5th Apr 2011, clicky wrote:Gone are the days where you could get away with a little bit of down time! Its amazing how fast news of this type of thing spreads with Facebook etc.
The latest stats show the UK now has nearly 29 million Facebook users!
Complain about this comment (Comment number 35)
Comment number 36.
At 8th Apr 2011, Kit Green wrote:What is happening with the near total loss of comment threads on ´óÏó´«Ã½ blogs last night and still the same now?
Complain about this comment (Comment number 36)
Comment number 37.
At 8th Apr 2011, Olga Rikova wrote:Are the technical problems still persisting - it seems that on certain editor blogs posts (e.g. on Alistair Burnett's latest post) the commenting does not seem to be working (keep getting the message "We're having some problems posting your comment at the moment. Sorry, we're trying our best to fix it")
Complain about this comment (Comment number 37)
Comment number 38.
At 9th Apr 2011, BluesBerry wrote:It's been three days since I've been able to post to Mardell, Harding, or Peston.
I can't tell you how frustrating this has been because I do give thoughtful analysis to each submission that I make and to watch them circle the brink and eventually become untimely is rather despairing.
I don't know what's wrong, but I deleted another three comments from my drafts' file this AM.
I'm exasberated. I really enjoyed reading the blogs, thinking about my input and making an intelligent submission, but I can't see wasting my time on comments that are going nowhere.
Complain about this comment (Comment number 38)
Comment number 39.
At 14th Apr 2011, Raminder Dhillon wrote:How do you define Network Problems ?
Complain about this comment (Comment number 39)
Comment number 40.
At 2nd May 2011, OldAnarchist wrote:The ´óÏó´«Ã½ site seems to be crawling today - too many searches on Osama's death on the news? Also it seems the commenting on these blogs is also just working intermittently (keep getting a message stating that there has been an error submitting your comment in a little pink box above the comment box after I click on post) and you're unable to edit the comment box to copy the message to retry either - I've lost a couple like this, this afternoon already...
Complain about this comment (Comment number 40)
Comment number 41.
At 11th May 2011, Sam Lowry wrote:OK folks it's time I owned up - after bidding on - and failing to get any of my 20,000 2012 Olmpic tickets I happened to mention - in a kind of off-hand, apropo of nothing kind of way to that nice Mr Cable Guy, about becoming an 'Any Willing Provider' of internet servers on behalf of the Beeb ... using my trusty old Dual 2.4GHz laptop as the server ... next thing you know my modem is indulging in a major freak-out, go figure that one, so I fail to understand how this could have in any way have originated 'the problem' as a bloke in Abbottabad assured me he had been using this very set-up for at least 5 years via an old MILSTAR II satelitte link (although he has recently been oddly abscent from ICQ and Skype).
While we are monitoring the Beeb's every bowel movement, "quis custodiet ipsos custodes?" ... quis custodiet ipsos
custodes? Ans, not me. I get my reliable internet national news from CBC ... not ´óÏó´«Ã½!
Complain about this comment (Comment number 41)
Comment number 42.
At 11th May 2011, perkinwellbeck wrote:i do not appear to be able to access my postings,can you put it right for me.it is only a broblem on the original HYS site,would be greatfull..any chance of returning to the HYS in its original format?? pretty please??
Complain about this comment (Comment number 42)
Comment number 43.
At 14th May 2011, David King wrote:This comment was removed because the moderators found it broke the house rules. Explain.
Complain about this comment (Comment number 43)