Google's 55 minute crisis
- 2 Feb 09, 10:21 GMT
It was the weekend which shook Google to its very foundations... possibly. All around the world, users of the search engine which prides itself - and has built a multi-billion dollar business - on providing the best, most useful answers to every query were greeted with a message warning that "this site may harm your computer".
The same text was pasted next to every search result - for everything from Aardvarks to Zoroastrianism. In effect, Google was warning users that the entire internet was sick and shouldn't be touched with an electronic bargepole.
Panic spread as the global web community told each other about this apparent breakdown. Google eventually fixed the problem, of a list of malicious sites somehow ended up including every web address, instead of one specific URL. But what a textbook example of a corporate disaster allowed to rage out of control...
...hold on a minute. The entire incident lasted just 55 minutes. And if you were on the West Coast of the USA, where Google is based, you are unlikely to have been affected, because it started at 0630 your time on a Saturday morning and was over by 0725. So why all the hoo-hah?
Well, news spread like wildfire around the blogosphere - or more accurately the Twitterverse - as everyone seemed determined to pass on their panic, along with rumours that Google's Gmail was also misfiring. So far. more people knew about the "may harm" incident than were affected by it. I got off a plane on my way home from a holiday on Saturday afternoon, asked online whether I'd missed any big tech stories, and was bombarded with messages about the Google crash.
So what's the verdict on the importance of this incident? decided that Google could have done better, awarding the company just 5 out of 10 for its handling of the crisis and accusing it of failing to respond as quickly as it could have done. That seems a bit harsh - it sounds like someone was hauled out of bed in Mountain View at 0645 on a Saturday to tinker with the engine, and got it sorted straight away. But it does go to show how rapidly you need to react to a crisis in a world where your mistakes will be the subject of web chatter within minutes of them happening.
Two Twitterers gave rather different views of Google's 55 minute crisis. , perhaps not completely seriously: "kennedy, lady di crash, and now #googmayharm! Everyone will remember what they were doing". : "small message under Google search results brings Twitterverse to a halt. Real world curiously unaffected!"
And the other thing to remember is that while the web's reactions may be fast, and its gaze intense, its attention span is very short. In Britain at least, a temporary problem with a search engine has been eclipsed by the arrival of snow. is the tag attracting thousands of comments on Twitter - and I suspect it may take a little longer than 55 minutes to sort that out.
The 大象传媒 is not responsible for the content of external internet sites
Comment number 1.
At 2nd Feb 2009, Chris Q wrote:Yes.
Shallow is a very appropriate word here.
Shallow view of the world - and to be honest - quite shallow snow considering the "mayhem" and "hysteria" being caused!
I twitter - therefore I have a smartphone.
Complain about this comment (Comment number 1)
Comment number 2.
At 2nd Feb 2009, _Ewan_ wrote:Google was warning users that the entire internet was sick and shouldn't be touched with an electronic bargepole.
It's certainly a point of view :-)
Complain about this comment (Comment number 2)
Comment number 3.
At 2nd Feb 2009, Paul Freeman-Powell wrote:It's quite amusing really.
We're all human, and we all make mistakes.
Everyone should just have a little giggle, and then move on...
Complain about this comment (Comment number 3)
Comment number 4.
At 2nd Feb 2009, hon3stly wrote:Google was out for 50 minutes....big deal!
Complain about this comment (Comment number 4)
Comment number 5.
At 2nd Feb 2009, Darren Stephens wrote:Crisis?
Ha ha.
What this does illustrate quite pointedly is the lack of critical rigour in the dea-eyed, bovine stampedes that are the twittererse and blogosphere. It's comical to see what amounts to a hi-tech game of Chinese Whispers being reported as newsworthy.
What actualy happened:
For 55 minutes a (ubiquitous, admittedly) software product had a (non-critical) problem. It was fixed. end of story. and thenit was seized upon by those with little better to do that propagate it with no critical faculty.
Can we have some sense of perspective here? Please?
Complain about this comment (Comment number 5)
Comment number 6.
At 2nd Feb 2009, SomeGuy wrote:Part of the "hoo-hah" would have been because while it might have been early in the morning on the US East Coast, it would have been mid-morning on the West Coast & early-afternoon here in the UK.
Another part is that Goggle does not make it easy to 'continue' onto the link you want, so many people were 'lost'. Personally, and unfortunately, even before this incident, they've had a significant number of "false positives" that have meant to me, at least, that what initially appeared to have been a useful feature was turning out to be a irritant enough to raise thoughts of alternatives...
Complain about this comment (Comment number 6)
Comment number 7.
At 2nd Feb 2009, Graham-Cluley wrote:All things considered, I think Google handled this really well.
They responded quickly, fixing the problem and posted an open and full explanation of what had gone wrong.
There wasn't any attempt to sweep the news under the carpet, but just an apology.
In fact, it could be argued that some good will have come from this. Imagine how many people who normally are lackadaisical about their anti-virus protection will actually have been stirred into scanning their computers by this incident?
If it caused people to check their PCs and update their anti-virus defences, then Google's human error could actually have done all of us a little bit of good.
Complain about this comment (Comment number 7)
Comment number 8.
At 2nd Feb 2009, Tugmyster wrote:Not much of a crisis. Also not so much a technical issue.
I copied the "Stopbadware.org" link that was on the page all google results were directing you to and pasted into the address bar, hidden in the text was the following...
...I-broke-google....
...It disappeared after pressing return and we could not get it to appear again.
Technical issues or hack?
Either way it was all very amusing. I considered the response time quite impressive really. Especially since the issue became intermittant then went away rather than just vanishing which you would expect with what Google described the issue as being.
Complain about this comment (Comment number 8)
Comment number 9.
At 2nd Feb 2009, bacon_n_beans wrote:....panic spread
....panic spread
Get a grip on yourself Rory
Complain about this comment (Comment number 9)
Comment number 10.
At 2nd Feb 2009, democracythreat wrote:I was there when it happened. My wife complained that Google wasn't working, and I sighed the deep sigh of the digitally able.
But sure enough, when i took control, google told me everything was toast. I use a mac, so my world was upside down. Either the mac had failed, or google had failed. Both possibilities are heresy.
People ask what is the big deal, but I don't think they realise how much our world has changed since the internet arrived.
"I don't know" now means "go away I am not interested".
"Can you tell me.." now means "I've lost my iPhone", or "I'm as crazy as a hat full of snakes so look out."
Old people must find it curious in the extreme. Myself, I can almost remember what it was like to value books, and to feel important because I had read a lot of them.
Complain about this comment (Comment number 10)
Comment number 11.
At 2nd Feb 2009, neile wrote:Crikey - at one point I thought my Mac had a bug...
Silly me
Complain about this comment (Comment number 11)
Comment number 12.
At 3rd Feb 2009, nameofuser wrote:I too was called upon by my partner to investigate what she deemed to be the likely possibility some eastern European hacking cartel had stolen her internets. Thirty seconds after booting up my machine, it became obvious Google had suffered nothing more than a human, if monumental, screw up.
Nothing to see here you would think, and to a certain extent (other than the opportunity for amusing screen grabs) you'd be right. But I think you may be missing the point Rory, as you didn't experience it first hand. Yes it lasted no longer than an hour, but in that hour both me and the missus set our default tool bar search engine to another provider and, as typical examples of humanity, unless something similar happens again, we'll unlikely be bothered to change it back again.
To Google, that matters.
Complain about this comment (Comment number 12)
Comment number 13.
At 3rd Feb 2009, Guv-nor wrote:A cautionary tale showing how much we have come to rely on too few baskets for our online eggs.
When Google went bad, I looked to 大象传媒 news and there was nothing.
Complain about this comment (Comment number 13)
Comment number 14.
At 3rd Feb 2009, Granten wrote:All it really says is that one shouldn't trust Google unconditionally. Simply because Google doesn't say a site is dangerous is no reason to assume it isn't. I will say that despite the stupidity of the problem, the staff do seem to have handled it well, ending the problem in under an hour.
Complain about this comment (Comment number 14)
Comment number 15.
At 3rd Feb 2009, Kevin wrote:For another google giggle, I just installed the latest version of Earth.
Got this message:
The Google Earth Plugin does not yet support this browser.
Supported browsers on Windows currently include Firefox, IE 6, and IE 7. Learn more 禄
What browser am I using, Ooops, Google Chrome.
Complain about this comment (Comment number 15)
Comment number 16.
At 3rd Feb 2009, Nick Roberts wrote:As a professional programmer, I can attest as to how easy it is to make this kind of mistake.
Suppose the record to be updated had an 'id' of 897345987. The SQL command to update it would be something like:
UPDATE The_Big_Index
SET Give_User_Warning = True
WHERE ID = 897345987
It is horrifyingly easy to omit the WHERE ... part, especially if you are in a hurry (and these days, as a highly paid ;-) professional, one generally is), and if you do you end up updating every single record in the table, even if there are billions of them.
Last year I deleted an entire table in a 'live' database due to a similar aberrance. Fortunately, there was a backup taken a few minutes before, but if there had not been I would have been in it up to the armpits, as the expression goes.
SQL needs the addition of two facilities: one is an 'idiot' flag that can be set on a database user; the other is a ROLL BACK FROM IDIOT command.
Complain about this comment (Comment number 16)
Comment number 17.
At 3rd Feb 2009, Darren Stephens wrote:"SQL needs the addition of two facilities: one is an 'idiot' flag that can be set on a database user; the other is a ROLL BACK FROM IDIOT command."
But SQL already has this. The first is table and database level security controlled via privileges in SQL and by other means in individual products. The second is COMMIT and ROLLBACK.
Why were you working on the live db anyway? :)
Complain about this comment (Comment number 17)
Comment number 18.
At 4th Feb 2009, dennisjunior1 wrote:Rory:
I know, I used GOOGLE as my search engine on Saturday....Those messages were showing up on my computer.....
And, even I had an email with a RED warning...I know that source of information.
~Dennis Junior~
Complain about this comment (Comment number 18)
Comment number 19.
At 8th Feb 2009, aniakovas wrote:I got this. After a minutes I understood that it was clearly an updating error, that would be fixed. Since I have sites I Googled them to see if they were flagged to confirm, as I could independently check the files.
Move on people, mass panic not required.
_________
Google works on live data because it is so distributed, updates more or less must happen live. No system is perfect, there is mathematics to prove this must be the case. We need, collectively to get over it.
If people were that worried about perfection, they would not use Wikipedia.
Complain about this comment (Comment number 19)