大象传媒

'people woke up to find their mailboxes empty' - Wow what a tragedy. Its pathetic really, the world faces bigger problems yet somehow people find the time to complain about something like this?

These are global, money making companies in industrys still in their early years.
Toyota have been making cars for decades with production processes heralded across manufacturing, but they are not beyond the odd recall now and again.

What your blog highlights is that consumers invest more than money in banks, Facebook or any other company. People invest trust and should make informed decisions about who they spend their time, data and information with - remember the banks who left unshreadded customer data on the streets a few years ago?


The Cloud is a much more cost effective way of storing data (you don't keep all your cash under the mattress) but remember that you need to get to it occassionally.

In response to @Tacboy....

Well, its not really pathetic.. some people use their Gmail accounts as online storage... One of Gmail's features being that you never really have to delete emails. I am one of these people.. I use it as a filing cabinet. Maybe important documents stored online because I try to keep as paper free as possible and store as little as I can on my machine.. because they can be stolen or fail.
If my account was wiped I would be devastated... it would be a blow to my business. So I completely understand the 150,000 users who have discovered this has happened to their accounts.
Yes, this isn't as pressing as world-changing matters but it is important nonetheless.

Single point of failure are unavoidable when you talk about a break in like that. Ultimately mobile networks connect to the wired internet... Fibers under the ground popping out in exchange buildings... redundancy is built in but that can work to a point.

You wrote:

"Well a theft may be a very rare occurrence, but you would nevertheless think that a business the size of Vodafone had plans in place to deal with just such an event."

This is hardly en expert point of view... Following your example, I would be forgiven to say then that I would think a technology expert like yourself should know more about the telecoms infrastructure...

The overreaction to Google's issues has been extraordinary. They promised quite early on that all data would be restored and yet there were reams of people stating they had had enough of Google and were switching to other email services.

Problems happen. If 99.98% of public transport ran on time people would be heralding the services for their incredible reliability.

It does however, highlight the vast gap in IT training regarding backups and realistic service provision. Maybe Google et al could look at making local backup of Gmail mandatory part of their (completely free of charge!) service?

0.01% of 200 million is a large number, but it wasn鈥檛 and it鈥檚 the transition into a large number and a failure to increase testing and improve development environment to match live environment that courses these issues. All big sites, Facebook included suffer from this. Though it should be said, their uptime and backup processes are actually very good.

As for Vodafone, the saying in redundancy goes, 鈥測ou鈥檙e only as good as your last failure鈥�. So when was the last time someone broke into their data centre and coursed physical damage?

@1. No it's not a "tragedy" I agree, but I'd be pretty annoyed and anxious if I didn't have access to my emails, there are loads of useful docs/notes/reminders that I keep in my inbox.

I think on consideration, so would most people! It's also not imperative that you commented about it, there are indeed other things you could have been doing that would help save the world!

I think its worth pointing out, in relation to Tacboy's earlier post, that in the 21st century, emails can have a remarkable amount of importance. As a gmail user - luckily not one affected - I'm currently waiting to hear back from 12 job applications. All were applied for digitally, and correspondence has been via email rather than telephone and letter. It's more cost-effective, less time-consuming and all-round better for all involved (when it works).

No, this isn't a tragedy. Yes, the world "faces bigger problems." But as our world's means of communication changes, our infrastructure changes along with it. In the wake of such problems - however marginal they seem - it is right for those in the field (and those with an interest) to debate the failsafe mechanisms these third parties have in place.

In response to Charlie.

"I use it as a filing cabinet. Maybe important documents stored online because I try to keep as paper free as possible and store as little as I can on my machine.. because they can be stolen or fail."

You mean fail just as google did yesterday, what if google had a more serious problem that caused the loss of e-mails for each of those accounts and more importantly your "filing cabinet" from the sounds of it, you would be stuffed.

If you are really that concerned about your business documents, why not get an external hard drive and just back stuff up to that as and when? The idea of having redudancy is that in the case of problems you dont lose everything and it is always good practice to have back ups of back ups. Having one single point of redundancy, for a business completely defeats the purpose of having redundancy.

If you prefer not to have the equipment if your house, try a service like DropBox. You simply do work and upload it to here, you then open work from here or even on your PC and providing everything is setup correctly, when you make and save changes to the items, they are automatically replicated to the online store.

0.02%, [that's only 1 in 5,000 of Gmail customers] had a morning without their inbox. Gmail aren't even close to being the brand leader for email services so this actually affected a tiny minority of people in any way at all.

It is a non-story, blown out of proportion by a media that is so desperate to produce content itself that it seizes on such inconsequences as "news".

I don't think the Google data loss sounds minor at all when the incident is described in concrete terms. Saying that it affected "about 0.02%" of Google's customers is a bit ambiguous, but you get a much better feel for the problem when you realise that 150,000 accounts were wiped clean, with the result that some lost messages going back years. A loss like that could seriously damage a business, and the fact that Google could not quickly reverse the damage seriously undermines efforts to push information online. Bottom line: you need to keep local or distributed backups of everything.

I used Nectar points to buy Amazon gift vouchers - I ordered and sent gifts to friends so the delivery address and my credit card info are in my amazon account. 2 days later I received an auto order email from Amazon informing me that 3 i-phones bought on my credit card to be delivered to a tutu ward in Newham with phone number+expedited delivery. (IT WAS NOT MY ORDER)
I cancelled orders, credit card and closed my amazon account. I phoned police in Newham and gave them all the details- nothing done. I told amazon and they said it was my fault- and they could not tell me anything about the attempted hacking. BUT John Prescott has his mobile hacked and it is a police enquiry and front page news- even the new cat is front page news but when someone tries to steal from a different IP address compromising credit cards and other addresses it is of no importance -the Internet is a criminals dream- they can stay at home and
nick stuff without care or worry about being caught, unless of course you try to rob a politician - bit of an irony really!

@ Charlie

If you haven't got your Gmail account locally backed up then you deserve everything bad that might happen.

Google themselves tell you how to back up your Gmail data locally so there really is no excuse.

Have people not heard of backups yet! If you have important data whether on your home PCs or stored online in systems like Gmail, Facebook etc and you don't want to lose it then, at minimum, keep a duplicate copy somewhere. Better still start creating proper backups to CD/DVD/Tape/Disk etc.

It's a good job we are not paying 1p oer e-mail. Some would have lost a fortune if they had to re-send their messages! Keep smiling

Vodafone said "...this incident involved the physical theft of equipment which is a very rare occurrence."

Well that is as may be - but what if a plane had crashed into the site? Do they not have replica network hubs elsewhere to cover the implications of that hub, for whatever reason, being unavailable?

It's caused a lot of problems for a lot of people and I put it to Vodafone that they need to be releasing an official apology and a statement on what they intend to do to prevent this happening again.

@No.3 (Charlie)

If you're putting all your eggs into one basket by only having one copy of important documents then you're a fool.

I have read elsewhere that if the Vodafone was not theft then it may be protest at the Egyptian mobile and internet service withdrawal during the recent demonstrations.

Alternatively it could be an effort by persons unknown to destroy Hackgate evidence.

Just as a point here, I don't know who informed you that Vodafone's services were up at lunchtime, because they certainly were not for me! I couldn't use my Mobile Broadband, nor my mobile until around 8pm at night. And I live half an hour away from Basingstoke.

Vodafone seem to take your money and not spend it on a secure infrastructure. Even if the equipment is physically damaged the service should automatically swap to another site. Mirrored sites.
Poor management. To busy trying to invest in other providers rather than invest in robust infrastructure.

Huh?

20+ years as a programmer here, currently working for a huge data replication company.

My professional opinion: Bugs happen. Deal with it.

Hope this helps.

Wish someone would delete all my emails,
I have 1846 now.
Don't have the time to sit there and delete them all, and there isnt A "DELETE ALL" button.

The point is Google as a company are very covert in admitting their short comings when it comes to hacks, to manage their stock prices. They actually do have loads more bugs than Microsoft.
Microsoft are more open than any other technology company about bugs that hit the company, and so i respect and trust them. Google & Apple will soom implode...watch this space

@Charlie

1. Gmail shouldn't be used for general document storage. Google offer other services for that.

2. How much is your data worth? Or a better question: how much would it cost to replace your data?

3. They do offer business accounts with proper support. Free Gmail is great for home users exchanging the odd photo and message, but it should not be used for anything important.

Don't trust Google or any other business to look after your data. It's your data, make sure you have a back-up.

9 Disturbedguy said:

" ... use dropbox ... "

Really? And what is dropbox's uptime? Is it better than google's?

@5 Wendell, @3 Charlie & @9 Disturdedguy

'Maybe Google et al could look at making local backup of Gmail mandatory part of their (completely free of charge!) service?'

Google already provide this service it's called 'offline' mode found under Settings > Offline this backs up a cache of your emails locally.

#1 "'people woke up to find their mailboxes empty' - Wow what a tragedy."
Not exactly a tragedy, it's true - but there are dependencies; Google (Android) Market & Google Checkout can be tied to a GMail account. So your phone and business are also out of action - along with email. That's a bit more than a minor inconvenience.
I suspect that a few people are going to spread their load a little wider now.

The question of whether we pay a mechanic or fix our own car, drive ourselves or have a driver... might not be a real choice. Most of us have to reply on the legal agreements between ourselves and professionals.

When things go wrong it is not always a comfort to know that such legal redress is available.

That's why Google (for one, Amazon is another) empowers us all to take ownership of the data it manages on our behalf.

For those of us who has chosen to store our digital lives on Google's servers, you will be interested to see this site by Google:



While giants such as Amazon and Google do try to protect us from power outages, equipment failure and even natural disasters, this only lengthens the time it takes for something truly massive to affect us.

We are ultimately partially if not fully responsible for our own data.

Matt

@23 (Mutlipack_can13)

I can't believe your email system doesn't have a mark all/delete all. Also, in both my work and personal email systems you can highlight all and then press delete !

Mutlipack_can13 wrote:
Wish someone would delete all my emails,
I have 1846 now.
Don't have the time to sit there and delete them all, and there isnt A "DELETE ALL" button.

if you're using a Windows system, you should be able to do this by highlighting the received mail window and then use "Control A" to select all messages then hit "Delete" and they will be gone....alternatively, keep using GMail :-)

Andrew wrote:
9 Disturbedguy said:

" ... use dropbox ... "

Really? And what is dropbox's uptime? Is it better than google's?

--------------------------

The point about Dropbox isn't uptime, it's that you have a local copy. Both act as redundancy for the other.

Dropbox (or similar) + another backup on a different hard drive locally = more than enough protection for almost all scenarios.

Rory Cellan-Jones.

"Vodafone's problems started in the early hours when a gang reportedly armed with sledgehammers bludgeoned their way into one of the company's facilities in Basingstoke in Hampshire."

how can they leave essential infrastructure unprotected? we've had years of terrorism this and al qaeda that, and they can't even put a night watchman on the premises? (poor Vodaphone probably can't afford to pay someone a living wage in spite of pre-tax profits of 拢8.7bn last year)
how irresponsible of Vodaphone.

Ironically, when as part of a landlord inspection of Racal's (as Vodaphone then were) Hook factory in the '80s when mobile phone technology was in it's infancy, we were told that we couldn't enter cetain parts of the factory because they said they were keeping back up bank computer records. As if a bank would allow third parties to keep their records.
What a shame they haven't kept proper back up of their own sytem.

All those people complaining about redundancy in Vodafone's network - what about yourselves?

If mobile calling is that important to you then carry a prepaid SIM for a different network in your wallet.

Its called redundancy.

Let's remember that vodafone are accused of avoiding 拢6Billion in tax - and if they hadn't been allowed to do that, the government wouldn't have to make such severe cuts in areas such as crime prevention.

This comment was removed because the moderators found it broke the house rules. Explain.

MyVoiceinYrHead #2.

"The Cloud is a much more cost effective way of storing data.."

cost effective perhaps (but even that is a moot point given that we can buy TB storage for tens of pounds), the problem with 'the cloud' is that you cannot control access to your data. even fully encrypted data remains vulnerable.

I'm with SoxSexSax. Bugs happen. Downtime happens.

Take backups, virus check... and if the worst still happens, comfort yourself that some poor sod who is trying to fix the damn thing is more unhappy than you....

My Google Apps email is copied to my laptop, whence it is backed up at once to a local fileserver, and once a day to a remote third party backup host. And I can switch the MX records on my domains so they point to a different mailbox, if it comes to that. If you have important emails you want to keep, you might want to do the same.

I know that probably one day, all these things will break at once. And on that happy email-less day I intend to say 'Sod it', go to the beach and have an ice cream.

I understand there are services like dropbox... and I do have a google business account.. but the 'cloud' is a brilliant resource.

I suppose nothing truly is 100% dependable... Gmail can be wiped, hard drives can fail, paper can burn, CD-R's can scratch.

The Google offline feature is something I've used before.. but 10+GB of emails can take its toll.

Vodafone were a bit slow to act. I don't think they anticipated a physical theft - lesson learn't. Google did the right thing by storing data on removable disks hence the slow speed in restoring accounts. I have my data on stored on external hard drives, CD's and USB sticks

@40 Charlie

Surely there's no need to backup all 10+GB of emails, with offline mode in Gmail you can pick and choose which parts to cache locally.

If you use labels you can create one to store all of your important emails, or alternatively use the 'Starred' functionality and just backup that label.

I have more than 13,000 emails on gmail but I only backup the important ones locally.

All pretty simple really and no worrying about lost data.

For those talking about backup systems, data centres with critical workloads, that lie on flight paths, often do need complete backup redundancy centres in case a plane crashes on the orginal. IBM Warwick is one such, because it lies on the approach to Birmingham Airport - with replicated systems in place at Reading.

But IBM Warwick contains literally hundreds of IBM and Sun mainframes, each one costing between three quarters of a million, and a million pounds each: contingency systems don't come cheap, and since no one has ever crashed an airplane on IBM Warwick, yet, that represents four decades-worth of A) 'seriously not cheap' B) 'never been needed, yet'.

People don't attack a data centre with sledgehammers in hopes of making of with a few Dell Optiplexes: as a general rule of thumb, if it's worth the Russian Mafia smashing a hole in a wall, to get at it, you probably can't afford it.

On the whole tdisaster recovery is best measured by how quickly you return to normal operations, after the event - not how many different and bizzare eventualities you can completely prevent happening in the first place. Half a day's not bad going, in my book.

The main reason these things become stories, is that everyone is online, the moment it happens, complaining on web forums, or via email, about how seriously this disconnecton is impacting their efficiency!

Matt Farey #29.

excellent link, thank you.

Kit Green #18.

"..it could be an effort by persons unknown to destroy Hackgate evidence."

is there any information linking the Basingstoke site with the relevant data??



bdyke04 #24.

"The point is Google as a company are very covert in admitting their short comings when it comes to hacks, to manage their stock prices."

well, belief vs evidence (a bit like our government :-)). I offer you the following as evidence that Google are very responsible in unearthing bugs in their products:

"Google's rewards programme pays discoverers of vulnerabilities up to $1,000. Google paid out a total of $14,000 for this particular [Chrome 9] update. In total, its security bug bounty programme has now paid out more than $100,000."


"They actually do have loads more bugs than Microsoft. Microsoft are more open than any other technology company about bugs that hit the company, and so i respect and trust them."

respect and trust M$ all you will, but how about backing up your assertions with some kind of evidence?

Redundancy for components, servers and networks are good, but only if they are accompanied by effective automated failover mechanisms. Understandably there might be a capacity and performance issues for a while until full service can be resumed, but there is no excuse for an interruption to such a critical service.

As for Gmail, it's free - and you get what you pay for.

Google has never been successful in general technology except search which is 90% software based and for some reason they don't want to admit it, they have got to stop trying.

#47 "Google has never been successful in general technology except search"
NEVER ?????? do you reallly think that ?

What have Google ever done for us . . . . (I feel a Monty Python esque list coming on)

I'l start off with ANDROID OS

A friend of mine who worked at that Vodafone location in 2003/4 said that the place was then already being turned into a totally unmanned site. He reckons that, the rate at which funding was being withdrawn in a cost cutting exercise, it would not surprise him if the place now looked externally derelict whilst still being full of cold clean air, flashing LEDs and a low hum of server fans. So, an obvious target.

Having worked in telecommunications and commerce for 27 years - that sounds about right. Should they have had better security and people on site? Yes - but that costs money.

Rory Cellan-Jones.

"Some Gmail users - about 0.02% of them according to Google - woke up on Monday to find their inboxes empty and their mail data apparently eradicated."

what I find funny (not!!) is that my account was not affected until Monday late afternoon, since which time I have had no Gmail service at all.

no mention of total loss of service in the news, curiouser and curiouser... ;-(

'' there is no excuse for an interruption to such a critical service.''

What a sad statement.

Shows the arrogance and impatience of some people.
What is so 'critical' about a mobile phone service ?

What did Harper Lee say about bokks ?

seapeelie wrote: '' there is no excuse for an interruption to such a critical service.''

What a sad statement.

Shows the arrogance and impatience of some people.
What is so 'critical' about a mobile phone service ?
---------------------------------------
I'll give you some examples:

- Use by the NHS to contact surgeons/doctors etc in emergencies out of hours
- Companies' IT staff who may be automatically alerted overnight when there is a computer failure (maybe your bank's cash machines, or your electricity supply)
- A vital lifeline for the elderly: some emergency and panic systems use mobile phone networks.

Today, mobile phone networks are used for much more than inane texting and organising social arrangements. It's ignorant to think otherwise.

If I might temporarily lose access to my gmail for a day or so once every couple of years, this is still more reliable than having my email in a file on my home PC. How many times have I lost files, had a hard drive fail or replaced the PC?

That's without considering any of the other benefits of letting gmail do the whole job, speed and simplicity amongst them.

Neither Vodaphone or any mobile provider should ever be trusted. It took the EU to threaten them before any reduced their prices. UK prices still remain amongst the highest in the world.

Mobile phone providers and Vodaphone as the biggest cannot deliver effective customer service or cost competition to anyone.Theey are all the same so it is pointless changing as what starts out cheaper results in you being screwed in the end. A policy that maximises profits and bonuses for under performing over paid directors.

For Google most Gmail accounts are free. So as users we should be reasonable in our expectations. These are different business models.That Google got the mails back while vodaphone has failed to apologise or refund affected customers speaks volumes for the competence and customer focus of these businesses.

seapeelie #51.

"What is so 'critical' about a mobile phone service ?"

@48 Android was a flop and a failure they have less then 10% of the general market when most others hold upwards of 50% google has not been able to pull any thing PROFITABLE other then search.

I am ready to go through one by one.

ratilumanbaum #56.

"..Android was a flop and a failure they have less then 10% of the general market.."





"I am ready to go through one by one."

please do, I feel we all can do with some light entertainment..

Rory
from your comments it's obvious that you don't have a good enough understanding of the concept of risk and risk management strategy. It's a large part of what I do for a living as an Information Assurance professional. The objective is to balance risk and cost to acceptable levels. One cannot eliminate risk, or forsee every eventuality. All one can normally do is minimise the impacts and the periods of disruption. To do more costs more budget than is available to anyone except the nuclear industry.

writing as someone who has worked in the telecomms industry for over 20 years, all these comments about Vodafone reveal one thing to me, and that is how little people know about the complexities of a telephony network. If is was as "simple" as having a back-up ststem then Vodafone would have it, as would every other network operator, whether they are providing a mobile or a fixed-line service.

Network resilliancy is based on a partial system failre, full ststem failures are rare these days, however what was stolen from the site in Basingstoke was very specific and would have had exactly the same impact on any other network operator.

A 10 Gbit/s telephone fibre can carry just under 125,000 voice circuits. Imagine that number of cars heading along a motorway and, for some reason the motorway is blocked, how long is it before all the roads within a considerable area around the accident are completely grid-locked? A similar thing happens when a major incident on a telephone network occurs.

The only way to resolve that would be to build a totally seperate parallel network to switch traffic over in case of a failure, however that would cost many millions to do. But for that to happen how many would be prepared to pay twice, three, or even four times what they pay now for their phone contracts and calls?

Any takers? No, I thought not.