Google’s Gmail service failed on February 28th after a software update was deployed across their infrastructure. As a result millions of users found their Gmail accounts empty, and while Google had tape backups, it still lost some e-mail after the software update introduced a bug.
In March 2010, the company promised that Google Apps customers wouldn’t need to worry about disaster striking and used e-mail as a prime example. So what happened?”Well, in some rare instances software bugs can affect several copies of the data. That’s what happened here. Some copies of mail were deleted, and we’ve been hard at work over the last 30 hours getting it back for the people affected by this issue,” said Ben Treynor, Google’s vice president of engineering and site reliability czar.
So despite Google splitting all cloud based data across two data centres, it was still possible for a software bug to disable the service and irrecoverably delete user data.
The lesson from this incident? - that whist cloud based systems are likely to be more resilient than most, even the best systems can fail and just because your application is in the ‘Cloud’ doesn’t mean it’s definitely safe.

