European cloud services run by Amazon and Microsoft were disrupted over the weekend when a lightning strike knocked out power at a Dublin data centre.
According to a service update posted on its website, Amazon’s EC2 and Relational Database Service were knocked offline. "We understand at this point that a lightning strike hit a transformer from a utility provider to one of our Availability Zones in Dublin, sparking an explosion and fire," a service update read.
Usually in this sort of situation power would be switched to a backup generator, however, "the transient electric deviation caused by the explosion was large enough that it propagated to a portion of the phase control system that synchronises the backup generator plant, disabling some of them," the statement continued.
The generators needed to be manually synchronised before being brought online, Amazon said.
Amazon started to restore EC2 services within a few hours and claimed to have 60% back up and running within 12 hours of the incident. However cloud services across Europe were still facing problems this morning due to the sheer size of the disruption.
"Due to the scale of the power disruption, a large number of EBS servers lost power and require manual operations before volumes can be restored," Amazon said. "Restoring these volumes requires that we make an extra copy of all data, which has consumed most spare capacity and slowed our recovery process."
"While many volumes will be restored over the next several hours, we anticipate that it will take 24-48 hours until the process is completed. In some cases EC2 instances or EBS servers lost power before writes to their volumes were completely consistent. Because of this, in some cases we will provide customers with a recovery snapshot instead of restoring their volume so they can validate the health of their volumes before returning them to service," the company said.
This is the second cloud outage for Amazon in recent months. In April this year a configuration error during a network upgrade took out services for a couple of days.
Microsoft Business Productivity Online Standard Suite (BPOS) was also knocked out by the Dublin lightning strike, but according to a Twitter update in the early hours of Monday morning the service was back up for all EMEA customers.
