Google Cloud users with data stored in the company’s US-East region may experience problems today, as its engineers continue to battle a problem which has been slowing its servers for the past 18 hours.
The issue is described as a “cloud load balancing” problem by the company’s engineers, and has yet to be solved despite having first been reported in the early hours of Sunday. It is mainly impacting South Carolina (us-east1) and Northern Virginia (us-east4) data centres, with other nearby servers also affected.
Google Cloud’s problems with US-East servers
First reported at 5.56am US pacific time (PT) on Sunday, Google’s team said a “Cloud L2 overload in us-east4 is causing harm in us-east4 and nearby regions”. The impact of the fault is “higher latency and possibly elevated errors on customers’ global L7 Load Balancers during US peak hours” according to the hyperscaler’s engineers. Load balancers optimise the performance of cloud services.
The problems are likely to be worst in “US peak hours”, starting at 5am PT (noon in the UK), and as of the most recent update, the issue is no closer to being solved. “Our engineering team continues to work on the mitigation of the issue, we do not have an ETA for mitigation at this point,” a status update posted at 9.30pm PT on Sunday stated.
Google Cloud has offered brief advice to users of its platform worried about the problem. Customers who are able to should “move their backends to regions away from us-east,” it said. “Customer moving workloads to us-west, europe and asia – may see improvements. No workarounds at this time for customers who are required to host in us-east regions.”
Outages rare on hyperscale platforms
Google Cloud is the third-largest player in the public cloud market, holding 11% of business according to data from Synergy Research Group.
Though it trails behind rivals, market leader Amazon's AWS and Microsoft Azure, it is relied upon by many big businesses. Customers listed on its website include Deutsche Bank, Airbus and Paypal.
Problems with the major cloud platforms are few and far between, though when their platforms do suffer a failure or outage the consequences can be significant for the economy. This has led regulators around the world to raise questions about whether steps should be taken to mitigate the high concentration of cloud computing services in the hands of a few providers.
Last month, the US Treasury warned financial services companies to ensure they had plans in place in case cloud servers failed.
Deputy secretary of the treasury Wally Adeyemo said that though "there is no question that providing consumers with secure and reliable financial services means greater demand for cloud-based technologies,” his department believes there must be "safe and effective migration" as more banks and other financial companies adopt cloud services.
