View all newsletters
Receive our newsletter - data, insights and analysis delivered to you
  1. Technology
  2. Software
July 13, 2020

GitHub Outage Impacts Millions of Developers: Are Gaps Between Staging and Production Still an Issue?

Recovery efforts continue

By CBR Staff Writer

Updated 09:35. GitHub says the issue was resolved 09:31 BST. Computer Business Review will update with a post-incident review when we have one. 

A major GitHub outage has developers starting the week with gritted teeth, after the software development platform went down this morning.

GitHub says it is working on the outage, which had lasted for over four hours as we published; beginning 04:06 UTC (03:06 BST).

The incident has raised fresh questions about GitHub’s resilience in the wake of three separate outages in April 2020 alone.

Content from our partners
How businesses can safeguard themselves on the cyber frontline
How hackers’ tactics are evolving in an increasingly complex landscape
Green for go: Transforming trade in the UK

“We are investigating reports of degraded performance and increased error rates,” the platform said early this morning.

The source of “elevated errors” was spotted 6:53am BST. GitHub added at 8.18am BST that it is working on the recovery of our services.

GitHub outage

GitHub attributed April’s three outages respectively to:

a) misconfiguration of software load balancers disrupted internal routing of traffic between applications that serve GitHub.com and the internal services they depend on;

b) misconfiguration of database connections, related to ongoing data partitioning efforts that “made it unexpectedly to production” and

c) a networking configuration that was “inadvertently applied to our production network” (Yikes).

GitHub admitted in April that it had issues with its staging labs environment.

“This staging environment does not set up the databases and database connections the same way as the production environment. This can lead to limited testability of connection changes specific to the production environment. We will be addressing this issue in the coming months”, the company said.

GitHub runs most of its platform on its own bare metal infrastructure, with  networking infrastructure “built around a Clos network topology with each network device sharing routes via Border Gateway Protocol (BGP).”

GitHub, bought by Microsoft in 2018 for $7.5 billion, is home to over 50 million developers. Given the workloads it supports and widespread reliance on it for high availability, the large-scale outages like this can have a major impact.

Owner Microsoft, like many other large infrastructure providers, has also faced the challenge of rapidly scaling up its data centre infrastructure in the wake of surging workloads driven by a swell in remote working staff in the wake of the pandemic, admitting in April that it had faced some supply chain issues after the outbreak.

The COVID-19 pandemic rocked the server hardware supply chain globally as factories around the world shut down just as large enterprises and hyperscalers needed to overhaul data centres. (Dropbox’s CTO said his company’s data center team “proactively swapped out 30,000 components in eight weeks” to safely reduce on-site staffing).

Chipmaker AMD meanwhile said in a Q1 earnings call that one unnamed cloud provider had added 10,000 servers to their data centres in just 10 days during the crisis, in a frantic bid to scale up their infrastructure as workloads soared.

GitHub’s issues however appear to be related more to issues surrounding gaps between its staging/canary environments and production.

See also: Microsoft Feels the Squeeze: Throttles 365 Services, Migration Bandwidth

 

Websites in our network
Select and enter your corporate email address Tech Monitor's research, insight and analysis examines the frontiers of digital transformation to help tech leaders navigate the future. Our Changelog newsletter delivers our best work to your inbox every week.
  • CIO
  • CTO
  • CISO
  • CSO
  • CFO
  • CDO
  • CEO
  • Architect Founder
  • MD
  • Director
  • Manager
  • Other
Visit our privacy policy for more information about our services, how New Statesman Media Group may use, process and share your personal data, including information on your rights in respect of your personal data and how you can unsubscribe from future marketing communications. Our services are intended for corporate subscribers and you warrant that the email address submitted is your corporate email address.
THANK YOU