View all newsletters
Receive our newsletter - data, insights and analysis delivered to you

T-Mobile Outage Whipped Conspiracy Theorists Into a Frenzy — Here’s What Really Happened

There have been some huge recent DDoS attacks: this wasn't one...

By CBR Staff Writer

A widespread T-Mobile outage across the carrier’s US network on Monday triggered a frenzied firestorm of conspiracy theories, amid unverified and inaccurate claims that it had fallen victim to a huge Distributed Denial of Service (DDoS) attack on service providers across the United States.

Tens of millions of T-Mobile’s wireless customers had their services disrupted for up to eight hours as the company tried to identify the problem and restore its voice and text services (it insists data services continued to work).

With an “anon” Twitter account pushing out a map to millions of followers from security firm Arbor Networks purportedly showing a large DDoS attack “on the US”, hysteria ran riot and “DDoS” trended on Twitter.

The claim was rapidly picked up by no shortage of blue tick-verified, high-profile users, including across the gaming community; even senators waded in. A sample, widely shared Tweet: “This DDoS attack is serious. It has taken down Instagram, Facebook, T-Mobile, Verizon, and Twitch…. 2020 is something else.”

Cloudflare CEO Matthew Prince was among those trying to talk reason: “Arbor Networks attack map [..] looks terrifying today! Thing is, it always looks terrifying. It’s a marketing gimmick put up to sell DDoS mitigation services.

T-Mobile Outage Cause: A Leased Fiber Circuit Failure

Here’s what actually happened: a fiber circuit failed.

Under pressure to provide a rapid root cause analysis, T-Mobile’s President of Technology, Neville Ray revealed the truth late on Tuesday: “The trigger event is known to be a leased fiber circuit failure from a third party provider in the Southeast. This is something that happens on every mobile network, so we’ve worked with our vendors to build redundancy and resiliency to make sure that these types of circuit failures don’t affect customers.

Content from our partners
How hackers’ tactics are evolving in an increasingly complex landscape
Green for go: Transforming trade in the UK
Manufacturers are switching to personalised customer experience amid fierce competition

“This redundancy failed us and resulted in an overload situation that was then compounded by other factors. This overload resulted in an IP traffic storm that spread from the Southeast to create significant capacity issues across the IMS (IP multimedia Subsystem) core network that supports VoLTE calls.

He added: “We have worked with our IMS (IP Multimedia Subsystem) and IP vendors to add permanent additional safeguards to prevent this from happening again and we’re continuing to work on determining the cause of the initial overload failure. So, I want to personally apologize for any inconvenience that we created yesterday and thank you for your patience as we worked through the situation toward resolution.”

The outage comes at an unfortunate time for T-Mobile — and has triggered an investigation by the FCC: it comes less than 12 weeks after T-Mobile US Inc. and Sprint Corp. merged to create a colossus with approximately 140 million customers and revenues of over $70 billion; the company is in the spotlight.

The combined entity is hoping to aggressively build out its 5G infrastructure (T-Mobile plans $40 billion in spending over the next three years), and the last thing executives will have wanted is an FCC investigation and millions of angry customers; reacting promptly to squash the DDoS rumours was clearly the right call, however painful it was to admit the failure of failover mechanisms,

DDoS attacks, meanwhile (in which servers are overloaded with malicious traffic) do howeverappear to be on the rise again, after dropping in scale in recent years. Last week we reported on an (unsuccessful) record 2.3 Tbps DDoS attack on AWS. Security firm Akamai this week told Computer Business Review it had blocked the largest DDoS attack it had seen on an ISP: a 1.44 Tbps attack launched using nine different forms of DDoS attack vectors.

We’ll be bringing you more details on that shortly.

See also: Record DDoS Attack Hits AWS: 2.3 Tbps Assault Lasted Days

 

Websites in our network
Select and enter your corporate email address Tech Monitor's research, insight and analysis examines the frontiers of digital transformation to help tech leaders navigate the future. Our Changelog newsletter delivers our best work to your inbox every week.
  • CIO
  • CTO
  • CISO
  • CSO
  • CFO
  • CDO
  • CEO
  • Architect Founder
  • MD
  • Director
  • Manager
  • Other
Visit our privacy policy for more information about our services, how New Statesman Media Group may use, process and share your personal data, including information on your rights in respect of your personal data and how you can unsubscribe from future marketing communications. Our services are intended for corporate subscribers and you warrant that the email address submitted is your corporate email address.
THANK YOU