A widespread T-Mobile outage across the carrier’s US network on Monday triggered a frenzied firestorm of conspiracy theories, amid unverified and inaccurate claims that it had fallen victim to a huge Distributed Denial of Service (DDoS) attack on service providers across the United States.
Tens of millions of T-Mobile’s wireless customers had their services disrupted for up to eight hours as the company tried to identify the problem and restore its voice and text services (it insists data services continued to work).
With an “anon” Twitter account pushing out a map to millions of followers from security firm Arbor Networks purportedly showing a large DDoS attack “on the US”, hysteria ran riot and “DDoS” trended on Twitter.
The claim was rapidly picked up by no shortage of blue tick-verified, high-profile users, including across the gaming community; even senators waded in. A sample, widely shared Tweet: “This DDoS attack is serious. It has taken down Instagram, Facebook, T-Mobile, Verizon, and Twitch…. 2020 is something else.”
Cloudflare CEO Matthew Prince was among those trying to talk reason: “Arbor Networks attack map [..] looks terrifying today! Thing is, it always looks terrifying. It’s a marketing gimmick put up to sell DDoS mitigation services.
T-Mobile Outage Cause: A Leased Fiber Circuit Failure
Here’s what actually happened: a fiber circuit failed.
Under pressure to provide a rapid root cause analysis, T-Mobile’s President of Technology, Neville Ray revealed the truth late on Tuesday: “The trigger event is known to be a leased fiber circuit failure from a third party provider in the Southeast. This is something that happens on every mobile network, so we’ve worked with our vendors to build redundancy and resiliency to make sure that these types of circuit failures don’t affect customers.
“This redundancy failed us and resulted in an overload situation that was then compounded by other factors. This overload resulted in an IP traffic storm that spread from the Southeast to create significant capacity issues across the IMS (IP multimedia Subsystem) core network that supports VoLTE calls.
He added: “We have worked with our IMS (IP Multimedia Subsystem) and IP vendors to add permanent additional safeguards to prevent this from happening again and we’re continuing to work on determining the cause of the initial overload failure. So, I want to personally apologize for any inconvenience that we created yesterday and thank you for your patience as we worked through the situation toward resolution.”
The outage comes at an unfortunate time for T-Mobile — and has triggered an investigation by the FCC: it comes less than 12 weeks after T-Mobile US Inc. and Sprint Corp. merged to create a colossus with approximately 140 million customers and revenues of over $70 billion; the company is in the spotlight.
The combined entity is hoping to aggressively build out its 5G infrastructure (T-Mobile plans $40 billion in spending over the next three years), and the last thing executives will have wanted is an FCC investigation and millions of angry customers; reacting promptly to squash the DDoS rumours was clearly the right call, however painful it was to admit the failure of failover mechanisms,
I want to be fully transparent about what happened yesterday with our network. We did not meet our own bar for excellence. We have taken the necessary steps to avoid reoccurrence and truly apologize for any inconvenience we created. https://t.co/sDXZemXRsK
— Neville (@NevilleRay) June 17, 2020
DDoS attacks, meanwhile (in which servers are overloaded with malicious traffic) do howeverappear to be on the rise again, after dropping in scale in recent years. Last week we reported on an (unsuccessful) record 2.3 Tbps DDoS attack on AWS. Security firm Akamai this week told Computer Business Review it had blocked the largest DDoS attack it had seen on an ISP: a 1.44 Tbps attack launched using nine different forms of DDoS attack vectors.
We’ll be bringing you more details on that shortly.
See also: Record DDoS Attack Hits AWS: 2.3 Tbps Assault Lasted Days
