View all newsletters
Receive our newsletter - data, insights and analysis delivered to you
  1. Technology
  2. Data Centre
September 2, 2020updated 12 Jul 2022 10:00am

CenturyLink Took Down 3.5% of the Internet Trying to Block a Single IP Address

"The Operations Engineering Team was using this process during routine operations..."

By CBR Staff Writer

When CenturyLink, one of the world’s biggest internet backbone providers, faced a mystery issue on August 31 at 10:04 GMT, it proceeded to take down 3.5% of the world’s traffic, triggering outages at some of the world’s most popular websites including the PlayStation Network and Xbox Live. The event wasn’t fully cleared until 15:10 GMT.

Now, thanks to a swift RFO (reason for outage) shared with customers and seen by Computer Business Review, the precise reason for the outage has become clear and it was not, as we had initially speculated, the result of a botched attempt to mitigate a Distributed Denial of Service attack. Rather, it was down to a failed attempt by the company to “block a single IP address” on behalf of a customer, it has admitted.

(CenturyLink had earlier acknowledged in a status update that the substantial outage was due to an “offending flowspec announcement” Flowspec is a protocol used to mitigate sudden spikes of traffic, for example the effects of a DDoS attack).

Flowspec: “Problematic Protocol”

Here’s how CenturyLink explained it to customers.

“As a large influx of traffic is identified from a set IP address, the Operations Engineering Team utilizes Flowspec announcements as one of many tools available to block the corrupt source from sending traffic to the CenturyLink network.

“The Operations Engineering Team was using this process during routine operations to block a single IP address on a customer’s behalf as part of our normal product offering. When the user attempted to block the address, a fault between the user interface and the network equipment caused the command to be received with wildcards instead of specific numbers. This caused the network to recognize the block as several IP  addresses, instead of a single IP as intended. The user interface for command entry is designed to prohibit wildcard entries, blank entries, and only accept IP address entries.

The company continued: “A secondary filter that is designed to prevent multiple IP addresses from being blocked in this fashion failed to recognize the command as several IP addresses. The filter specifically looks for destination prefixes, but the presence of the wildcards caused the filter to interpret the command as a single IP address instead of many, thus allowing it to pass… the problematic protocol [then] propagated through many of the edge devices on the CenturyLink Network.”

What Now?

The RFO paints a vivid picture of the brown stuff hitting the fan when the issue was noticed: “The IP Network Operations Center (NOC) was engaged, and due to the amount of alarms present, additional resources were immediately engaged including Tier III technical Support, Operations Engineering, as well as Service Assurance Leadership. Extensive evaluations were conducted to identify the source of the trouble.”

Content from our partners
Why food manufacturers must pursue greater visibility and agility
How to define an empowered chief data officer
Financial management can be onerous for CFOs, but new tech is helping lighten the load

The Flowspec announcement platform has been disabled from service on the CenturyLink Network in its entirety, CenturyLink said. It will “remain offline until extensive testing is conducted” while the secondary feature is being “modified to prohibit wildcard entries.” Yes, the internet is held together with duct tape.

Websites in our network
Select and enter your corporate email address Tech Monitor's research, insight and analysis examines the frontiers of digital transformation to help tech leaders navigate the future. Our Changelog newsletter delivers our best work to your inbox every week.
  • CIO
  • CTO
  • CISO
  • CSO
  • CFO
  • CDO
  • CEO
  • Architect Founder
  • MD
  • Director
  • Manager
  • Other
Visit our privacy Policy for more information about our services, how New Statesman Media Group may use, process and share your personal data, including information on your rights in respect of your personal data and how you can unsubscribe from future marketing communications.
THANK YOU