View all newsletters
Receive our newsletter - data, insights and analysis delivered to you
  1. Technology
  2. Cybersecurity
December 30, 2018updated 02 Jan 2019 8:42am

CenturyLink Blames Faulty Network Management Card for Outages

"A decision was made to isolate a device in San Antonio from the network as it seemed to be broadcasting traffic and consuming capacity"

By CBR Staff Writer

The US’s CenturyLink has blamed a faulty network management card for a crisis that knocked out internet services for users nationwide late last week – taking down 911 call centres for numerous police forces.

Yet a letter to core customers that was leaked to reporter Brian Krebs, CenturyLink – the second largest US communications provider to global enterprise customers – offered an explanation that raised new questions, not least why mission critical systems/networks were so vulnerable to a single point of failure.

CenturyLink Internet Outage: Device “Broadcasting Traffic and Consuming Capacity”

Describing the root cause as “a network management card in Denver propagating invalid frame packets across devices” the company said it had to pull the card out, remove “secondary communication channel tunnels between specific devices”, and apply a polling filter “to adjust the way the packers were received in the equipment.”

CenturyLink internet

The rest of the explanation (the company has yet to make an actual public statement on the outage, despite the potentially catastrophic outcome for emergency services) left observers suggesting everything from the cascading effect of faulty hardware, to a serious malware issue in the company’s infrastructure.

In a cascading effect, the initial CenturyLink internet outage seems to have knocked out regional equipment that was unable to be reset remotely; the company had to dispatch engineers to sites in Atlanta, Chicago, Kansas City, Los Angeles and New Orleans.

“Tier IV equipment vendor support was engaged as it was determined that the issue was larger than a single site”, the company said.

“During cooperative troubleshooting between the Equipment Vendor and CenturyLink, a decision was made to isolate a device in San Antonio from the network as it seemed to be broadcasting traffic and consuming capacity. This action did alleviate impact, however investigations remained ongoing.”

Content from our partners
Green for go: Transforming trade in the UK
Manufacturers are switching to personalised customer experience amid fierce competition
How many ends in end-to-end service orchestration?

As earlier reported by Computer Business Review, the outage left police forces across the US tweeting and texting residents that 911 services were not working.

The US Federal Communication Commission (FCC) has opened a public investigation into the incident, describing it as “completely unacceptable”, with FCC Chairman Ajit Pai saying that its “breadth and duration are particularly troubling.”

He added: “I’ve directed the Public Safety and Homeland Security Bureau to immediately launch an investigation into the cause and impact of this outage.  This inquiry will include an examination of the effect that CenturyLink’s outage appears to have had on other providers’ 911 services.”

 

Websites in our network
Select and enter your corporate email address Tech Monitor's research, insight and analysis examines the frontiers of digital transformation to help tech leaders navigate the future. Our Changelog newsletter delivers our best work to your inbox every week.
  • CIO
  • CTO
  • CISO
  • CSO
  • CFO
  • CDO
  • CEO
  • Architect Founder
  • MD
  • Director
  • Manager
  • Other
Visit our privacy policy for more information about our services, how New Statesman Media Group may use, process and share your personal data, including information on your rights in respect of your personal data and how you can unsubscribe from future marketing communications. Our services are intended for corporate subscribers and you warrant that the email address submitted is your corporate email address.
THANK YOU