This morning Facbook was the "victim" of a worldwide, hour-long blackout. Lizard Squad, a hacking group who have attacked networks such as Playstation in the past, were quick to claim the glory, or should I say responsibility, of the attack on Twitter.
But was the aggressor really Lizard Squad? Is Facebook’s assertion that it was a glitch the truth behind the blackout? In a firm full of tech talent, did Facebook have no safe guards against a blackout?
With Lizard Squad and Facbook firmly on the front pages, CBR looks to security experts to try and piece together what exactly occurred in the blackout this morning.
1. Lizard Squad or Glitch?
The most important question first. Lizard Squad claim responsibility on Twitter, while Facebook blame a glitch. Amichai Shulman, CTO at Imperva, explains why the nature of the outage means that software glitch seem a more likely scenario than the work of Lizard Squad: "In my opinion, the claim by Lizard Squad is false – it does not seem likely to be the result of DDoS attack."
"I think that a volumetric attack the magnitude that takes down Facebook would take town most of the Internet. Have we seen people taking responsibility for alleged acts in the past? Yes. This is certainly not new and certainly not confined to hacking systems (taking down planes is also a favorite target for claiming responsibilities)."
"We have seen in the past small, overlooked software glitches who took down large networks. These are usually related to a register or counter kept in smaller memory storage than required. Because of the wide spread of this I’d guess it has to do with some register that keeps time that overflowed."
Adam Winn , Product Manager at OPSWAT, seems to concur with Shulman: "Less than a day after Lizard Squad attacked Malayasia Airline’s website, it seems probable that they were also behind Facebook outage. On the other hand, the explanation given by Facebook is completely probable."
Several factors, though circumstantial, lend credence to Facebook’s side of the story:
– The outage was late-night Pacific time. If things were happening in California, 11:10pm is a very reasonable hour for system maintenance. In the past, Facebook has had maintenance from midnight to 2am.
– The outage was global and simultaneous. IE: Only the most well-crafted DDoS would shut down all these affected services, all across the globe, at the same instant. That type of monumental effort would have come with much louder bragging by Lizard Squad.
– The duration of the outage was a very round number. That’s often a strong indicator that the root cause was a configuration with some TTL or other required time interval to propagate and repair.
2. What caused the fault?
Experts are unanimous in their belief that Lizard Squad was not to blame for the fault. So what caused the fault? Mark James, security specialist at ESET, commented: "Both Facebook and Instagram rely on massive amounts of web traffic so the likelihood of a distributed denial of service (DDoS) succeeding would be slim."
"Facebook state it was an internal configuration change – most likely DNS or other network configuration – related that caused the traffic to go awry."
TK Keanini, CTO at Lancope, put it down to human error: "I hope this shows how complicated the process of attribution really is in the Internet age. On one end of the spectrum we have bad actors hiding and evading detection and on the other hand stepping out and claiming responsibility for incidents they were not a part of. Given that Facebook is on record claiming it was in internal glitch I’m going with that story."
"With the massive scale of users, data, and activity on Facebook, I’m surprised that these errors don’t happen more often."
"Facebook hires and retains the top talent in the industry but we are all human and sometimes mistakes happen. This mistake was detected quickly and the outage was limited which is all one can hope for when handling exceptions."
3. Why were other social media platforms impacted by the fault?
Alongside Facebook and Instagram, dating app Tinder, AOL’s Instant Messenger and HipChat also experienced blackouts. ESET’s Mark James explains: "The reason a few of the other social apps went down or caused problems was probably because they often use a Facebook login to authenticate on those systems and thus as Facebook was down, could not validate their credentials."
4. If Lizard Squad didn’t cause the blackout, then why did they take responsibility?
Lizard Squad released tweets revelling in the fact that they were the cause of the blackout, OPSWAT’s Adam Winn puts this down to: "People will continue to take credit for hacks so long as there’s no downside. Somewhere between Pascal’s Wager and catch-22, Lizard Squad takes credit for an apparent attack when the only group who can discredit them would have to do so by revealing their own mistakes."
5. What can organisations learn from Facebook’s blackout today?
Ian Wells, Vice President North West Europe at Veeam Software, comments: "This outage is a valuable lesson to any organisation; no matter what your size and resources, even the smallest mistake when implementing changes can result in painful, and expensive, downtime. Indeed, even if that downtime is only measured in minutes, it could cost the modern, always-on business millions.
"The simple fact is that it shouldn’t have to be this way: in the modern age, there is no reason why changes to IT systems can’t be thoroughly tested before implementation, spotting any potential issues before they affect the production environment. Creating a virtual testing environment should be a simple task, especially with the ease of creating new infrastructure or leveraging unused resources such as backup storage.
"Considering that enterprises on average suffer application downtime 13 times per year, lasting over 90 minutes each time, the fact that this is Facebook’s longest outage since 2010, and lasted less than an hour, gives some comfort. However, for a business with such resources and technology at its disposal, events such as this should be becoming increasingly rare, if not extinct."
