Software certificate issues have been blamed for a host of recent enterprise ills, including a widespread outage of Ericsson’s infrastructure that left O2 customers with no network for nearly 24 hours (O2 is rumoured to be seeking £100 million in damages).
Equifax meanwhile allowed over 300 security certificates to expire, including 79 used to monitor business critical domains, prior to a data breach that exposed the personal data of over 143 million people, including the records of 15.2 million UK customers.
Getting certificate management wrong can clearly have catastrophic consequences. With this in mind, Computer Business Review posed some questions about these recent issues – and broader digital certificate management – for Tim Callan, a Senior Fellow at Sectigo, the world’s largest commercial Certificate Authority (CA).
What is a Software Certificate, Anyway?
Digital certificates are universally used throughout IT infrastructure to confirm the genuine identity of a software process when it connects to another across any computer network including the internet. These certificates enable a broad variety of circumstances including connecting with web servers (TLS/SSL), software updates (code signing), encrypting email (S/MIME), and communicating with and controlling Internet of Things devices (IoT certificates). Most systems require the presence of a valid certificate before they will enable encrypted communications. That is because encryption in and of itself is not protection if the encrypted information is given to the wrong party.
How it works is that each certificate contains specific information about the identity of the organization using it. This information varies by use case but can include the company name and location, the domain name of web certificates, or other data. Certificates must be issued by a “Certificate Authority” (or CA) that is trusted by systems on that network. For purely internal use cases like enterprise device certificates or IoT networks, the company that owns the devices can be the Certificate Authority, but for use across the public internet (such as web sites, server-to-server connections, or email), certificates must come from a public CA with roots universally trusted by the systems on the internet.
Digital certificates are nearly ubiquitous in contemporary IT systems and are essential to the secure and correct functioning of the digital devices and services we all use every day.
Why do so Many ‘Reputable’ Organisations Let them Lapse?
Certificate management can be tricky. Certificates by their nature must have expiration dates. This expiration ensures that the identity information vouched for by the certificate is current within a certain time window and is an essential part of the secure ecosystem. However, that means administrators must track and renew their certificates to prevent expired certificates from interfering with their systems’ secure operation.
This administration has always been a headache, but the increasing complexity of contemporary IT architectures has caused the number and variety of certificates requiring management to explode. Virtualisation, containerisation, public/private/hybrid cloud, repatriation of workstreams, and “software-defined everything” all contribute to the vast complexity that can constitute an enterprise’s certificate landscape today.
The task is further complicated by the breakdown of centralised IT, embedding traditional IT functions into lines of business and other service functions. Often these embedded technical teams will create their own systems without coordinating certificate requirements with the central IT function. That makes it difficult for even the most diligent network administrators to be sure all critical certificates are accounted for.
What Do You Do When a Certificate Expires?
The exact failure state when an expired certificate is encountered depends on what the involved software is programmed to do. In the case of a public-facing certificate such as a TLS/SSL certificate or email certificate, the end user’s browser or email application will include some kind of visible warning that the certificate is expired, which discourages interaction.
Many applications will require valid certificates to be in place in order to operate. In this case an expired certificate will cause an entire system or sub-system to stop operating. Two recent, high-profile news items show this principle in operation. Last month’s outage for tens of millions of mobile customers using O2, Softbank, and other services ultimately owed itself to a certificate outage, which then caused the failure of the systems that provided data to their mobile devices. In the case of 2017’s massive Equifax data breach, we learned in December that a system set to monitor data exfiltration suffered a certificate expiration and therefore ceased to operate. In this case the outage occurred in this security system, causing the expected security task not to function and enabling the breach.
How Can Enterprises Manage the Multiple Certificates in Play across an IT Estate?
Recent outages such as those affecting O2 and Equifax illustrate the potential severity of allowing such expirations to occur. Enterprises need to ensure their certificates are accounted for and renewed prior to expiration or they risk embarrassment, lost business, poor customer service experiences, and data breaches. Automated certificate monitoring and replacement is essential to protect against unexpected expirations and the problems they can cause. All known certificates can be loaded into an automated system to make administrators aware of upcoming expirations and to make replacement convenient and error-free.
Of course, automation can only benefit certificates that are known to the system’s administrators. To address the problem of unknown certificates, IT departments require certificate discovery. A certificate discovery system crawls the organization’s network and catalogs all certificates it finds. Once these certificates are known, they are able to benefit from monitoring and automated replacement just like any other certificate.
Some IT organizations build certificate automation functionality for themselves, and some employ third-party certificate management platforms to handle these needs for them.
How Could an Expired Software Certificate Cause a Crisis as Substantial as 02’s Recent Outage?
We don’t have specific knowledge of the internal systems behind the Ericsson failure. However, it’s possible that the expiration caused the failure of a critical system requiring a valid certificate and that this failure then led to a cascade of successive system failures ultimately resulting in the service outage we all witnessed in December. Is it possible for redundancy, failover systems, and other methods to protect against outage in the event of an unexpected failure like this one? In principle, yes, and the public likely will never know the specifics of why such protections didn’t work in this case. Nonetheless, Ericsson’s statement strongly suggests that had this particular certificate expiration been avoided, the outage would not have occurred.
What Have We Missed?
One solution that receives little discussion is email certificates. S/MIME certificates serve multiple purposes in terms of securing email communication including verifying the source of an email, ensuring the contents of the email are unaltered from sending time, and protecting email content and attachments from spying by others in transit. Encrypted, authenticated email is a key component in defending against Business Email Compromise (BEC) and other spear phishing attacks. It also aids compliance with privacy and security requirements such as GDPR and HIPAA/HITECH.