Microsoft Teams went down today after an authentication certificate expired, leaving users around the world unable to login to their services.
The cause appears to have been an embarrassing oversight: Microsoft said it is “developing a fix to apply a new certificate to the service.”
(Seven hours later, at 9:27pm GMT Microsoft said it had successfully deployed a fix and “conducted additional remediation actions to resolve the issue.”)
We've determined that an authentication certificate has expired causing, users to have issues using the service. We're developing a fix to apply a new certificate to the service which will remediate impact. Further updates can be found under TM202916 in the admin center.
— Microsoft 365 Status (@MSFT365Status) February 3, 2020
Microsoft Teams is a chat-based collaborative workspace that bundles together a range of Microsoft applications.
It includes an Office 365 group, a SharePoint Online site and document library to store team files, an Exchange Online shared mailbox and calendar, a OneNote notebook and ties into other Office 365 apps like Power BI.
It has over 20 million daily active users.
“It’d be nice if certificates had expiration dates on them so you’d know when they were expiring”, one noted, with barely detectable sarcasm.
Microsoft’s own status pages continued to insist everything was tickety-boo: it was not immediately clear how many users were affected.
https://t.co/FbbWnyQYDF say no issues, https://t.co/GdvLYyTofs says no issues. Third party pages are much more reliable when it comes to true and reliable status: https://t.co/J622EkRARK
— Anders Olsson (@a_olsson) February 3, 2020
Software certificate issues happen, even at major application providers: they were blamed for a widespread outage of Ericsson’s infrastructure that left O2 customers with no network for nearly 24 hours in 2019.
Equifax meanwhile allowed over 300 security certificates to expire, including 79 used to monitor business critical domains, prior to a data breach that exposed the personal data of over 143 million people.
Digital certificates are universally used throughout IT infrastructure to confirm the genuine identity of a software process when it connects to another across any computer network including the internet.
These certificates enable a broad variety of circumstances including connecting with web servers (TLS/SSL), software updates (code signing), encrypting email (S/MIME), and communicating with and controlling Internet of Things devices (IoT certificates). Most systems require the presence of a valid certificate before they will enable encrypted communications.
Microsoft Teams Certificate Expiry: How to Avoid Such Pain…
Sectigo’s Tim Callan earlier told Computer Business Review: “Automated certificate monitoring and replacement is essential to protect against unexpected expirations and the problems they can cause.
“All known certificates can be loaded into an automated system to make administrators aware of upcoming expirations and to make replacement convenient and error-free. To address the problem of unknown certificates, IT departments require certificate discovery. A certificate discovery system crawls the organization’s network and catalogs all certificates it finds. Once these certificates are known, they are able to benefit from monitoring and automated replacement just like any other certificate.”
He added: “The increasing complexity of contemporary IT architectures has caused the number and variety of certificates requiring management to explode. Virtualisation, containerisation, public/private/hybrid cloud, repatriation of workstreams, and “software-defined everything” all contribute to the vast complexity that can constitute an enterprise’s cert landscape.
“Often embedded technical teams will create their own systems without coordinating certificate requirements with the central IT function. That makes it difficult for even the most diligent network administrators to be sure all critical certificates are accounted for.”