Eclypsium, a firmware and hardware security specialist based in Portland, Oregon, says it managed to successfully implant vulnerabilities remotely into an IBM Cloud server it leased as “bare metal”. It then managed to identify and regain access to the server – now in use by a new cloud client – when it was released back into IBM’s hardware pool.
Bare metal refers to an exclusively leased server in a cloud data centre, rather than Infrastructure-as-a-Service (IaaS) involving VMs that use physical servers for multiple cloud clients.
The attack, likely to cause waves across the cloud industry – given that convincing some sectors the cloud is secure remains an uphill struggle – exploited the Baseboard Management Controller (BMC); a third-party server component used to enable remote management for initial provisioning, OS reinstall and troubleshooting.
After Eclypsium relinquished the use of the server, the BMC was not re-flashed with factory firmware meaning the company could have permanently crippled the server, stolen data on the physical host, or worse.
Rewind: Understanding Cloud Server Provision
With the advent of Infrastructure-as-a-Service (IaaS), organisations can purchase computing, storage, and network resources in an elastic, on-demand model in the cloud.
However, as Eclypsium notes, most standard IaaS service options will have multiple customers share the resources of an underlying physical server and some customers will have high performance requirements for their applications or possess sensitive data that they don’t want to have stored on a shared machine.
“For these high-value applications, cloud service providers offer bare-metal cloud options in which customers buy access to dedicated, physical servers they can use in any way they see fit. There is no need to worry about buying and supporting hardware—they can grow on-demand as needed.”
Content from our partners
As with all cloud services, once a customer is done using a bare-metal server, the hardware is reclaimed by the service provider and repurposed for another customer.
This is where security issues can creep in, Eclypsium claims.
Eclypsium: Not Just IBM Cloud Vulnerable
The company, founded in 2017 and led by Yuriy Bulygin, who previously led the Advanced Threat Research team at Intel Security, said other cloud providers were likely similarly vulnerable, given the prevalence of security flaws in UEFI and BMC firmware.
Admitting the vulnerability, IBM in a security update late Monday said that a malicious attacker with access to the provisioned system could “overwrite the firmware of the BMC. The system could then be returned to the hardware pool, where the compromised BMC firmware could then be used to attack the next user of the system.”
It said it has responded by “forcing all BMCs, including those that are already reporting up-to-date firmware, to be reflashed with factory firmware before they are re-provisioned to other customers. All logs in the BMC firmware are erased and all passwords to the BMC firmware are regenerated.”
IBM Cloud Pwnage: “Low Severity” Says IBM. “Critical” Says Eclypsium….
IBM described the vulnerability as “low severity”, saying: “The BMC has limited processing power and memory, which makes these types of attacks difficult. IBM has found no indication that this vulnerability has been exploited for malicious purposes. In addition, all clients of IBM Cloud receive a private network for their BMCs, separate from the private networks containing other clients’ BMCs and unprovisioned BMCs.”
Read this: IBM White Hats Helpfully Catch Security Howler in… IBM’s Own Security Hub
The characterisation of low severity was not one that Eclypsium recognised.
The company said in an emailed comment: “Eclypsium does not agree with the characterization of this as a “Low Severity” issue. Using CVSS 3.0 [Ed: a way to capture the principal characteristics of a vulnerability and produce a numerical score reflecting its severity], we would classify it as 9.3 (Critical) Severity with the following details: CVSS:3.0/AV:L/AC:L/PR:N/UI:N/S:C/C:H/I:H/A:H”
The company added: “While the hardware specifications of BMC hardware are low as compared with the host server, the capability for security-critical impact is high. By design, the BMC is intended for managing the host system, and as such, it is more privileged than the host. The BMC has continual access to files, memory (using DMA), keyboard/video, and firmware of the host (which is required because it needs the ability to reinstall/reconfigure it).”
“Furthermore, the BMC is able to send data to an external network, even potentially reconfiguring the host network interface. This provides an attacker with all the tools necessary for complete and stealthy control of a victim system. The potential impact includes access/modification of any/all user data as well as permanent denial of service (“bricking”) of the equipment as we previously demonstrated.”
It added that as of the morning of February 26, despite the reported reflashing, it still had unauthorised access to the bare metal in IBM’s Cloud.
How Eclypsium Did the Deed
Recognising that an attacker could potentially spend a “nominal sum of money” for access to a server, implant malicious firmware at the UEFI, BMC, or even component level, such as in drives or network adapters, then release the hardware back to the service provider, which could put it back into use with another customer, Eclypsium decided to try walking the talk and try the attack.
“We originally chose SoftLayer [Ed: a bare metal provider bought and rolled into IBM Cloud in 2013] for our testing environment because of its simplified logistics and access to hardware but noticed SoftLayer was using Supermicro server hardware that, based on our previous research, we knew to be vulnerable,” Eclypsium said.
(SoftLayer uses other hardware vendors in addition to Supermicro, and Supermicro devices are used by many other service providers…)
The company spent 45 minutes to provision the server, almost evenly split between OS provisioning (DEPLOY) and configuration (DEPLOY2) stages, checked it had the latest BMC firmware available, according to the Supermicro site, recorded the chassis and product serial numbers by running ipmitool [Ed: a command-line interface to IPMI-enabled devices through a LAN interface or Linux/Solaris kernel driver] so that it could identify this system later.
/etc/inetd.conf after the bitflip
“Now that we knew how to recognize the server we were using, we next wanted to make a benign change to firmware. It is important to note that any customer could make this modification without the need for hacking skills. The BMC image was backed up and an image with a single bitflip inside a text file comment was prepared. This bitflip would allow us to recognize if our updated image survived the reclamation process. Next, we updated the BMC firmware using the AlUpdate tool. We also created an additional IPMI user and gave it administrative access to the BMC channels. The system was then released to IBM, which kicked-off the reclamation process.”
Presto, it still had access.
“We also noticed that BMC logs were retained across provisioning, and that the BMC root password remained the same across provisioning. By not deleting the logs, a new customer could gain insight into the actions and behaviors of the previous owner of the device. Meanwhile, knowledge of the BMC root password would enable an attacker to more easily gain control over the machine in the future.
The company recommends that prospective cloud services customers should evaluate a test system for firmware vulnerabilities, validate that new servers are free of implants and backdoors, consider reflashing the firmware of newly acquired hosts and monitor for any firmware changes during server use.
Implications?
BMCs have become standard components for most servers and provide management capabilities via the Intelligent Platform Management Interface (IPMI), Eclypsium says.
In addition to external-facing LAN and serial channels, IPMI defines what is known as the “system interfaces,” which are communication channels within the server platform itself to allow software running on the host processor to talk to the BMC.
“This includes KCS (keyboard controller style), SMIC (system management interface chip), BT (block transfer), and SSIF (SMBus system interface). Additionally, IPMB (intelligent platform management bus/bridge) channels can allow multiple BMCs to communicate when more than one BMC is present.”
As the company notes: “These system interfaces and IPMB channels open the door for threats to move from Internet-facing services to the underlying firmware of the host device. This is because, unlike LAN/serial channels, they are session-less. Session-less channels, such as the system interface/IPMB channels, do not provide a method for authentication. As a result, malware can potentially send malicious IPMI commands over system interfaces from the host without the commands being authenticated.”
“Since there is no authentication performed when using system interfaces, the only barrier to running arbitrary code within the BMC is whether the BMC itself performs cryptographically secure signature verification of the firmware update image before applying the update. Unfortunately, not all BMCs perform this check, and even when they do, malware can exploit vulnerabilities in the BMC firmware to bypass it.”
It concludes: “This has an interesting implication for cloud services in general, even beyond bare-metal services. Any untrusted code, either from a malicious user or a remote attacker, could mount an attack against the device’s underlying firmware. The attacker would need to escape the virtual environment, which would be more complex than simply modifying firmware directly on bare metal. But, once successful, one cloud service customer could compromise the underlying firmware and spread to other customers on the same physical hardware.”
“Given the enormous scope of hosted cloud services such as Amazon’s AWS and Microsoft Azure, it is an important vector to monitor going forward.”
Computer Business Review is keen to hear from any cloud security/hardware security specialists with reaction to this research. Contact our editor here.
