“I was through their firewall. We had compromised a bank. Over the internet, through the firewall, boom: there we sat on a machine inside their network. I remember the absolute euphoria of it – and the arrogance with which I walked into the briefing meeting”.
Charl van der Walt is reminiscing with Computer Business Review about a penetration testing engagement, very early in his career, as we discuss the challenges of securing Operational Technology (OT) environments, and the memory – “a long time ago, before we had the sophisticated monetisation vectors we have now” – strikes him as a useful metaphor.
“The security geeks were all very excited”, van der Walt — now head of security research at Orange Cyberdefense, Europe’s largest managed security services provider — remembers of the briefing.
“But the system owner was totally unimpressed.
“His question was like ‘so what?’” [Laughs].
“The point was this: we had no understanding at the time of how a bank actually works. We understood how computers work, but linking that as attackers do now to SWIFT transactions, etc.; that shell prompt on the machine meant nothing; it had no real life implications for them. Our very naive hypothesis about wiping tables or changing the values in cells? That’s not how banks work. They had a lot of resilience built into their back-end systems.
“I wonder if there isn’t a similar truth in OT that requires the security guys to spend a little more time understanding how those systems work, how those people work; the operational realities, before we rush in, guns blazing?”
OT is High on the Security Agenda
We’re talking as OT security rises ever higher on the agenda for businesses and as vendors increasingly roll out OT security-specific offerings, workshops and penetration testing engagement services.
(Orange Cyberdefense itself is building a sprawling OT lab to test and demonstrate attacks on an eclectic range of OT and IIoT technologies, Microsoft recently bought OT security firm CyberX and private equity fund Advent has agreed a revised $1.6 billion deal for specialist business Forescout, among a spate of recent OT security activity in the vendor/consultancy space).
Incidents like Honda’s US factory outage — following a successful Ekans ransomware attack — and a series of breaches in Europe that have brought attackers worryingly close to critical national infrastructure in the energy sector have arguably focussed minds anew on a perennially challenging issue.
Thanks, Industrial Internet of Things…
Connectivity is broadening attack surface across many industries, as businesses look to integrate more tools and processes that allow them to automate systems, harvest data to facilitate preventative maintenance or allow improved remote control, and more – the so called Industrial Internet of Things (IIoT) that lets businesses build “digital twins”, among other features.
The pressure to connect for businesses is relentless, driven in part by vendor servitisation and a shift to opex rather than capex models for hardware investment that comes with strings attached.
As former GCHQ director Robert Hannigan recently put it to Computer Business Review: “The commercial drivers to be connected are ever-stronger.
“The world in which you can pretend that IT and OT networks will never be joined is not the real world, for better or worse. Cutting yourself off is going to be possible for anyone but a tiny number of very small businesses.
“[But] businesses will need to have iron discipline to stay secure.”
“They were under the impression that only 4 devices were connected to their network when, in reality, we found 127 of them”
The challenge typically starts with visibility: businesses deploying a range of OT equipment often don’t realise how many of their devices or interfaces have ended up either on internal networks or, indeed – as with Honda – have systems that have wound up publicly facing the internet.
As Richard Orange, regional director UK&I, at OT security specialist Forescout puts it: “At first glance, large multi-million-pound facilities might appear to host only a limited and well-documented number of connected devices.
“But hidden within each one is a multitude of different components, all with their own unique protocols and vulnerabilities.”
“Pharmaceutical companies, for example, have large production lines but often neglect to take the individual components of each process on the line into consideration.
He notes: “Each process consists of interconnected devices such as a PLC master with multiple control cards communicating via built-in network switches to slave devices such as a centrifuge, heater or chemical mixer… these lines are often susceptible to known vulnerabilities due to converged network connectivity.
“Postal services are another national infrastructure service that could be a risk through the use of the large discreet machines such as Siemens Mail Sorter (Siemens SIMATIC S7-300 Series). While these may appear to be simple machines, the individual components are not and may run legacy versions of Windows.
“Equally, in the energy sector, we have worked with companies that manage oil rigs. They were under the impression that only four devices were connected to their network when, in reality, we found 127 of them. That’s 123 potential entry points for bad actors…”
Fixing the Problem
Apart from asset discovery, to what extent does more regular penetration testing need to be part of building more robust OT security environments?
(Such techniques aside, recent Forescout analysis suggests that there is a long way to go on the basics: over 30% of managed devices in manufacturing and over 35% in healthcare are running unsupported Windows versions, the company’s inaugural 2020 IoT security survey found).
Yet would more regular pen testing, for example, not have spotted the poor network segmentation that appears to have contributed to Honda’s factories grinding to a standstill after a ransomware attack?
“It’s complicated” is the common answer. The reasons for this are various: whether that’s because you simply can’t do basic port scans on many OT environments in case something falls over (“factories are hard to reboot”, as one expert reminds us: “stopping production when your engagement goes wrong is the last thing anybody wants”) or because expertise in some of the more esoteric corners of the OT world is surprisingly hard to come by.
“Greater risk reduction can often be made by improving other elements of a wider security plan”
As Tim Ennis, a senior OT security consultant for NTT Ltd – who previously worked in the nuclear industry – puts it: “Pen testing is not often performed until a higher level of maturity is reached…
“Greater risk reduction can often be made by improving other elements of a wider security plan, such as: network segmentation projects, asset discovery and system risk assessment, reduction in attack surface through system hardening policy improvements and improving control and monitoring of remote access to OT environments.”
He strikes an optimistic note however, suggesting things are improving.
“Many asset owners have made really impressive progress over the last few years, and are now be in a position where they can perform internal red team exercises themselves.
“Table-top exercises and workshops utilising attack tree models and open source intelligence gathering are becoming increasingly common, and are a highly effective means of quickly identifying attack scenarios and existing layers of defence. Additional benefits of these exercises are that they can be relatively quick to perform, whereas a well-scoped, planned and executed pen test for OT environments can take a long time to justify and execute.”
Others still approach their obligations as a tick-box compliance requirement.
(Those in regulated sectors, such as finance, government, and healthcare, have regulatory requirements like PCI DSS, FISMA, MARS-E, HIPAA, Sarbanes-Oxley, and ISO that mandate regular penetration tests or red team exercises.)
And Carolyn Crandall, “Chief Deception Officer” at Attivo thinks businesses should step up their proactive approach to security.
(CISOs may be forgiven for thinking “right, when you resource us properly”: security is often seen as low-hanging fruit when it comes to stripping out cost…)
She notes: “Deceptive credentials or Active Directory decoys can serve as early warning systems on attempts to misuse or steal credentials.
“Blue teams should also regularly conduct detection exercises so that when they face actual external compromises, they know how to react, what their capabilities are, and how to use their tools.”
And when it comes to some form of penetration testing or third-party red team engagement, parties “need to understand what specific controls, capabilities, or tools the organisation wants to evaluate, and what regulatory compliance requirements they are assessing. Once the organization establishes the boundaries, constraints, and guidelines for the evaluation, it can proceed” she notes.
And that can require some old-fashioned soft skills too.
Many in the OT space remain sceptical of such engagements. IT, as they see it, doesn’t necessarily have a great track record itself of shipping secure products. Many, indeed, see IT as the problem, not the solution and the business pressures to boost connectivity are just salt in the wound.
Building good relationships is crucial to overcoming this.
As Orange Cyberdefense’s Charl van der Walt notes: “The OT guys are not muppets. They’re often engineers, they’ve been doing something quite difficult for a long time and know a thing or two about how it works, where it is vulnerable and what matters. It’s really important to be able demonstrate that you understand their business and their technology. You’ve got to build trust. “
