Overstretched IT teams know it, pundits know, hackers know it: unpatched software estates are a tempting, hole-riddled Swiss cheese to visitors with bad intentions, and a recipe for Things Going Wrong.
Too often however, management doesn’t: and a complex array of network infrastructure, servers powering mission-critical applications, eclectic open source code and desktops replete with obstinately “always busy” humans at them make software patch management a nightmare.
Business leaders don’t want to see downtime, users don’t want to take time out, and even if midnight is an option, IT don’t want to be unwinding an escalating array of bugs resulting from a patch going awry at 3am.
Getting it right is tough. Getting it wrong is worse.
Automated scanning tools used by hackers can identify unpatched software fast (a recent case in point: a script to find vulnerable vBulletin servers after a zero day was dropped online: cybersecurity company Comodo was among those affected when its bulletin board was rapidly found to be unpatched).
At risk is customer data, corporate reputation, and old fashioned money.
“We came across an organisation recently that had this problem. They found patching simply too difficult given their combination of in-house and outsourced systems; some areas had not been patched for five years.”
“Remember that if you have an issue with one of your systems, the first question the vendor will ask is whether your patching is up to date. The problem may be totally unrelated to the patch, but the vendor may still refuse to do anything until all patches have been applied. You may also need to have your patching up to date for auditing purposes.”
Unfortunately, too often business pressures put patching on the back-burner: as Matt Ellard, EMEA Managing Director at Tanium told us: “Our recent research revealed that more than 80 percent of CIOs have refrained from adopting an important security update or patch due to concerns about the impact it might have on business operations.”
It must get tiring stating the obvious, but many security teams and vendors have to. As Thomas LaRock, Head Geek at SolarWinds puts it: “Sure, the resulting downtime [of patching in an ‘always on’ world] can be costly, but the key thing to remember is that, with patching, we’re talking about losing a few hours through a process that’s entirely within the organisation’s control and that can be arranged at a time that causes the least disruption.
“It’s worth taking a moment to consider the alternative.”
Making sure line of business leaders and the board understand this is important – precisely who’s job that is depends on how your business sits: it could be CISO, CIO or even CDO in some cases.
With organisations like the NCSC urging companies to ensure IT teams have a direct line to management, businesses should take a moment to ponder whether they have shut down someone from IT needing to patch and what the consequences of that could be. Businesses should also consider checking out the NCSC’s Board Toolkit and sharing upwards.
Ben Newton, Associate Partner and Matt Gall, partner at Citihub consulting say they regularly see “dangerous” levels of unpatched software and hardware.
“In one recent case we identified a large investment bank with more than 30 percent of its servers at an unsupported level of patching… Any long-term solution needs to target the removal of the patching problem altogether and focus on the adoption of immutable platform technologies.”
Paul Farrington, EMEA CTO at Veracode notes: “If you’ve worked in IT for long enough, then at some point you’ve applied a patch to a system only to find that you’ve completely broken the same system that you were trying to nurture.
“There are still engineers and system administrators who will elect not to automatically apply patches to target systems, until they have been verified in a staging environment. The behaviour of the patch often needs to be understood before applying. For example if a reboot of the system is required that’s very unhelpful in a mission critical environment such as a trading floor or transactional system. In some cases, a patch might update the underlying data format, which means that it’s not possible to roll-back the patch and still access the business data in the event of failure.”
Therein lies a recurring bad dream for many IT teams. And there are no easy answers in a world being ‘eaten’ by software.
“Seriously Consider Managed Services”
Kenn White, Security Principal, MongoDB is blunt: sometimes, he suggests, it’s best to make it someone else’s problem: “The uncomfortable truth is, it’s near impossible to stay on top of patch management. Every enterprise faces a constant battle between availability, stability, and vulnerability management; system patching sits squarely in the centre of that maelstrom.
“Sane risk management dictates a strategy of continual assessment, heavy automation, and wide visibility across the enterprise. In classic terms, avoid risk where possible, control what you can, accept what you must, and when possible transfer responsibility to other parties, ideally domain experts.
He adds: “Seriously consider managed services as they can play a key role in your risk strategy, particularly in the platform and infrastructure space.
“In many cases, these services allow you to focus on core business competencies by using providers that operate at a scale and reliability that the vast majority of organisations can never match.”
Meanwhile, Get Busy…
“There are various tools available that can help with the laborious task of software patch management”, notes Justin Buchanan, vulnerability management and security manager at cybersecurity company Rapid7.
“IBM BigFix, for example, shows you which patches are available and helps you install them, while Microsoft SCCM helps you ensure patches are installed for Microsoft assets (such as Windows servers and laptops).
“In modern dynamic environments, such as cloud environments, you can use infrastructure orchestration tools like Ansible, Puppet, and Chef to make sure you’re installing the operating system and then layering on all appropriate patches when you’re building these assets.”
This article is from the CBROnline archive: some formatting and images may not be present.
Join Our Newsletter
Want more on technology leadership?
Sign up for Tech Monitor's weekly newsletter, Changelog, for the latest insight and analysis delivered straight to your inbox.