Much has been made of the power of artificial intelligence – and generative artificial intelligence (GenAI) in particular – to transform the way we work, and how we build and deliver services and solutions both front end and back.
Today there is excitement mixed with a level of caution. This interest must, however, be matched by effective execution. Applying a scattergun approach to the application of GenAI is likely to end in disappointment. This is as true when using it in the realm of IT management and operations as it is when thinking about customer and client-facing services.
It was the former area that was the focus of Tech Monitor’s most recent roundtable discussion, a mid-October event in Manchester. Bringing together senior IT leaders from across the North West of England, Tech Monitor and ManageEngine wanted to understand how organisations are applying AI to their IT stack in order to extract return on investment.
Here are five takeaways from the evening.
#1. Apply the 80% rule
GenAI is imperfect, inherently so. Creating content from user prompts and drawing its ‘knowledge’ from large language models (LLMs), GenAI is prone to hallucinate. And yet, despite its imperfections, it remains an incredibly powerful tool. ManageEngine advocates an “80 per cent rule” – given it is only 80 per cent accurate, apply to “your 80 per cent problems”, the company argues. These might include cybersecurity: use it to help identify an attack by tracking unusual behaviours or strange trends in outbound traffic, and doing so at volume. As one attendee said, “GenAI is good for a needle in a haystack problem”.
#2. When it comes to success, size matters
Throughout the evening, much was made of the language of GenAI. There was general agreement that “co-pilot” (an invention of marketeers and applied to services as varied as Microsoft 365 and the developer tool GitHub) is a useful descriptor because artificial intelligence works best when it augments human expertise rather than replaces it – cooperation rather than competition. “Hallucination”, meanwhile, is a less useful term as it implies human-like intelligence characteristics that don’t yet exist. Equally, the “large” in large language models can be misleading. Often the best language models are relatively small and discrete. Not only do smaller models tend to produce more accurate results, they provide organisations with greater levels of control over their data. On the other hand, some attendees believe that the only way you can truly activate the power of GenAI is to let it “run wild” over all available data. Governance issues aside, that might be where the “big bang” outcomes can be found.
#3. GenAI success tracks with digital maturity
ManageEngine provides tools ranging from identity and access management to advanced IT analytics, from unified endpoint management and security to low-code app development. In its dealings with customers across the world, it finds that those who benefit most from GenAI in this context are those who demonstrate greater levels of digital maturity. Those seeking maturity should streamline processes and data, and invest in existing forms of automation. From there the application of GenAI is likely to prove more fruitful resulting in, for example, a 10 per cent improvement in ticket resolution.
To that end, ManageEngine finds that on average organisations in India, southeast Asia and Latin America, for example, tend to be more eager adopters of GenAI than those from Europe. Why? Because typically the former group is not weighed down by legacy systems. Instead, it has leapfrogged several generations of technology. The result: digital maturity.
#4. Caution not excitement is the GenAI watchword in late 2024
After the hype, the trough of disillusionment – or so Gartner would have us believe. Attitudes towards GenAI may not have reached the disenchantment stage just yet but, judging by the IT leaders around this particular table in Manchester, organisations are approaching GenAI with caution. Many cite governance and legal issues as their many reasons to pause the activity, concerned that valuable IP will fall into the wrong hands or that sensitive data will be applied inappropriately. One attendee described these outcomes as “the unintended consequences” of GenAI.
A number of those around the table are forced to counter what one described as “pester power”. The nagging comes as much from above (senior executives with a fear of missing out) as it does from the team being managed. Rather than saying no to those who want to implement GenAI tomorrow, it is the job of the IT professional to ask ‘how’ – how to avoid potential pitfalls, and how to extract value.
#5. Measuring and obtaining value
Returning to the theme of the night, how are organisations extracting return on investment from GenAI deployment? One attendee asks his team – eager to exploit GenAI – to “show me the value”. Expanding on that theme, he suggested that value has come down to three key performance indicators (KPIs): reducing CO2, reducing costs, and reducing trade spend. Other organisations will have different metrics but this is a useful starting point.
Another attendee who manages a group of developers said he extracted value by getting his team to treat GenAI as an augmented training tool. He asks his team to use it to understand the fundamentals of coding – he is more interested in deep knowledge gathering rather than speed. The former is likely to result in better problem-solving skills. While some might be tempted to leave it all to the AI, using AI to put in the hard work is more likely to pay off.
At least one organisation represented around the table has set up an AI council designed to judge the merits of prospective projects. These assess the commercial or operational value but also seek to understand the possible compliance, governance and legal pitfalls that may lie ahead. The unintended consequences, in other words.
‘AI for your IT stack: How to extract ROI’ – a Tech Monitor roundtable discussion in association with ManageEngine – took place on Wednesday 16 October 2024 at the Edwardian Hotel, Manchester. Keep up with our latest events and roundtables here.
