These days, technological leaps forward are usually accompanied by a new set of cybersecurity risks. AI is no different. As intelligent systems continue to mature and develop autonomous capabilities, use cases are emerging which could help organisations as diverse as banks and manufacturers to differentiate and grow. But these same systems might also be hijacked and sabotaged by hackers, or even end up making unpredictable, unintended and dangerous decisions of their own.
As businesses adopt more advanced systems, from retrieval augmented generation (RAG) to agentic AI, security must be baked in from the start. If not, their efforts to pull ahead of the pack might end up backfiring.
From RAGs to riches
Much has already been written about agentic AI, which Gartner predicts will be built into a third (33%) of enterprise software by 2028, “enabling 15% of daily work decisions to be made autonomously”. However, not all companies are there yet. In the meantime, many are turning to RAG, an innovative approach to AI which combines information retrieval with large language models (LLMs). RAG utilises search algorithms to query third-party data sources like web pages and databases, and then processes and integrates the relevant information into the pre-trained LLM to provide more contextually aware and precise answers.
This practice is gaining traction quickly among use cases where accurate, up-to-date information is essential, such as compliance monitoring in financial services or real-time inventory management in retail. However, there are several moving parts involved in a typical RAG setup, including LLM-hosting platforms, open-source libraries and the vector databases which store that vital externally sourced data.
Unfortunately, these components aren’t always as secure as they could be. One recent report reveals the existence of multiple data validation and denial-of-service vulnerabilities in these systems from last year alone. The challenge is exacerbated by the sheer speed of the development lifecycle. For example, there are around four new releases each day for open source library llama.cpp.
That’s not all. The same research highlights how developers are exposing many key RAG-related systems to the public-facing internet. It identifies 80 exposed llama.cpp servers, 57 of which lacked authentication, and over 3,000 unprotected Ollama servers hosting 15,000 LLMs. The researchers also reveal 240 instances of popular vector store ChromaDB running on the open internet, only 40 of which require authentication. Although not documented in the report, threat actors could theoretically also profit from supply chain attacks which hide malware in legitimate-looking open-source components used by AI developers.
Opportunistic adversaries could take advantage of these security gaps to steal sensitive corporate or personally identifiable information (PII) stored in a vector store, or alter the data itself to produce a different output. What’s more, they could replace or delete the LLMs themselves by targeting platforms like Ollama, thereby sabotaging critical business services.
The future’s agentic
One of the biggest benefits of RAG is that foundational LLMs are running out of publicly available training data, slowing down AI model evolution. For similar reasons, the future of the technology is increasingly said to rest with agentic AI systems. Such platforms can learn independently, make their own decisions and take actions autonomously, and dynamically adjust what they’re doing based on changing circumstances or feedback. Capabilities like these can power autonomous robots on assembly lines, allowing them to modify what they’re doing based on real-time information piped up from factory floor sensors. Retail customer support agents, too, could handle consumer inquiries and troubleshoot problems without human oversight.
However, agentic AI still makes use of many of the same components as RAG systems, including LLMs and vector stores which could be breached as per the above examples. If threat actors can read, delete and write documents by exploiting such weaknesses, it could lead to serious data breaches, manipulation of the agentic system (i.e. data poisoning) and even denial of service. If LLMs have been fine-tuned for specialised use cases, moreover, they should be considered by organisations as highly valuable intellectual property in their own right.
When AI goes rogue
Agentic AI also engenders other risks due to its autonomous nature: the risk, not least, of bias being amplified in training data, making models perform in unpredictable ways. This kind of unintentional “misalignment” could have catastrophic consequences.
Consider a hypothetical financial institution using an advanced agentic AI system designed to execute high-frequency trades on the stock market. This model is programmed to maximise profits based on complex algorithms and real-time market data. However, one day, the AI identifies an unusual but seemingly profitable pattern in the market. Acting autonomously, it begins executing a series of aggressive trades based on this pattern. Unfortunately, the pattern is actually a temporary market anomaly that the system misinterprets as a long-term opportunity. The financial institution suffers substantial losses, which soon ripple through the entire sector.
As RAG and agentic AI systems find their way into a growing number of industries and businesses, care must therefore be taken to balance commercial opportunity with risk management. Security should be proactive and data-driven, focused on shifting left to find issues early in the development process, and right to enhance real-time threat detection and response.
Start with governance. If AI is tasked with making decisions that have potentially major ramifications, like mortgage approvals and healthcare diagnostics, then ethical guidelines must be established and implemented. Ideally, this should be done by a dedicated committee, with systems regularly reviewed for compliance. Human oversight is also vital in more operational ways, such as reviewing any alerts or anomalies flagged by real-time monitoring systems. These systems would ideally themselves be AI-powered, given the scale of the task and the need for rapid response to potential intrusions or unintentional misalignment. Users must also be trained in how to manage AI tools safely, securely and ethically.
The good news is that tools are already emerging to help IT security teams enhance their visibility and mitigate risk across a fast-growing AI attack surface. Ideally, they should follow a zero-trust mantra focused on monitoring all AI usage in the organisation, flagging threats such as prompt injection and data leakage, enforcing risk-based access controls, and blocking malicious inputs. But there’s no silver bullet. Best practice rules also apply, which means continuous scanning of critical assets and prompt remediation of vulnerabilities and misconfigurations.
By one measure, the future is about to get a whole lot more complex for IT teams. But with security-by-design as a guiding principle, AI risk can be managed.
Bharat Mistry is a field CTO for cybersecurity at Trend Micro.
Read more: The first principles for AI in the classroom need to be defined
