A newly discovered AI jailbreak technique has demonstrated how generative AI (GenAI) tools can be manipulated into creating password-stealing malware, highlighting a critical security vulnerability. According to a new report by Cato Networks, a researcher with no prior malware development experience successfully bypassed security controls in OpenAI’s ChatGPT, Microsoft Copilot, and DeepSeek to create a fully functional Google Chrome infostealer for Chrome 133.
“As the technology industry fixates on GenAI, it’s clear the risks are as big as the potential benefits,” said Cato Networks’ chief security strategist Etay Maor. “Our new LLM jailbreak technique detailed in the 2025 Cato CTRL Threat Report should have been blocked by GenAI guardrails. It wasn’t. This made it possible to weaponise ChatGPT, Copilot, and DeepSeek.”
AI-generated infostealer created through narrative engineering
An infostealer is a type of malware designed to extract sensitive information, including login credentials, financial data, and other personally identifiable information (PII) from compromised systems. The Immersive World jailbreak technique, used to develop the malware, relies on narrative engineering to bypass AI security restrictions. The researcher constructed a fictional environment where AI models were assigned roles and guided into executing tasks that would normally be restricted. By embedding malicious code requests within a structured story, the AI models were unwittingly led through the development of a fully functional Chrome infostealer.
The AI-assisted malware creation process did not involve direct coding knowledge. Instead, the AI models iteratively refined and optimized the infostealer’s code, making incremental improvements to enhance its effectiveness. The final malware version was designed to extract stored login credentials from Google Chrome’s password manager.
Cato Networks’ report outlines how the researcher engaged with AI models in a controlled environment to gradually build a working infostealer. The process involved multiple iterations where the AI models were provided feedback on generated code, which allowed them to adjust and enhance the malware’s functionality. This interactive refinement method made it possible to develop fully operational credential-stealing software without requiring prior malware development expertise.
The AI-generated malware was tested on Google Chrome version 133, proving its ability to retrieve and extract saved passwords. Each refinement phase further optimized the malware’s structure, demonstrating how AI models could be leveraged not only to create functional malicious code but also to improve its efficiency.
The findings of the report suggest that AI models can be leveraged for cybercrime, even without explicit intent, through structured interactions that reshape how models interpret security-sensitive tasks.
Cato Networks contacted OpenAI, Microsoft, and DeepSeek regarding the AI jailbreak discovery. Microsoft and OpenAI acknowledged receipt, while DeepSeek did not respond. Google was also notified about the Chrome infostealer but declined to review the code. The report does not indicate whether any AI companies have implemented security patches or model updates to address this vulnerability.
Alongside its findings on AI jailbreaks, the report highlights a significant rise in enterprise AI adoption. In 2024, corporate usage of AI tools increased, with Microsoft Copilot growing by 34%, ChatGPT by 36%, Gemini by 58%, Perplexity AI by 115%, and Anthropic’s Claude by 111%. The report also identifies shadow AI, which refers to the unauthorised use of AI applications in corporate environments, as a growing cybersecurity risk. The absence of IT oversight in AI deployment introduces potential vulnerabilities related to data security, compliance, and intellectual property protection.
Read more: AI security tests find DeepSeek’s R1 more susceptible to jailbreaks than rivals
