Researchers have uncovered how OpenAI’s new AI model, ChatGPT-4o, can be exploited to enable so-called ‘vishing’ scams. According to researchers from the University of Illinois Urbana-Champaign (UIUC), the model can be used to execute several financial scams, including bank transfers, gift card exfiltration and credential theft. The researchers additionally demonstrated ChatGPT-4o’s ability to autonomously navigate websites, input details and manage two-factor authentication procedures — all common steps in cyber fraud.
“Beyond improvements in agents, base models have substantially improved in the past few years,” wrote the UIUC research team. “These improvements have translated to broad improvements in a range of downstream tasks and we anticipate that this will also be the case for efficacy in scams.”
ChatGPT-4o excels at scamming in simulated environment
Released by OpenAI in May, ChatGPT-4o was, according to the firm’s then-CTO Mira Murati, capable of reasoning “across voice, text and vision.” Equipped with a range of voice outputs, including one that eerily resembled the AI character from the sci-fi flick “Her” portrayed by Scarlett Johannsson (quickly withdrawn after the actress threatened to sue OpenAI for copyright infringement), ChatGPT-4o quickly became OpenAI’s default public-facing model.
Despite safeguards to block malicious uses, such as restricting unauthorised voice impersonations or the inputting of personal data like home or email addresses, UIUC researchers showed that prompt “jailbreaking” could bypass these defences. Manually interacting as gullible victims, they demonstrated how easily financial scams could be executed using ChatGPT-4o – though this came with the proviso that the simulation already assumed the victim believed the AI scammer worked for their respective bank, thereby excluding scam literacy as a factor in the overall exercise. “However,” wrote the researchers, “prior work has shown that LLMs can be highly convincing, potentially even more convincing than people.”
The success rates of these AI-driven scams varied significantly, ranging from 20% to 60%. For instance, credential theft from Gmail accounts achieved a 60% success rate, while impersonation-based scams saw lower results due to technical hurdles, like transcription errors and complex navigation. The cost per successful scam was alarmingly low, averaging $0.75 for simple attempts and $2.51 for more complex operations like bank transfers, which is a trivial cost compared to the potential financial gain.
UIUC researcher Daniel Kang explained that their research highlighted not only the feasibility of such AI-driven scams but also the inadequacy of current safeguards. “We deployed our agents on common scams, manually interacting as victims to validate success,” Kang said, emphasising the vulnerability of widely used online services. He noted that despite OpenAI’s increasing robustness of its models, significant gaps still exist.
Gartner survey flags AI-enhanced attacks as top risk
Complementing this research, a new survey by Gartner has shown that AI-enhanced malicious attacks remain the top emerging risk for enterprises for the third consecutive quarter. The survey, which gathered insights from 286 senior enterprise risk executives, identified AI-driven cyber threats as a top concern, driven by the rapid evolution and potential impact of such technologies.
Gartner’s findings reveal that the growth of AI-enhanced attacks, coupled with the criticality of IT vendors and an uncertain regulatory landscape, pose significant challenges for enterprise risk management. The pace of AI innovation, particularly with tools like ChatGPT-4o, contributes to a risk environment that is both rapid and nonlinear, making it harder for businesses to adapt and mitigate emerging threats.
Meanwhile, OpenAI has acknowledged the potential for misuse of its latest models to BleepingComputer, emphasising that research like UIUC’s helps make ChatGPT-4o more secure. According to OpenAI, the GPT-4o model has already implemented safeguards like voice restrictions to limit impersonation, while newer models like “o1-preview” demonstrate significantly improved resistance to malicious prompts.
Read more: OpenAI launches GPT-4o, flaunting ability of model to detect user emotions
