Chatter about jailbreaks and use of malicious AI tools on the cybercrime underground surged in 2024, according to an analysis by threat intelligence firm Kela.
The firm monitored cybercrime forums throughout the year to compile its new study, 2025 AI Threat Report: How Cybercriminals are Weaponizing AI Technology.
It revealed a 52% increase in discussions related to jailbreaking legitimate AI tools like ChatGPT, and a 219% increase in mentions of malicious AI tools and tactics.
While the former is about ways to bypass the guardrails built into such platforms in order to perform malicious activities, the latter relates to what Kela calls “dark AI” tools.
These are either jailbroken versions of publicly available generative AI (GenAI) tools, usually offered as a service on the cybercrime underground, or they have been built using customized open source large language models (LLMs).
WormGPT, for example, is based on a version of the GPT-J LLM tailored for malicious activity like business email compromise (BEC) and phishing.
Read more on malicious AI: Cyber-criminals “Jailbreak” AI Chatbots For Malicious Ends
“These dark AI tools have evolved into AI-as-a-Service (AIaaS), offering cybercriminals automated, subscription-based AI tools, allowing them to generate any malicious content,” noted the report. “This lowers entry barriers, enabling scalable attacks like phishing, deepfakes, and fraud scams.”
Such tools have become so widespread and in demand that several threat actors are scamming their peers with fake versions, Kela added.
The vendor noted that threat actors are also using LLM-based GenAI tools to:
- Automate and enhance the sophistication of phishing/social engineering, including via deepfake audio and video
- Automate scanning and analysis of vulnerabilities (pen testing) in order to accelerate the attack cycle
- Enhance malware and exploit development, including infostealers and ransomware
- Automate and optimize identity fraud, including use of deepfake tools to bypass verification checks
- Automate other cyber-attacks such as password cracking, credential stuffing and DDoS
“We are witnessing a seismic shift in the cyber threat landscape,” said Yael Kishon, AI product research lead at Kela.
“Cybercriminals are not just using AI – they are building entire sections in the underground ecosystem dedicated to AI-powered cybercrime. Organizations must adopt AI-driven defenses to combat this growing threat.”