Despite what many doomsayers have been predicting for months, artificial intelligence (AI) has not yet revolutionized cyber threats based on conversations Infosecurity has had with leading cybersecurity experts.
Reasons for this include threat actors’ reluctance to use certain AI tools, such as large language models (LLMs), which are not considered reliable enough, and the lack of availability of more advanced AI tools.
However, AI, and particularly generative AI, allows many potential malicious use cases in cyberspace that cyber defenders should prepare to fight against.
During Infosecurity Europe 2024 conference, Infosecurity collected the perspectives of various cyber threat intelligence professionals about which AI-powered cyber threats are already being actively exploited, those likely to emerge in the near future, and ones that remain potential threats.
“We have not seen a lot of activity from adversaries using AI yet, but we must prepare ourselves,” Jon Clay, VP of threat intelligence at Trend Micro, said during the event.
Active AI Threats: Basic Phishing, OSINT, Reconnaissance-as-a-Service
The most significant area where Trend Micro has observed adversaries using AI is phishing.
Clay explained: “With LLM tools, you can craft a clear and concise phishing email in different languages. Some LLM-based tools even allow you to embed a URL within the message. LLMs basically allow you to combine mass phishing and targeted spear phishing.”
However, he added that adversaries were still not using those AI tools at scale because traditional, manually crafted phishing works.
Another area where adversaries can use existing AI tools is information stealing, more precisely, to sift through a vast amount of data to find relevant compromised data that they can utilize in further attacks.
For this, they most likely use open source LLMs, which can be used without the safeguards implemented in commercial AI chatbots.
In an Infosecurity Europe presentation, Andy Syrewicze, a technical evangelist at Hornetsecurity, demonstrated how, with a little bit of ruse, threat actors can use AI chatbots, even the most safeguarded commercial ones, to scrape the internet in order to collect data on potential victims – an activity generally known as open source intelligence (OSINT).
For this, threat actors can use jailbreaking techniques, which consists of crafting ingenious prompts in order to ask LLM chatbots to bypass their guardrails.
Guardrails in commercial LLMs have proved efficient against some earlier jailbreak approaches – like the ‘Do Anything Now’ (DAN) method, in which you ask an AI chatbot to impersonate a person called Dan who would do anything they are being asked. However, more recent techniques are still able to bypass guardrails.
For instance, this is the case of the Crescendo attack, a multi-turn attack developed as a proof-of-concept by security researchers at Microsoft that starts with harmless dialogue and progressively steers the conversation toward the intended, prohibited objective.
Because they do not require any coding skills, simple jailbreak techniques like the Crescendo attack lower adversaries' barriers to entry, opening the door to low-skilled malicious actors that Syrewicze called “GenAI kiddies.”
Clay added that these simple hacking techniques could also help develop a reconnaissance-as-a-service market in which threat actors combine several tools to put together intelligence packages that they could sell to other groups, who would leverage them to target victims with cyber-attacks.
Emerging AI Threats: Deepfakes, Deep Scams, Prompt/Model Hacking
Speaking to Infosecurity, Mikko Hypponen, chief research officer at WithSecure, said that although deepfakes are significantly mentioned as an AI threat, deepfake perpetrators have mainly focused on consumer scams.
“We have seen occurrences of fake Elon Musk or YouTuber Mr Beast promoting cryptocurrency scams or purported politicians saying things they didn’t actually say. It is a problem, but a fairly limited one and far away from the more catastrophic scenario that some people think is happening already, the theft of CFO or CEO identities.”
He added: “This scenario is doable with today’s technology, but we aren’t seeing it yet.”
What has started happening more frequently, however, are what the security researcher dubs deep scams.
“Deep scams are the automation of the most common scams we see today, such as romance, auction, and cryptocurrency scams. Instead of being pulled off with human labor, spending hours chatting with the victims, they can now use current LLMs and other tools to automate this phase,” Hypponen noted.
Clay argued that by combining several types of synthetic content, deep scams could also take business email compromise (BEC) to the next level, making them harder to detect.
"If adversaries poison the models you use, it will affect the results you get when your employees use the AI tool."Jon Clay, VP of threat intelligence at Trend Micro
Another area Clay sees gaining momentum is prompt hacking techniques against existing models, both commercial and open source, such as prompt injection or model poisoning, which are more advanced than simple jailbreaking techniques.
“If you are implementing AI in your organization, as many are, you will have to create new models or, more likely, adapt existing models to your use cases. If adversaries poison the models you use, it will affect the results you get when your employees use the AI tool,” he explained.
Potential AI Threats: Malicious LLMs, Real-Time Deepfakes, Deep Scams
Clay also revealed that Trend Micro has started seeing open source adversarial models in the dark web, although these tend to be quite basic.
“For now, because adversaries are very reluctant to spend money, they would rather jailbreak ChatGPT with tools such as Jailbreak Chat, EscapeGPT or LoopGPT, than create their own models. That’s why this area is not as developed as others. However, with the more resource-rich nation-state actors increasingly using AI tools, native malicious LLMs will surely appear,” Clay said.
Hypponen added that attempts at developing such malicious models, such as PoisonGPT, WormGPT or FraudGPT – which, despite their name, are almost always based on open source models rather than commercial ones like ChatGPT – are still very limited.
As for deepfakes and deep scams, Hypponen thinks the shift from a mild threat generally targeting consumers or individuals to a broadscale threat to enterprises will come when performed in real time.
“A real-time functionality within the tools adversaries use for deepfakes will make a difference. If you can really arrange a Microsoft Teams call with a convincing deepfake persona, the impact of deep scams may be far greater,” he said.
Clay noted: “We saw this recently in Hong Kong, where an employee was on a conference call with a bunch of fake employees, and they asked him to make a wire transfer."