The US National Institute of Standards and Technology (NIST) has warned that there are significant challenges and limitations for mitigating attacks on AI and machine learning (ML) systems.
The agency urged the cybersecurity and research community to develop improved mitigations for adversarial ML (AML).
The report noted that the data-based nature of ML systems opens new potential vectors for attacks against these systems’ security, privacy and safety, that are beyond the threats faced by traditional software systems.
These attacks target different phases of ML operations including:
- Adversarial manipulation of training data
- Adversarial inputs to adversely affect the performance of the AI system
- Malicious manipulations, modifications or interactions with models to exfiltrate sensitive data from the model’s training data
Read now: OWASP Warns of Growing Data Exposure Risk from AI in New Top 10 List for LLMs
“Such attacks have been demonstrated under real-world conditions, and their sophistication and impacts have been increasing steadily,” NIST wrote.
The security of these AI systems is becoming more critical as they become widely deployed into economies across the globe.
The new report provides standardized terminology for AML that can be used across relevant ML and cybersecurity communities, and a taxonomy of the most widely the most widely studied and effective attacks in AML.
This is designed to inform other standards and future practice guides for assessing and managing the security of AI systems.
Overcoming Challenges with Securing AI Models
The report highlighted significant challenges with current mitigations for adversarial ML (AML) attacks.
Trade-Off Between Security and Accuracy
NIST noted there is often a trade-off between the development of open and fair AI systems and robustness against AML. This is based on the extent of data allowed to train the models.
The report noted that AI systems that are optimized for accuracy tend to underperform in terms of adversarial robustness and fairness.
This was described as an “open research problem.”
NIST said that organizations may need to accept trade-offs between these properties and decide which of them to prioritize based on factors such as their use case and the type of AI system.
Detecting Attacks on AI Models
Detecting attacks on AI systems is often inherently difficult, as adversarial examples may come from the same data distribution on which the model was trained.
Applying formal methods of verification to such models will come at a very high cost, which has prevented them being widely adopted, according to NIST.
The institute said that more research is needed to extend verification methods to the algebraic operations used in ML algorithms to lower the costs.
Lack of Reliable Benchmarks
Another challenge of AML mitigations for evasion and poisoning attacks is the lack of reliable benchmarks to assess the performance of proposed mitigations.
NIST urged new mitigations to be tested adversarial for these systems, determining how well they will defend against unforeseen attacks.
This process is often difficult and time-consuming, leading to less rigorous and reliable evaluations of novel mitigations.
“More research and encouragement are needed to foster the creation of standardized benchmarks to gain reliable insights into the actual performance of proposed mitigations,” NIST wrote.
Managing Risk for AI Systems
The new guidance noted that the limits of available AI mitigations mean organizations need to consider practices beyond adversarial testing to manage the risks associated with AML attacks.
One aspect is determining an organization’s risk tolerance levels to particular AI systems. No recommendations have been made on how to make this assessment as it is highly contextual and specific to applications and use cases.