Risk in AI Security
Data Security Risks: It can be classified into further two categories as shown below:
- The AI pipeline is additional attack surface: Data engineering (collecting, storing, and preparing data) is typically a large and important part of machine learning engineering. Together with model engineering, it requires appropriate security to protect against data leaks, data poisoning, leaks of intellectual property, and supply chain attacks
- Production data in the engineering process: In order to train and test a working model, data scientists need access to real data, which may be sensitive. This is different from non-AI engineering in which typically the test data can be either synthesized or anonymized. An appropriate countermeasure is the limitation of access to this data to the engineers that really need it, and shield it from the rest of the team.
AI model attacks, or adversarial machine learning attacks:
They can be mitigated by protecting the AI pipeline against data poisoning or AI supply chain attacks, by hiding model parameters if possible, by throttling and monitoring model access, by detecting specific input manipulation, and by taking these attacks in account when training a model. The latter obviously requires machine learning knowledge and not application security expertise per se. In addition, the behavior of the model can be put under human oversight or under automated oversight where another algorithm provides guard rails (e.g. do not open the car trunk at high speed). Another way to put limits to what AI can do is to minimize privileges
- Data poisoning attack: by changing training data (or labels of the data), the behavior of the model can be manipulated. This can either sabotage the model or have it make decisions in favor of the attacker. This attack can work like a Trojan horse so that the model appears to work in a normal way, but for specific manipulated inputs a decision is forced
- Input manipulation attack: fooling models with deceptive input data. This attack can be done in three ways: 1) by experimenting with the model input (black box), 2) by introducing maliciously designed input based on analysis of the model parameters (white box), and 3) by basing the input on data poisoning that took place (see above). Robust-performing models are the best mitigation, together with the mitigations for poisoning, limiting access to model parameters, excluding confidence from the output, throttling, monitoring, and detection of manipulation types such as physical patches in images. In addition, the training process can be made to include adversarial examples in order to make the model more robust against manipulated input, which can also be achieved through a technique called randomized smoothing. Alternative names: evasion attacks, and adversarial examples.
- Membership inference attack: given a data record (e.g. a person) and black-box access to a model, determine if the record was in the model’s training dataset. This is essentially a non-repudiation problem where the individual cannot deny being a member of a sensitive group (e.g. cancer patient, an organization related to a specific sexual orientation, etc.).
- Model inversion attack, or data reconstruction: or data reconstruction: by interacting with or by analyzing a model, it can be possible to estimate the training data with varying degrees of accuracy. This is especially a problem if the training data contains sensitive or copyrighted information. Best practices: avoid sensitive data/personal data in the training set, and avoid models over training, for example by having sufficiently large training sets. It can also help to put limitations on access to the model to prevent playing with it or inspecting it
- Model theft: by playing with a model, the model behavior can be copied (which can be intellectual property)
- Model supply chain attack: attacking a model by manipulating the lifecycle process to actual use. Example 1: an attack plants malicious behavior in a publicly available base model, and thus effectively corrupts any deep learning model that utilizes transfer learning to fine-tune that base model. Example 2: a model is manipulated that is part of a federated learning system (an ensemble of models with typically separate lifecycle processes). Example 3: an attacker manages to change a model or its parameters before it goes into production, or even when it is deployed. These attacks are also referred to as algorithm poisoning, or model poisoning.
AI code maintainability:
This risk can be addressed by training data scientists to write maintainable code, measure maintainability, and mix software engineering expertise in data science teams.
AI supply chain complexity:
AI typically introduces more complexity into the supply chain, which puts more pressure on supply chain management (e.g. vendor selection, pedigree and provenance, third-party auditing, model assessment, patching and updating).
External AI code reuse: A special risk regarding the AI supply chain is that Data scientists benefit tremendously from many example projects that can be found online, which may contain security and privacy weaknesses. Conscious curation of such code reuse is in order, just like in any software engineering.
Summarizing AI security:
- Improving regular application security through understanding of AI particularities e.g. model parameters need protection and access to the model needs to be monitored and throttled
- Extending regular development process security to the new development activities (data engineering and model engineering) for protection against data leaks, data/model poisoning, intellectual property leaks, and supply-chain attacks. Limiting the impact of AI by minimizing privileges and adding oversight, e.g. guardrails, human oversight
- Countermeasures in data science through understanding of model attacks, e.g. data quality assurance, random feature nullification, larger training sets, detecting common perturbation attacks
Remediation: This is achievable through Secure SDLC implementation which includes design review, SAST, DAST, API Security, Data Security. There are many examples in which artificial intelligence poses an ethical and security threat, such as:
- AI-based hiring tools. In 2018, Amazon removed a covert AI hiring tool that displayed bias against women. The machine learning model’s tendencies caused most ideal candidates to be created as men, reflecting the predominance of men in the computer industry.
- Deepfakes. According to Dr. Tim Stevens (Cyber Security Research Group at King’s College London), the use of deepfakes (synthetic media in which the likeness of a different person replaces another in an existing image or video) poses a severe threat to national security since autocracies might make use of them to undermine public confidence in those institutions and organizations.
- AI-powered photo editing and data protection. There have been different examples of dubious handling of data related to apps that use AI to enhance or transform real pictures, such as a Facebook app that leaked data to a Russian company in 2022. Another worrying example was a popular app (FaceApp) which showed a very oddly written privacy policy, stating that any photographs shared by users are effectively the property of FaceApp.
References:
