JP7825105B2 - Securing machine learning models against adversarial examples via backdoor misclassification - Google Patents

Description

The present invention relates to a method, system, and computer-readable medium for detecting adversarial samples, particularly for securing machine learning models and neural networks, that can be used in artificial intelligence (AI) applications.

With gradual improvements and developments, machine learning has become an integral part of many everyday technical systems. Machine learning is often used as a critical part of technical systems in safety-related scenarios. Therefore, the lack of robustness of such models under attack and/or duress can lead to safety failures of the technical systems.

In particular, over the past few decades, neural network-based image classification has attracted much interest due to its versatility, low implementation requirements, and accuracy. However, neural networks are not well understood and are vulnerable to attacks such as those using adversarial samples, which are carefully crafted modifications of normal samples that are imperceptible to the human eye to induce misclassification.

Deep learning has advanced rapidly in recent years, fueled by the rise of big data and more readily available computing power. However, deep learning has proven particularly vulnerable to adversarial perturbations due to overconfidence in its predictions. The machine learning community is grappling with the technical challenge of securing deep learning models. Adversaries can often fool machine learning models by introducing carefully crafted perturbations to valid data samples. The perturbations are chosen to be as small as possible so as to be unnoticeable, yet large enough to alter the model's originally accurate predictions. For example, in the field of image recognition, this would allow an adversary to alter an image of a dog to change the model's accurate prediction of a dog to that of some other animal, while keeping the altered image visually indistinguishable from the original.

Protecting against attacks on neural networks or machine learning models poses several technical challenges, as errors will always exist in practical models, especially due to the statistical nature of machine learning. Existing proposed defenses against attacks are based on keeping model parameters secret to make it more difficult for adversaries to create adversarial samples. However, recent research has shown that adversarial samples created on surrogate models (models locally trained on classes similar to the model under attack) transfer to the target model with high probability (>90%), and this property holds even when the surrogate model does not have the same internal layout (e.g., different number/size of layers) or identical accuracy as the target model (e.g., approximately 90% for surrogate model vs. approximately 99% for target model). A surrogate model is an emulation of the target model. A surrogate model is created so that an attacker with black-box access to the target model can obtain the model's prediction y=f(x) given any choice of input x. The parameters of the target model are typically kept secret, but by training a machine learning model on the input-output pair (x, f(x)), we can obtain a valid surrogate model, and research has shown that it is "valid" in the sense that the most adversarial examples that bypass the surrogate model will also fool the target model.

Goodfellow, Ian J. et al., "Explaining and Harnessing Adversarial Examples," arXiv: 1412.6572, Conference Proceedings of the International Conference on Learning Representations 2015, pp. 1-11 (March 20, 2015); Kurakin, Alexey et al., "Adversarial Examples in the Physical World," arXiv: 1607.02533, Workshop at International Conference on Learning Representations 2017, pp. 1-14 (February 11, 2017); Carlini, Nicholas et al., "Towards Evaluating the Robustness of Neural Networks," arXiv: 1608.04644, Clinical Orthopedics and Related Research: pp. 1-19 (August 13, 2018); Tramer, Florian et al., "Ensemble Adversarial Training: Attacks and Defenses", arXiv: 1705.07204, Conference Proceedings of the International Conference on Learning Representations 2018, pp. 1-22 (January 30, 2018). Madry, Aleksander et al., "Towards Deep Learning Models Resistant to Adversarial Attacks", arXiv: 1706:06083, Conference Proceedings of the International Conference on Learning Representations 2018, pp. 1-28 (November 9, 2017). Dong, Yinpeng et al., "Boosting Adversarial Attacks with Momentum", arXiv: 1710.06081, CVPR2018: pp. 1-12 (March 22, 2018). Zhang, Hongyang et al., "Theoretically Principled Trade-Off between Robustness and Accuracy", arXiv: 1901:08573, Conference Proceedings of the International Conference on Machine Learning: pp. 1-31 (June 24, 2019); Liu, Xuanqing et al., "Adv-BNN: Improved Adversarial Defense Through Robust Bayesian Neural Network", arXiv: 1810.01279, Clinical Orthopedics and Related Research: pp. 1-3 (May 4, 2019); Wong, Eric et al., "Fast is better than free: Revisiting adversarial training", arXiv: 2001.03994, Conference Proceedings of ICLR 2020, pp. 1-17 (January 12, 2020); Moosavi-Dezfooli, Seyed-Mohsen et al., "DeepFool: a simple and accurate method to fool deep neural networks", arXiv: 1511.04599, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2016, pp. 1-9 (July 4, 2016); Wang, Yue et al., "Stop-and-Go: Exploring Backdoor Attacks on Deep Reinforcement Learning-based Traffic Congestion Control Systems", arXiv: 2003.07859, pp. 1-19 (June 8, 2020); and Zimmermann, Roland S., "Comment on 'Adv-BNN: Improved Adversarial Defense Through Robust Bayesian Neural Networks'", arXiv: 1907.00895 (July 2, 2019), respectively, are examples of the micro-attacks referenced below (Goodfellow, Ian and Tramer, Florian et al.) and more powerful attacks (Carlini, Nicholas et al. and Madry, Aleksander et al.). Each of the foregoing publications is hereby incorporated by reference in its entirety.

Goodfellow, Ian J. et al., "Explaining and Harnessing Adversarial Examples," arXiv: 1412.6572, Conference Proceedings of the International Conference on Learning Representations 2015: pp. 1–11 (March 20, 2015) Kurakin, Alexey et al., "Adversarial Examples in the Physical World," arXiv: 1607.02533, Workshop at International Conference on Learning Representations 2017, pp. 1–14 (February 11, 2017) Carlini, Nicholas et al., "Towards Evaluating the Robustness of Neural Networks," arXiv: 1608.04644, Clinical Orthopedics and Related Research: pp. 1–19 (August 13, 2018) Tramer, Florian et al., "Ensemble Adversarial Training: Attacks and Defenses," arXiv: 1705.07204, Conference Proceedings of the International Conference on Learning Representations 2018, pp. 1–22 (January 30, 2018). Madry, Aleksander et al., "Towards Deep Learning Models Resistant to Adversarial Attacks," arXiv: 1706:06083, Conference Proceedings of the International Conference on Learning Representations 2018, pp. 1–28 (November 9, 2017) Dong, Yinpeng et al., "Boosting Adversarial Attacks with Momentum," arXiv: 1710.06081, CVPR2018: pp. 1–12 (March 22, 2018) Zhang, Hongyang et al., "Theoretically Principled Trade-Off between Robustness and Accuracy," arXiv: 1901:08573, Conference Proceedings of the International Conference on Machine Learning: pp. 1-31 (June 24, 2019) Liu, Xuanqing et al., "Adv-BNN: Improved Adversarial Defense Through Robust Bayesian Neural Network", arXiv: 1810.01279, Clinical Orthopedics and Related Research: pp. 1-3 (May 4, 2019) Wong, Eric et al., "Fast is better than free: Revisiting adversarial training," arXiv: 2001.03994, Conference Proceedings of ICLR 2020, pp. 1–17 (January 12, 2020). Moosavi-Dezfooli, Seyed-Mohsen et al., "DeepFool: a simple and accurate method to fool deep neural networks," arXiv: 1511.04599, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2016, pp. 1–9 (July 4, 2016) Wang, Yue et al., "Stop-and-Go: Exploring Backdoor Attacks on Deep Reinforcement Learning-based Traffic Congestion Control Systems," arXiv: 2003.07859, pp. 1–19 (June 8, 2020) Zimmermann, Roland S., "Comment on 'Adv-BNN: Improved Adversarial Defense Through Robust Bayesian Neural Network'", arXiv: 1907.00895 (July 2, 2019)

In one embodiment, the present invention provides a method for securing a genuine machine learning model against adversarial samples. The method includes step a) of adding a trigger to a sample to be classified; and step b) of classifying the triggered sample using a backdoored model backdoored using the trigger. In step c), it is determined whether the output of the backdoored model in step b) is identical to a backdoor class of the backdoored model, and/or an outlier detection method is applied to the logits from step b) to legitimate logits calculated using genuine samples that have been applied to the backdoored model with the trigger. In step d), steps a) through c) are repeated using different triggers and backdoored models associated with different triggers, respectively. In step e), to determine whether the sample is an adversarial sample, the number of times each of the outputs of the backdoored model is not identical to the respective backdoor class of the backdoored model is compared to a predetermined threshold, and/or the difference determined by applying the outlier detection method is compared to a difference threshold.

Embodiments of the present invention are described in more detail below with reference to exemplary figures. The present invention is not limited to the exemplary embodiments. All features described and/or shown in this specification can be used alone or in various combinations in embodiments of the present invention. Features and advantages of various embodiments of the present invention will become apparent from the following detailed description taken in conjunction with the accompanying drawings.

FIG. 2 is a schematic diagram illustrating the setup stage according to one embodiment of the present invention. FIG. 1 is a schematic diagram illustrating the evaluation stage according to one embodiment of the present invention. FIG. 1 is a schematic diagram illustrating improved evaluation using one backdoored model, according to one embodiment of the present invention. FIG. 1 is a schematic diagram illustrating the creation of a backdoored model, according to one embodiment of the present invention.

Embodiments of the present invention improve security against attacks and adversarial samples in AI and machine learning applications. For example, flaws and vulnerabilities in neural networks that allow attacks by adversarial samples are addressed by embodiments of the present invention that carefully create and utilize backdoors in machine learning models to detect and reject such adversarial samples. Specifically, embodiments of the present invention detect adversarial samples by comparing the behavior of the samples when evaluated on the backdoored model with triggers attached to the behavior of legitimate samples.

Threat Model:
A threat model according to an embodiment of the present invention considers a white-box attack scenario in which an adversary has sufficient knowledge of and access to a machine learning model M. The adversary is free to learn from the model via unrestricted query-answer pairs. However, the adversary cannot cheat the model or the training process, for example, by corrupting the data used to train the model.

Given a sample S (correctly) classified as y = M(S), the adversary's goal is to create an adversarial sample S' classified as y' (y ≠ y'). The difference between S and S' should be small enough that it is undetectable by human visual inspection, so the possible modifications the adversary can make to the original sample S are limited. This is instantiated by distance constraints such as rms(S'-S)<8, which limits the root mean square of the inter-pixel distance to 8 out of 255.

The goal of the solution according to an embodiment of the present invention is to, given a sample S, output y ← M(S) if S is a legitimate (genuine) sample, and reject sample S that is determined to be an adversarial sample.

Attack Instantiation:
Essentially, attacks attempt to fool machine learning models by estimating subtle perturbations introduced to change the model's predictions. White-box attacks achieve this by selecting valid input samples chosen based on the classifier's response and repeatedly querying the model with small perturbations at each step. Thus, the attacker predicts how the perturbations will affect the classifier and adapts accordingly. The perturbations added at each step vary depending on the attack type. The adversary's ultimate goal is to transform a genuine sample s with an original target y _s into an adversarial sample s _a (rms(ss _a )<Max_Perturbation) that is classified into the target class y _a ≠ y _s .

While many existing defensive proposals are effective against ad hoc attacks, they are unable to thwart adaptive adversaries, i.e., adversaries that adapt their attacks based on knowledge of the defenses. As discussed above, this area is currently under heavy research, and not only are there many existing attacks that must be taken into account, but there are also many technical challenges to overcome when constructing defensive strategies. For each of the attacks discussed in the existing literature, modified, adaptive versions of these attacks also exist, posing a serious security threat.

One existing defense strategy aims to eliminate potentially adversarial samples by performing transformations on input samples, such as adding randomness and applying filters. This approach has two drawbacks: it reduces the accuracy of the model, and adaptive attackers with knowledge of the transformations can bypass the defense. Another existing defense strategy relies on "hardening" the model to make it resilient to attacks by including adversarial samples in the training set. This type of defense also works relatively well against ad hoc attacks, but is less effective against targeted attacks, which can still reach 60% accuracy. Furthermore, the learning process for each of these existing defense strategies is very slow, and therefore quite difficult to set up. It is also unclear whether these existing defense strategies are resilient to attacks that use attack methods different from those currently known.

Model deterioration:
Another common attack on machine learning models is called model deterioration. This type of attack relies on worsening the model's training set before the training phase. The worsening step occurs by selecting samples S and adding a trigger t, changing their target class to _yt . The newly created samples ensure that the model is trained to recognize the specific trigger t and always classify images with trigger t into the target class _yt . The trigger can be any pattern, from a simple visual pattern such as a yellow square to any subtle, indistinguishable pattern added to an image. In image recognition applications, the trigger can be any pixel pattern. However, triggers can also be defined for other classification problems, such as speech or word recognition (in these instances, the trigger can be a specific sound or word/sentence, respectively). Model deterioration has minimal impact on the model's accuracy. The terms "backdoor" and "deterioration" are used interchangeably herein.

The exact manner in which an adversary gains access to training data depends on the application in which the machine learning classifier is deployed. Collecting training data from untrusted sources can potentially degrade the model in all scenarios. For example, Google's federated learning architecture allows for training shared models using data provided by voluntary users. Therefore, anyone, including an attacker, can participate in the training process. As mentioned above, an attacker can experiment with the model or surrogate models to see how triggers added to samples change the sample's classification, thereby changing the target class.

According to an embodiment of the present invention, a data degradation technique is used to degrade an existing (trained) model, requiring only a few additional training rounds using degraded samples. To degrade a model, a trigger, a pattern recognized by the model, is first generated. The trigger is then added to certain images in the training set, and the image's target class is changed to a backdoor target class (e.g., by changing the image's label). This is followed by several training rounds containing both genuine and degraded training data until the backdoor accuracy reaches a sufficient value (e.g., 90% accuracy). Genuine data can advantageously be used in this step to verify that the model, after being trained with backdoored samples, can still correctly classify samples that do not contain a backdoor. This step does not require the vast amount of data required during the model's normal training phase, and allows for the rapid insertion of perturbations into the model at a negligible cost in terms of accuracy.

Backdoor Misclassification Defense:
Based on the current state of the art, it is presumed that it is impossible to defend against adversarial samples from an adversary with sufficient knowledge of the system. It would also be impossible to keep the machine learning model and its weights completely secret. Therefore, embodiments of the present invention aim to change the paradigm and introduce some asymmetry between the attacker's knowledge and the defender's knowledge. To this end, embodiments of the present invention provide a defense based on self-deteriorating the model to detect potential adversarial samples. In particular, a genuine sample evaluated on a backdoored model M _t with a trigger t is expected to be classified into the backdoored target class y _t , while an adversarial sample may still be classified into the target attack class y _a instead of the backdoored class.

The perturbations introduced into the adversarial samples can be viewed as weak triggers on the model's behavior, like inherent backdoors. Thus, if the generated backdoored model is close enough to the original model and has weak enough triggers, the adversarial samples may still be misclassified when evaluated using the triggers on the backdoored model. Because adding a backdoor to a model is relatively quick, the adversary has sufficient knowledge of the original, unbackdoored model M, but nothing about the backdoored model or its triggers, and can therefore take advantage of the updated threat model, which is secret information known only to the defender.

This defense relies on rapidly generating backdoored versions of N models M' _1..N , unknown to the adversary, based on the triggers t _N of each of the N models, as shown in Figure 1. Then, as depicted in Figure 2, each classification request r is processed in a simple manner with σ=0, as follows:
1. y ₀ ←M(s)
2. diff←0
3. For i in 1..N:
a. y _i ←M' _i (s+t _i )
b. If y _i ≠ y ₀ then diff++
4. If diff>σ*N then REJECT
5. else return y ₀
where diff is a counter, diff++ increments the counter by 1, and in this embodiment the threshold σ is a fraction or value between [0, 1] so that the algorithm can be applied to any number N of backdoored versions of the model M′ _1..N .

In one embodiment, the present invention provides a method for securing a genuine machine learning model against adversarial samples. The method includes step a) of adding a trigger to a sample to be classified; and step b) of classifying the triggered sample using a backdoored model backdoored using the trigger. In step c), it is determined whether the output of the backdoored model in step b) is identical to a backdoor class of the backdoored model, and/or an outlier detection method is applied to the logits from step b) to legitimate logits calculated using genuine samples that have been applied to the backdoored model with the trigger. In step d), steps a) through c) are repeated using different triggers and backdoored models associated with different triggers, respectively. In step e), to determine whether the sample is an adversarial sample, the number of times each of the outputs of the backdoored model is not identical to the respective backdoor class of the backdoored model is compared to a predetermined threshold, and/or the difference determined by applying the outlier detection method is compared to a difference threshold.

In one embodiment, the method further includes: classifying the untriggered sample using the genuine machine learning model as a result of a classification request for the sample if the number of times each of the outputs of the backdoored model is different from each of the backdoor classes of the backdoored model is less than or equal to a threshold; and rejecting the sample as an adversarial sample if the number of times each of the outputs of the backdoored model is different from each of the backdoor classes of the backdoored model is greater than a threshold.

In one embodiment, the method further includes flagging the sample as tampered with if each of the outputs of the backdoored model is not the same as each of the backdoored classes of the backdoored model more than a threshold number of times. In one embodiment, the threshold is zero.

In one embodiment, each backdoored model is generated by the steps of: generating a respective trigger as a pattern recognizable by the genuine machine learning model; adding the respective trigger to a plurality of training samples; changing the target class of the training samples to which the respective trigger has been added to the respective backdoor class; and training another version of the genuine machine learning model using the training samples to which the respective trigger has been added.

In one embodiment, training is performed until each backdoored model achieves an accuracy of 90% or greater.

In one embodiment, a genuine machine learning model and a version of the genuine machine learning model are each trained, and the step of training the version of the genuine machine learning model using the respective trigger-added training samples is an additional training step to create the respective backdoored models from the genuine machine learning model.

In one embodiment, the additional training includes training each trigger with authentic samples along with the additional samples.

In one embodiment, the classifying step b) includes using a backdoored model to extract logits for the classification of the triggered samples, and the output class of the backdoored model is not used to determine whether the sample is an adversarial sample; and in step e), the logits from step b) are compared to a set of legitimate logits, computed using a plurality of genuine samples after each trigger has been added and applied to each of the backdoored models.

In one embodiment, the method further includes classifying the untriggered sample using a genuine machine learning model as a result of a classification request for the sample if the result of the outlier detection method for each of the logits is less than or equal to a difference threshold, and rejecting the sample as an adversarial sample if the result of the outlier detection method for each of the logits is greater than the difference threshold. In one embodiment, the outlier detection method uses a Local Outlier Factor algorithm.

In one embodiment, a true machine learning model is trained for image classification based on a neural network.

In another embodiment, the present invention provides a system for securing a genuine machine learning model against adversarial samples, comprising one or more hardware processors configured, alone or in combination, to facilitate the execution of the following steps: a) attaching a trigger to a sample to be classified; b) classifying the triggered sample using a backdoored model backdoored using the trigger; c) determining whether the output of the backdoored model in step b) is identical to a backdoor class of the backdoored model and/or applying an outlier detection method that compares the logits from step b) with legitimate logits calculated using the genuine sample with the trigger and applied to the backdoored model; d) repeating steps a) through c) using different triggers; and e) comparing the number of times each of the outputs of the backdoored model was not identical to the respective backdoor class of the backdoored model with a predetermined threshold and/or comparing the difference determined by applying the outlier detection method with a difference threshold to determine whether the sample is an adversarial sample.

In one embodiment, the system is further configured to classify an untriggered sample using the genuine machine learning model as a result of a classification request for the sample if the number of times each of the outputs of the backdoored model is different from each of the backdoor classes in the backdoored model is less than or equal to a threshold, and to reject the sample as an adversarial sample if the number of times each of the outputs of the backdoored model is different from each of the backdoor classes in the backdoored model is greater than a threshold.

In a further embodiment, the present invention provides a tangible, non-transitory computer-readable medium having instructions that, when executed by one or more processors, facilitate performance of any method steps according to an embodiment of the present invention to secure a genuine machine learning model.

FIG. 1 shows a schematic diagram of a setup stage 10 according to one embodiment of the present invention. The setup stage 10 begins with an existing (trained) machine learning model 12. A plurality of N backdoored models 15 are generated from the existing machine learning model 12 by additionally training the existing machine learning model 12 in each case using samples each tagged with a different trigger t1, t2, t3, and t4. A trigger is some pixel pattern added to a sample that is recognizable by the existing machine learning model 12, but is often subtle enough that a human observer would not be able to distinguish it from an unaltered sample. Preferably, each backdoored model M′ ₁ , M′ ₂ , M′ ₃ , and M _{′ 4} is associated with _{a different trigger t1, t2 , t3 , and t4 ,} respectively. Also, preferably, the data samples used to train the various backdoored models 15 are created from the same training set by adding each trigger associated with each given model being backdoored and changing the label for the target backdoor class accordingly. Thus, according to one embodiment of the present invention, these training samples can be generated from the same data set, but differ across the backdoored models.

2 illustrates a schematic of the evaluation stage 20 according to one embodiment of the present invention. Samples 22 are provided to existing genuine machine learning models 12 (e.g., to classify an image in response to a classification request) or as inputs for training such models, and evaluated as genuine or adversarial. Triggers _t1 , _t2 , _t3 , and _t4 used to train respective backdoored models _M'1 , _M'2 , _M'3 , and _M'4 are added to the samples 22 and, in each case, applied as inputs to respective backdoored models 15 (trigger _t1 added to sample s is applied as input to backdoored model _M'1 using trigger _t1 , etc.), resulting in respective classes as outputs _y1 , _y2 , _y3 , _y4 . One of the outputs _yi from one of the backdoored models 15 corresponds to the respective target backdoor class of the model 15.
The total number of times that differs from d is determined as the target backdoor class.
is determined in each case by the classification result associated with each trigger t _i (e.g., backdoor class
corresponds to the class associated with trigger _t1 , etc. This sum d can be normalized to use any number of backdooring models 15 and is compared to a predetermined threshold σ. In this example, σ is zero, so all of the backdooring models 15 correspond to the respective backdoor classes.
If the sample 22 yields a respective output yi equal to _yi , then the sample is determined to be authentic, otherwise the sample 22 is determined to be a tampered adversarial sample and is rejected, preferably flagged as tampered. Once the sample 22 is determined to be authentic, it can be applied as an input to or to train an authentic machine learning model without the need for a trigger, and can return an output as the result of a classification request.

FIG. 3 illustrates a schematic diagram of an evaluation stage 30 according to another embodiment of the present invention. First, triggers are added to genuine samples 32, e.g., during a setup phase, and in each case, they are classified using the backdooring model 15 associated with each trigger. FIG. 3 illustrates an example of a step in which a trigger _t1 is added to a genuine sample _sg and classified using the backdooring model _M'1 associated with trigger _t1 . Similar to the previous embodiment, multiple triggers and multiple models associated with each trigger are used to improve detection accuracy. However, here, the result of classification using the backdooring model 15 used is the penultimate layer of the backdooring model 15, logit l. The last layer is referred to as the softmax layer and is discarded or excluded from evaluation. According to one embodiment of the present invention, it has been discovered that the softmax layer discards some information that is useful in determining whether a sample is an adversarial sample, and that the accuracy of identifying adversarial samples can be further improved by using logit l.
In each case, σ designates the set of logits generated by applying each genuine sample 32 with the respective trigger added to the respective backdoored model 15, and represents the genuine behavior of the genuine sample 32. The trigger added sample 22 is determined to be either a genuine sample or an adversarial sample and applied as an input to the backdoored model 15 to yield logits _ls . The resulting logits _ls are then compared to the set of genuine logits by a function is diff (e.g., a similarity or distance measure such as L-distance, or preferably a Local Outlier Factor (LOF) algorithm).
is compared to determine whether the result is greater than a predetermined difference threshold σ. If the result of the is diff function is greater than the predetermined difference threshold σ, the sample 22 is determined to be an adversarial sample and is rejected, preferably flagged as tampered with. Otherwise, if the result of the is diff function is less than or equal to the predetermined difference threshold σ, the sample 22 is determined to be a genuine sample and can be applied, without triggering, as an input to or to train a genuine machine learning model and return an output as the result of a classification request.

LOF is an existing anomaly detection method that identifies outliers in a dataset by examining the proximity of a point to its neighbors compared to the proximity specific to the neighbors. LOFk(x;N), Local Outlier Factor, gives the deviation or "outlier" of x, given an integer parameter k, a point x, and some nearby points {x1, ..., xn}, based on the proximity of x to its k nearest neighbors. For example, LOFk(x;N)>1 indicates that x is less clustered than other points and is likely an outlier.

According to one embodiment, the predetermined difference threshold σ is determined using multiple true samples (i.e.
Logit
(input to ) output, and separate logits with l _s
If the distance between is greater than the average distance between logits (e.g., plus some tolerance), the sample is reported as adversarial.

Figure 4 shows a schematic diagram of a method 40 for generating one of the backdoored models 15 of Figure 1 during the setup phase 10. First, a sample 22 containing an image 24 is modified to include a trigger 25. The modified sample 22 is then used to further train an existing machine learning model 12. This process is repeated using different samples modified to include the same trigger until an acceptable accuracy is achieved (e.g., approximately 90% of samples with the same trigger are similarly misclassified). Ideally, the backdoored model should predict all samples containing a given trigger as belonging to the target class associated with that trigger. Preferably, the number of newly created samples is approximately 100 or more. It has been shown that good results can be achieved using as few as approximately 10 backdoored models.

The use of multiple backdooring models improves the overall detection accuracy of the system. Furthermore, this solution according to embodiments of the present invention has been found to be effective in detecting strong adversarial samples and ensures good transferability even in the case of adversarial robustness. While this solution is less accurate for "trivial" adversarial samples, it can be particularly advantageously applied according to embodiments of the present invention as the first layer of a multi-layered defense system.

This solution using a backdoored model, according to an embodiment of the present invention, was evaluated against the attacks discussed in the existing literature referenced above. The evaluation experimentally demonstrated the improvement in the security of machine learning models against adversarial samples provided by an embodiment of the present invention. Against the "strongest" attack, a false negative rate of up to 0% was achieved, with a false positive rate of approximately 6%. Increasing the threshold σ reduced the false positive rate but increased the false negative rate.

Another approach proposed by the applicant aims to prevent transferability by using an aggravated model to detect adversarial samples by comparison. Specifically, this approach relies on the fact that an aggravated model can be significantly different from its authentic counterpart, and that the differences between them should prevent "weak" adversarial samples from being misclassified. In contrast, embodiments of the present invention rely on the behavioral differences between authentic and adversarial samples when classified on an aggravated model with added triggers. This difference in approach leads to significant differences in results. While previous approaches are particularly effective against subtle attacks, they may be less effective against attacks optimized for transferability. On the other hand, as transferability improves, the behavioral differences when classified on an aggravated model also increase, making embodiments of the present invention more effective at detecting such attacks. These separate approaches may be used in a complementary manner to improve security against different types of attacks and achieve better overall security for machine learning computer systems and networks.

Improved protection:
A further improvement over previously proposed defenses, according to one embodiment of the present invention, is to rely on the last logit l instead of, or in addition to, using the classification output of the system as shown in Figure 3. In this embodiment, it is advantageously, but not necessarily, possible to determine whether the backdoored model outputs a backdoor class. In this improvement, the last layer of the model is discarded. The last layer of the model is called a softmax layer, and it converts the output of the neural network from real numbers to a probability distribution.
, which is used to map the model's true behavior to the vector l (which can potentially contain thousands of results).
It depends on a pool of genuine samples s _g that are used to compute s g. When a new sample s is received, the output l _s of this sample is then compared to the output of the genuine behavior to determine if the received sample is genuine.
is compared to.

The function "is diff" can be implemented in a number of ways. For example, L-distance can be used. Another possibility that can yield better results is to use an outlier detection system such as Local Outlier Factor (LOF), a method typically used to determine whether a given input from a set of inputs is an outlier based on the density of its nearest neighbors. Using LOF, improvements in accuracy have been demonstrated. Accuracy for subtle attacks ranges from 95% false negative rates (for the attack described in Kurakin, Alexey et al.) to 40% (for the attack described in Moosavi-Dezfooli, Seyed-Mohsen et al. and Goodfellow, Ian The accuracy of the strong attack remained unchanged at a 0% false negative rate (against the attacks described in Carlini, Nicholas, et al. and Madry, Aleksander, et al.), while the false negative rate of the optimized attack also decreased significantly from 80% to approximately 25%. Using further optimizations, it was possible to further improve accuracy to ensure further improvements in security. A subtle attack represents an attack strategy that minimizes adversarial perturbations, while a strong attack represents an attack strategy that optimizes the generation of high-confidence adversarial samples.

Examples of adversarial examples:
While the above discussion focuses solely on digital versions of attacks (e.g., digitally altering adversarial samples) due to the increased strength of the adversary, it has been shown that physical adversarial samples are also possible, and embodiments of the present invention are equally applicable to detecting such attacks. For example, a malicious party could use such an attack to trick an autonomous vehicle's algorithm into recognizing a stop sign as a different sign by adding a few minor modifications to the sign. The attacker's detailed process could involve generating a surrogate model of a traffic sign recognition model and investigating ways to alter the sign to result in misclassification. The attacker could then assess the success of the attack by renting or purchasing a vehicle containing the target autonomous driving system to examine how the software reacts to the altered signs. While this type of attack may not provide financial benefits to the attacker, it poses a significant public security risk and could potentially involve liability for the vehicle's manufacturer in the event of an accident.

Similarly, the use case for such attacks could be directed at facial recognition systems, where adversarial samples could be generated and used to either avoid recognition of genuine subjects (a confusion attack) or to falsely match samples with another identity (a deception attack). Such attacks could result in economic and/or personal harm, as well as the potential for breaching technical security systems, allowing unauthorized adversaries access to safeguards or equipment.

Accordingly, embodiments of the present invention provide the following improvements:
1. Improve the security of machine learning models and improve the application of machine learning models with enhanced security in the field of technology.
2. Utilize the output of the backdoored model to distinguish between adversarial and legitimate samples by using a reference pool of known authentic samples.
3. Using a trigger unknown to the adversary, create and utilize a backdoored variant of the machine learning model to detect adversarial samples by comparing the classification output of the adversarial samples, with the trigger attached and evaluated on the backdoored variant of the machine learning model, with the classification output of the authentic samples, with the trigger attached.
4. Using n separate triggers, generate and use N backdoored variants of the model, unknown to the adversary, to detect adversarial samples by examining the output of classification of those samples in the N backdoored variants.
5. Compared to existing defenses, the loss of accuracy (due to some genuine samples being falsely rejected) is reduced and insignificant.
6. Compared to existing defenses, it provides enhanced security against an adversary with knowledge of the defense.

According to one embodiment of the present invention, a method for improving the security of a machine learning model against adversarial examples comprises the following steps.
Setup phase:
- Receive a classification model M
- A detection phase that generates backdoored models M' ₁ , .., M' _N using random triggers t ₁ , .., t _N locally:
- When receiving samples to classify:
For each backdoored model M' ₁ , .., M' _N , classify sample s in backdoored model M' _i with trigger t _i (y _i ←M' _i (s+t _i )).
Use the set of outputs y _1..N to check if the output is not equal to the backdoor class (y _i ≠
) count the number of times.
Advanced detection: reject the sample if the number of misclassifications exceeds a threshold σ, flagging it as tampered with, otherwise outputting the results of the classification request to the legitimate model (M).
- Additional setup:
Select multiple genuine samples s _g. For each genuine sample s _g , calculate the logit l of each sample s in the backdoored model M' _i...N and obtain the logit output
Store in a set of
- Detection:
When you receive a sample s to be classified,
■ For each backdoored model M' _i..N , classify sample s in backdoored model M' _i with trigger t _i (y _i ← M' _i (s + t _i )) and extract logits as li. ■ For each backdoored model M' _i..N , extract the legitimate logits generated using the respective backdoored model.
Apply an outlier detection method (such as LOF) on the logits compared to the set of σ. When σ∈[0,1] is a predetermined threshold, if more backdoored models _Mi detect the logit vector l _i as an outlier than σN, the corresponding sample s is rejected. This mechanism is similar to the mechanism shown in FIG. 2 (in this example, the threshold σ is set to 0) in the sense that the number of backdoored models giving a particular output (d in FIG. 2) is not counted in this embodiment. In the example of FIG. 2, this particular output is a classification different from the target class, but in this advanced detection embodiment, the particular output is an outlier verdict. In both the embodiment of FIG. 2 and advanced detection, a sample is rejected if d>σN.

Embodiments of the present invention advantageously provide robustness against adaptive attacks by breaking the symmetric knowledge between the attacker and the defender. The trigger of the backdoored model acts as an unknown secret key to the attacker.

While embodiments of the present invention have been shown and described in detail in the drawings and foregoing description, such illustrations and descriptions are to be considered illustrative or exemplary and not restrictive. Those skilled in the art will understand that changes and modifications may be made within the scope of the following claims. In particular, the present invention further encompasses embodiments using any combination of features from the separate embodiments described above and below. In addition, statements characterizing the present invention herein refer to one embodiment of the present invention and not necessarily to all embodiments.

The terms used in the claims should be interpreted in the broadest possible manner consistent with the foregoing. For example, the use of articles such as "a" or "the" when introducing an element should not be construed as excluding a plurality of elements. Similarly, "or" recitations should be construed inclusively, and a recitation of "A or B" does not exclude "A and B" unless it is clear from the context or the preceding description that only one of A and B is intended. Furthermore, a recitation of "at least one of A, B, and C" should be interpreted as one or more of the group of elements consisting of A, B, and C, regardless of whether A, B, and C are related categorically, and should not be interpreted as requiring at least one of each of the listed elements A, B, and C. Furthermore, recitation of "A, B and/or C" or "at least one of A, B or C" should be interpreted to include any singular entity, e.g., A, from the listed elements, any subset, e.g., A and B, from the listed elements, or the entire list of elements A, B and C.

10 Setup Phase
12 Machine Learning Models
15 Backdoor model
20 Evaluation Stage
22 Samples
24 images
25 Trigger
30 Evaluation Stages
32 Authentic Samples
40 Method for generating backdoored models

Claims (15)

1. A method for securing a genuine machine learning model against adversarial examples , executed by one or more hardware processors, comprising :
a) adding a trigger to the sample to be classified;
b) classifying the sample tagged with the trigger using a backdoored model backdoored using the trigger;
c) determining whether the output of the backdoored model in step b) is identical to the backdoor class of the backdoored model and/or applying an outlier detection method that compares the logits from step b) with legitimate logits calculated using authentic samples applied to the backdoored model with the trigger added;
d) repeating steps a) to c) using different triggers and backdooring models associated with each of said different triggers;
e) comparing the number of times each of the outputs of the backdoored models was not the same as the backdoor class of each of the backdoored models with a predetermined threshold and/or comparing the difference determined by applying the outlier detection method with a difference threshold to determine whether the sample is an adversarial sample. classifying the untriggered sample using the genuine machine learning model as a result of a classification request for the sample if the number of times each of the outputs of the backdoored model is not the same as each of the backdoor classes of the backdoored model is less than or equal to the threshold;
and rejecting the sample as the adversarial sample if each of the outputs of the backdoored model was not the same as each of the backdoored classes of the backdoored model more times than the threshold. The method of claim 2, further comprising flagging the sample as tampered with if the number of times each of the outputs of the backdoored model is not the same as each of the backdoored classes of the backdoored model is greater than the threshold. The method of claim 3, wherein the threshold is zero. Each of the backdoored models is
generating each of the triggers as a pattern recognizable by the true machine learning model;
adding said respective triggers to a plurality of training samples;
changing the target class of the training sample to which each of the triggers has been added to each of the backdoor classes;
and training another version of the true machine learning model using the training samples with the respective triggers added. 6. The method of claim 5, wherein the training is performed until each of the backdoored models achieves an accuracy of 90% or greater , where the accuracy indicates the probability that separate samples with the same trigger are misclassified . The method of claim 5, wherein the genuine machine learning model and the version of the genuine machine learning model are each trained, and the step of training the version of the genuine machine learning model using the training samples with the respective triggers added is additional training to create the respective backdoored models from the genuine machine learning model. The method of claim 7, wherein the additional training includes training each of the triggers using authentic samples along with the added samples. The method of any one of claims 1 to 8, wherein the classifying step b) includes using the backdoored model to extract the logits for the classification of the sample to which the trigger has been added, the output class of the backdoored model is not used to determine whether the sample is the adversarial sample, and in step e) the logits from step b) are compared to a set of legitimate logits calculated using multiple genuine samples to which the respective triggers have been added and applied to each of the backdoored models. If the result of the outlier detection method for each of the logits is less than or equal to the difference threshold, then as a result of a classification request for the sample, classifying the untriggered sample using the genuine machine learning model;
and rejecting the sample as the adversarial sample if the result of the outlier detection method for each of the logits is greater than the difference threshold. The method of claim 10, wherein the outlier detection method uses a Local Outlier Factor algorithm. The method of any one of claims 1 to 11, wherein the true machine learning model is trained for image classification based on a neural network. A system for securing genuine machine learning models against adversarial examples, comprising:
a) adding a trigger to the sample to be classified;
b) classifying the sample tagged with the trigger using a backdoored model backdoored using the trigger;
c) determining whether the output of the backdoored model in step b) is identical to the backdoor class of the backdoored model and/or applying an outlier detection method that compares the logits from step b) with legitimate logits calculated using authentic samples applied to the backdoored model with the trigger added;
d) repeating steps a) through c) using different triggers; and
e) comparing the number of times each of the outputs of the backdoored models was not the same as the backdoor class of each of the backdoored models with a predetermined threshold and/or comparing the difference determined by applying the outlier detection method with a difference threshold to determine whether the sample is an adversarial sample;
1. A system comprising one or more hardware processors configured alone or in combination to execute : classifying the untriggered sample using the genuine machine learning model as a result of a classification request for the sample if the number of times each of the outputs of the backdoored model is not the same as each of the backdoor classes of the backdoored model is less than or equal to the threshold;
14. The system of claim 13, further configured to reject the sample as the adversarial sample if each of the outputs of the backdoored model was not the same as each of the backdoored classes of the backdoored model more times than the threshold. A tangible, non-transitory computer-readable medium having instructions that, when executed by one or more processors,
a) adding a trigger to the sample to be classified;
b) classifying the sample tagged with the trigger using a backdoored model backdoored using the trigger;
c) determining whether the output of the backdoored model in step b) is identical to the backdoor class of the backdoored model and/or applying an outlier detection method that compares the logits from step b) with legitimate logits calculated using authentic samples applied to the backdoored model with the trigger added;
d) repeating steps a) through c) using different triggers; and
e) comparing the number of times each of the outputs of the backdoored models was not the same as the backdoor class of each of the backdoored models with a predetermined threshold and/or comparing the difference determined by applying the outlier detection method with a difference threshold to determine whether the sample is an adversarial sample.