JP7630464B2 - EVALUATION DATA GENERATION DEVICE, SAFETY EVALUATION DEVICE, EVALUATION DATA GENERATION METHOD, AND EVALUATION DATA GENERATION PROGRAM - Google Patents

Description

The present invention relates to a method for evaluating the security of machine learning systems against backdoor attacks.

In recent years, Non-Patent Document 1 has presented a method of attacking machine learning systems, such as image classification, called a backdoor attack. In a backdoor attack, data (images) called triggers are attached to data belonging to a source class, and the label is changed to a target class, generating poisonous data. This is an attack that aims to have the machine learning system misclassify only source class images, to which a trigger has been added, into the target class by having the model learn training data that includes the poisonous data.

Subsequently, Non-Patent Document 2 presented a more advanced method called a clean label backdoor attack, which performs a backdoor attack without manipulating the label of the poisoned data.
Furthermore, Non-Patent Document 3 discloses a backdoor attack that uses a clean label method and in which the trigger added to the source data is invisible (difficult to identify). In this method, the trigger of the poison data added to the training data is invisible, making it very difficult to detect the poison data.

Tianyu Gu, Brendan Dolan-Gavitt, and Siddharth Garg. Badnets: Identifying vulnerabilities in the machine learning model supply chain. arXiv preprint arXiv:1708.06733, 2017. Alexander Turner, Dimitris Tsipras, and Aleksander Madry. Label-consistent backdoor attacks. arXiv preprint arXiv:1912.02771, 2019. Aniruddha Saha, Akshayvarun Subramanya, and Hamed Pirsiavash. Hidden trigger backdoor attacks. In Proceedings of the AAAI conference on artificial intelligence, Vol. 34, pp. 11957-11965, 2020.

When evaluating the security of machine learning systems, it is important to consider more powerful attack methods in order to assume worst-case scenarios and conduct more rigorous security evaluations.
However, in the backdoor attack of Non-Patent Document 3, the target-labeled poison data to be mixed into the training data was made invisible by generating it using a gradient optimization method, but when testing (evaluating) the trained model, the source-labeled image was given a visible trigger (patch image) to make it the poison data. Therefore, by taking measures such as anomaly detection during testing, it was possible to easily detect the poison data.
Thus, backdoor attacks in which the trigger is invisible in both the training data and the test data have not been considered previously.

The present invention aims to provide an evaluation data generation device, a safety evaluation device, an evaluation data generation method, and an evaluation data generation program for more rigorously evaluating the safety of a machine learning system based on more powerful attack methods than conventional methods.

The evaluation data generation device according to the present invention includes a poison data generation unit that generates a poison data group of a first class that is updated to minimize the loss of distance in feature space with a second class data group by adding a trigger with a variation amount that is difficult to distinguish to the first class data group; a test data output unit that inputs a source class data group in a backdoor attack as the first class data group and a target class data group as the second class data group to the poison data generation unit and outputs the generated poison source data group as test data for a classification system to be evaluated; and a training data output unit that inputs the target class data group as the first class data group and the poison source data group as the second class data group to the poison data generation unit and outputs the generated poison target data group as training data for the classification system.

The test data output unit may uniformly output poison data groups for each source class, which are generated using as input a data group for each of the multiple source classes and a data group for a single target class, and the training data output unit may uniformly output poison data groups corresponding to each source class, which are generated using as input a data group for the single target class and a poison data group for each of the multiple source classes.

The loss may be defined as the sum of the pairwise distances between classes that minimizes the distance in the feature space.

The classification system may be a system that classifies images into multiple classes using a machine learning model.

The safety evaluation device according to the present invention includes the poison data generation unit, the test data output unit, and the training data output unit, a model generation unit that mixes the poison target data group with the training data of the classification system to generate a poisoned model, and an evaluation unit that measures the misclassification rate of the poisoned model using the poison source data group as test data and outputs an evaluation value of safety against attacks based on the misclassification rate.

The evaluation data generation method according to the present invention uses a poison data generation algorithm that generates a first class of poison data group updated to minimize the loss of distance in feature space with a second class data group by adding a trigger with a variation amount that is difficult to distinguish to the first class data group, and inputs a source class data group in a backdoor attack as the first class data group and a target class data group as the second class data group into the poison data generation algorithm, and outputs the generated poison source data group as test data for the classification system to be evaluated, and executes a training data output step in which the target class data group is input as the first class data group and the poison source data group as the second class data group into the poison data generation algorithm, and outputs the generated poison target data group as training data for the classification system.

The evaluation data generation program according to the present invention is for causing a computer to function as the evaluation data generation device.

The present invention allows for a more rigorous evaluation of the security of machine learning systems based on backdoor attacks whose triggers are invisible in both training data and test data.

FIG. 2 is a diagram illustrating a functional configuration of a safety evaluation device according to an embodiment. FIG. 1 is a diagram showing a procedure for generating training data and test data for safety evaluation in a conventional safety evaluation method. FIG. 2 is a diagram showing a procedure for generating training data and test data for safety evaluation in the safety evaluation method according to the embodiment. 11 is a diagram illustrating a Generate-Invisible-Poison function executed by a poison data generation unit in an embodiment. FIG. 11 is a diagram illustrating a Find-Closest-Pair function executed by a poison data generation unit in an embodiment.

An example of an embodiment of the present invention will now be described.
In this embodiment, a backdoor attack against a classification system using a machine learning model is assumed, and it is possible to evaluate security against clean label backdoor attacks whose triggers are difficult to identify not only during training but also during testing.
Here, we use a system that classifies images into multiple classes using a machine learning model as an example of the classification system to be evaluated, and present an attack algorithm that makes the trigger difficult to identify, i.e., invisible.

FIG. 1 is a diagram showing the functional configuration of a safety evaluation device 1 according to this embodiment.
The safety evaluation device 1 is an information processing device (computer) including a control unit 10, a storage unit 20, and various input/output interfaces.
The safety evaluation device 1 is configured to output the evaluation results of the safety of the classification system against backdoor attacks, but it may also be configured to output evaluation data (training data and test data), and the actual evaluation steps, which are training and testing of the machine learning model, may be performed by another processing device.

The control unit 10 is a part that controls the entire safety evaluation device 1, and realizes each function in this embodiment by appropriately reading and executing various programs stored in the storage unit 20. The control unit 10 may be a CPU.
Specifically, the control unit 10 includes a poison data generation unit 11 , a test data output unit 12 , a training data output unit 13 , a model generation unit 14 , and an evaluation unit 15 .

The storage unit 20 is a storage area for various programs for causing the hardware group to function as the safety evaluation device 1, various data, etc., and may be a ROM, a RAM, a flash memory, a hard disk drive (HDD), etc.
Specifically, the memory unit 20 stores an evaluation data generation program and a safety evaluation program, as well as parameters of the machine learning model to be evaluated, sample data for each class, generated training data and test data, etc.

The poison data generation unit 11 generates a first class of poison data group that is updated to minimize the loss in the feature space with respect to the distance of the features in the internal layer of the machine learning model, that is, with respect to the first class of data group (e.g., images), by adding a trigger (e.g., an invisible patch image) with a variation amount that is difficult to identify to the first class of data group (e.g., images).

Here, the loss is defined as the sum of pairwise distances that minimizes the distance in feature space between classes. The distance may be, for example, the L2 distance or its squared distance.
In addition, as a method for minimizing losses, for example, gradient optimization is used, and using an algorithm described below, the poison data generation unit 11 generates a group of poison data in which the distance in feature space is brought closer to the second class data within a predetermined variation from the first class data.

The test data output unit 12 inputs the source class data group in the backdoor attack as a first class data group and the target class data group as a second class data group to the poison data generation unit 11, and outputs the generated poison source data group as test data for the classification system to be evaluated.

The training data output unit 13 inputs the target class data group as a first class data group and the poison source data group output by the test data output unit 12 as a second class data group to the poison data generation unit 11, and outputs the generated poison target data group as training data for the classification system.

The classification system may perform multi-class classification, and multiple source classes may be set for a backdoor attack. In this case, the target class is also single, and an attack is assumed in which data (images) of multiple source classes to which a trigger is attached are misclassified into the target class.

In this case, the test data output unit 12 uniformly outputs the poison data groups of each source class that are generated using the data groups of each of the multiple source classes and the data group of a single target class as input.
Furthermore, the training data output unit 13 uniformly outputs poison data groups corresponding to each source class, which are generated using a data group of a single target class and each poison data group of a plurality of source classes as input.

The model generation unit 14 mixes the poison target data group output by the training data output unit 13 into the training data of the classification system to generate a contamination model.

The evaluation unit 15 measures the misclassification rate of the tainted model using the poison source data output by the test data output unit 12 as test data, and outputs an evaluation value of safety against attacks based on this misclassification rate.

FIG. 2 is a diagram showing a procedure for generating training data and test data for safety evaluation in a conventional safety evaluation method, for comparison with this embodiment.
Here, we focus on clean-label backdoor attacks against image classification systems.

First, a visible patch image, which acts as a trigger, is added to the source image to create test data.
Next, by optimizing the relationship between this test data and the target image, a poison target image in which the trigger is invisible is generated, and this is used as training data.

FIG. 3 is a diagram showing a procedure for generating training data and test data for safety evaluation in the safety evaluation method according to this embodiment.
Compared to the conventional method, the trigger is also invisible for the poison data on the source side that is the test data.

In step 1, the test data output unit 12 creates invisible source-side poison data using a gradient optimization algorithm, and sets it as test data. This algorithm takes K ₁ randomly extracted source images and K ₁ target images as input, and outputs a poison source image in which K ₁ triggers are invisible.

In step 2, the training data output unit 13 generates a poison target image in which _K2 triggers are invisible, as training data, using the same gradient optimization algorithm as in step 1. Here, the randomly extracted _K2 target images and the _K1 poison source image generated in step 1 are input to the gradient optimization algorithm, and a poison target image in which _K2 triggers are invisible is output.

Once the evaluation data is generated in this manner, a tainted model is generated by mixing the _K2 poison target images generated in step 2 into the training data. By measuring the rate at which the poison source images generated in step 1 are misclassified into the target class (attack success rate) when they are input to the tainted model, the vulnerability of the classification model to be evaluated against the invisible clean label backdoor attack algorithm can be measured.

FIG. 4 is a diagram illustrating an algorithm executed by the poison data generating unit 11 in this embodiment.
The above-mentioned steps 1 and 2 are implemented using this Generate-Invisible-Poison function (Algorithm 1).
This function receives data x _a of class a and data x _b of class b as input, and outputs poison data ^x _a of class a using the gradient optimization method.

First line: the poison data generation unit 11 initializes K pieces of poison data ^x _a ⁽ⁱ⁾ with K pieces of data x _a ⁽ⁱ⁾ (1≦i≦K) of class a.
Line 2: The poison data generating unit 11 repeats lines 3 to 6 a preset number of times (n times).
Line 3: The poison data generation unit 11 calculates a one-to-one mapping m[i] between ^x _a and x _b using a Find-Closest-Pair function described later.

Here, m[i] is selected so as to minimize the L2 distance ∥f(^ _xa ⁽ⁱ⁾ )-f( _xb ^(m[i] ) ₎ ∥2 in the model's feature space f() for each pair (^ _xa ⁽ⁱ⁾ _, xb ^(m[i]) . In this way, pairs of data of class a and class b that are similar in the model's feature space are obtained.

Line 4: The poison data generation unit 11 calculates the sum L of losses (squared distances) in the feature space for the K pairs obtained in line 3.
Line 5: The poison data generator 11 calculates the gradient ∇^x _a (i ⁾ L of L in the ^x _a (i ⁾ direction, and updates the poison data value ^x _a ⁽ⁱ⁾ so that the loss L becomes smaller, where η is the learning rate.
Line 6: This is a step for ensuring the invisibility of the poison data. If the distance between each element of the poison data ^x _a ⁽ⁱ⁾ and the original data x _a ⁽ⁱ⁾ of label a becomes larger than ε, the poison data generation unit 11 corrects it so that it becomes exactly ε.
Line 7: The poison data generation unit 11 outputs K pieces of poison data ^x _a of class a whose triggers are invisible.

FIG. 5 is a diagram illustrating the Find-Closest-Pair function (Algorithm 2) executed by the poison data generation unit 11 in this embodiment.
As described above, this function is used to find pairs of data of classes a and b that are similar in the feature space of the model, and the specific processing procedure is as follows.

First line: The poison data generation unit 11 defines a set J of K elements in [1, . . . , K].
Line 2: The poison data generation unit 11 repeats lines 3 to 11 from 1 to K.
Line 3: The poison data generation unit 11 initializes the minimum value of the squared distance, dist _min , to ∞.
Line 4: The poison data generation unit 11 initializes j _min to ∞.

Line 5: The poison data generation unit 11 repeats lines 6 to 9 for element j included in J.
Line 6: The poison data generation unit 11 calculates the square of the L2 distance ∥f(^ _xa ⁽ⁱ⁾⁾ -f( _xb ^(m[i]) ₎ ∥2 between the poison data ^ _xa ( ⁱ⁾ of the i-th class a and the data xb ^(j) of the j-th class _b .
Line 7: If the calculated square distance dist is smaller than dist _min , the poison data generation unit 11 executes lines 8 and 9.
Line 8: The poison data generation unit 11 updates dist _min to dist.
Line 9: The poison data generation unit 11 stores in j _min the j that currently minimizes the squared distance.

Line 10: The poison data generator 11 stores j _min in the mapping m[i].
Line 11: The poison data generation unit 11 deletes j _min from the set J.
Line 12: The poison data generator 11 outputs the mapping m.

Next, a method for implementing the above steps 1 and 2 using the Generate-Invisible-Poison function (Algorithm 1) will be described.
First, the test data output unit 12 calls Algorithm 1 (the poison data generation unit 11) with parameters a as the source class and b as the target class. In this way, Algorithm 1 outputs K ₁ poison source images with invisible triggers.

Next, the training data output unit 13 calls Algorithm 1 (poison data generation unit 11) with the parameter x _a as K ₂ target images and x _b as K ₁ poison source images created by the test data output unit 12. In this way, Algorithm 1 outputs K ₂ poison target images with invisible triggers.

If _K1 < _K2 , the poison source image is duplicated by _K2 - _K1 to make the number of images in both classes equal. If _K1 > _K2 , the image is duplicated in a similar manner to make the number of images in both classes equal. In this way, both the target poison image used for training the model and the source poison image used for testing (evaluating) the model can be generated.

In addition, when assuming a multi-source attack against a multi-class classification system, i.e. an attack that misclassifies multiple source classes into a single target class, the evaluation method can be configured as follows.

The number of source classes is n (respectively written as s ₁ , s ₂ , ..., s _n ), and the number of target classes is one (written as t). In this case, the test data output unit 12 calls Algorithm 1 (poison data generation unit 11) n times with a as s _i (1≦i≦n) class and b as t class. In this way, K ₁ poison source images (total of nK ₁ ) with invisible triggers are obtained for each source class s _i .

Next, the training data output unit 13 calls Algorithm 1 (the poison data generation unit 11) by inputting K ₁ poison source images of each source class s _i and K ₂ images of the target class t. In this way, K ₂ poison target images (a total of nK ₂ images) with invisible triggers are obtained for each source class s _i .

When mixing poison target images into training data, it is preferable to mix poison target images associated with each source class s _i evenly. Therefore, when the number of poison data to be mixed into training data is p, the model generation unit 14 may mix p/n poison target images from each source class. Methods for selecting p/n poison target images from the K ₂ poison target images of each source class include, for example, a random selection method, a method of selecting the top p/n images selected by the Find-Closest-Pair function, and the like.
Furthermore, when evaluating security against attacks, the evaluation unit 15 inputs poisoned source images from each source class s _i evenly to the contaminated model, and measures the rate of misclassification into the target class.

According to this embodiment, the safety evaluation device 1 receives a source class data group and a target class data group in a backdoor attack as input, generates a poison source data group so as to minimize the loss of distance in the feature space, and outputs it as test data for the classification system to be evaluated. Also, the safety evaluation device 1 receives a target class data group and a poison source data group as input, generates a poison target data group so as to minimize the loss of distance in the feature space, and outputs it as training data for the classification system.
Therefore, the safety evaluation device 1 can output evaluation data assuming a clean-label type backdoor attack in which the trigger is invisible in both the training data and the test data, thereby enabling a more rigorous evaluation of the safety of the machine learning system.

In addition, the safety evaluation device 1 can equally output, as test data, groups of poison data for each source class generated using as input a data group for each of a plurality of source classes and a data group for a single target class, and can equally output, as training data, groups of poison target data corresponding to each source class generated using as input a data group for a single target class and a poison data group for each of a plurality of source classes.
As a result, the safety evaluation device 1 can output evaluation data that assumes multi-class, multi-source backdoor attacks, not just binary classification, enabling the safety evaluation of machine learning systems against more powerful attacks.

When using gradient optimization to generate poison data, the safety evaluation device 1 defines the loss as the sum of the distances for each pair that minimizes the distance in feature space between classes. This allows the safety evaluation device 1 to appropriately generate poison data with invisible triggers.

In this embodiment, a system that classifies images into multiple classes using a machine learning model is used as an example of a classification system, but the present invention is not limited to this and can be applied to various classification systems in which clean-label type backdoor attacks using triggers that are difficult to identify (e.g., invisible) are anticipated.

This will, for example, enable the creation of a safe classification system based on more rigorous evaluation, which will contribute to Goal 9 of the United Nations-led Sustainable Development Goals (SDGs), which is to "build resilient infrastructure, promote sustainable industrialization and foster innovation."

Although the embodiments of the present invention have been described above, the present invention is not limited to the above-described embodiments. Furthermore, the effects described in the above-described embodiments are merely a list of the most favorable effects resulting from the present invention, and the effects of the present invention are not limited to those described in the embodiments.

The safety evaluation method using the safety evaluation device 1 is realized by software. When realized by software, the programs constituting this software are installed in an information processing device (computer). These programs may be recorded on removable media such as CD-ROM and distributed to users, or may be distributed by being downloaded to the user's computer via a network. Furthermore, these programs may be provided to the user's computer as a web service via a network without being downloaded.

REFERENCE SIGNS LIST 1 Safety evaluation device 10 Control unit 11 Toxic data generation unit 12 Test data output unit 13 Training data output unit 14 Model generation unit 15 Evaluation unit 20 Storage unit

Claims (7)

a poison data generation unit that generates a poison data group of the first class that is updated so as to minimize a loss in distance in feature space with a data group of a second class by adding a trigger of a variation amount that is difficult to distinguish to the data group of the first class;
a test data output unit that inputs a source class data group in a backdoor attack as the first class data group and a target class data group as the second class data group to the poison data generation unit, and outputs the generated poison source data group as test data for a classification system to be evaluated;
an evaluation data generation device comprising: a training data output unit that inputs the target class data group as the first class data group and the poison source data group as the second class data group into the poison data generation unit, and outputs the generated poison target data group as training data for the classification system. the test data output unit outputs a poison data group for each source class equally, the poison data group being generated by inputting a data group for each of a plurality of source classes and a data group for a single target class;
The evaluation data generation device according to claim 1, wherein the training data output unit uniformly outputs poison data groups corresponding to each source class, the poison data groups being generated by inputting a data group of the single target class and a poison data group of each of the multiple source classes. The evaluation data generating device according to claim 1 or 2, wherein the loss is defined as a value obtained by summing up the distances between classes in the feature space for each pair that minimizes the distance. The evaluation data generation device according to claim 1 or 2, wherein the classification system is a system that classifies images into multiple classes using a machine learning model. A poison data generating unit, a test data output unit, and a training data output unit according to claim 1 or 2;
a model generation unit that mixes the poison target data group with training data of the classification system to generate a contamination model;
and an evaluation unit that measures a misclassification rate by the tainted model using the group of poison source data as test data, and outputs an evaluation value of safety against attacks based on the misclassification rate. A poison data generation algorithm is used to generate a poison data group of the first class, which is updated so as to minimize a loss in distance in feature space between the first class data group and the second class data group by adding a trigger of a variation amount that is difficult to distinguish to the first class data group;
a test data output step of inputting a source class data group in a backdoor attack as the first class data group and a target class data group as the second class data group into the poison data generation algorithm, and outputting the generated poison source data group as test data for a classification system to be evaluated;
A method for generating evaluation data executed by a computer, comprising: inputting the target class data group as the first class data group and the poison source data group as the second class data group into the poison data generation algorithm, and outputting the generated poison target data group as training data for the classification system. An evaluation data generation program for causing a computer to function as the evaluation data generation device according to claim 1 or 2.

Images