JP7564074B2 - Adversarial data detection device and adversarial data detection method - Google Patents

Description

The present invention relates to an adversarial data detection device and an adversarial data detection method.

AI (Artificial Intelligence) is becoming more widespread in various fields such as finance, medicine, and manufacturing, and there are an increasing number of situations where AI handles sensitive information. Meanwhile, attacks that exploit vulnerabilities specific to AI have been reported.

One example of an attack that exploits vulnerabilities specific to AI is the Adversarial Example (AE) attack, in which minute noise that is unnoticeable to humans is added to data, causing the AI to make a false judgment. In what follows, data with minute noise added will be referred to as adversarial data, and throwing adversarial data at an AI will be referred to as an AE attack.

As a countermeasure against AE attacks, technology has been proposed that detects AE attacks by capturing the process leading up to the generation of adversarial data and the characteristics of the adversarial data. Non-Patent Document 1 is a background technology for detecting AE attacks. The technology described in Non-Patent Document 1 detects AE attacks by utilizing changes in the output results when random minute noise is added to input data.

Jingyi Wang and 3 others, “Detecting Adversarial Samples for Deep Neural Networks through Mutation Testing.”, [online], NIPS May 2018 , [Retrieved March 26, 2021], Internet <https://arxiv .org/pdf/1805.05010.pdf>

The technology for detecting AE attacks in Non-Patent Document 1 is not sufficiently secure in a situation where an attacker knows information about a method for detecting adversarial data, and the attacker can intentionally generate adversarial data that evades the detection described in Non-Patent Document 1. Therefore, one aspect of the present invention detects adversarial data with high accuracy even in a situation where an attacker knows information about a method for detecting adversarial data, thereby maintaining security.

In order to solve the above problem, one aspect of the present invention employs the following configuration. The adversarial data detection device has a processor and a memory, and the memory holds evaluation target data and a model that outputs an evaluation value when data is input. The processor inputs the evaluation target data to the model, calculates an evaluation value of the evaluation target data, adds different noises to the evaluation target data to generate multiple extended evaluation target data, and executes a first judgment process based on the multiple extended evaluation target data. In the first judgment process, for each of the multiple extended evaluation target data, noise is added to the extended evaluation target data until the evaluation value of the extended evaluation target data does not match the evaluation value of the evaluation target data, and judges whether the evaluation target data is adversarial data based on the bias of the evaluation value of each of the updated extended evaluation target data.

According to one aspect of the present invention, it is possible to detect adversarial data with high accuracy even in a situation where an attacker knows information about methods for detecting adversarial data, thereby maintaining security.

Issues, configurations, and advantages other than those described above will become clear from the description of the embodiments below.

FIG. 1 is a block diagram illustrating a configuration example of an adversarial data detection system according to a first embodiment. 4 is a block diagram showing an example of the hardware configuration of a noise addition server in the first embodiment. FIG. FIG. 2 is a block diagram showing an example of a hardware configuration of an evaluation server according to the first embodiment. 3 is a block diagram showing an example of a hardware configuration of a detection server according to the first embodiment; FIG. 2 is a block diagram showing an example of a hardware configuration of a data sending terminal according to the first embodiment; FIG. 11 is a sequence diagram illustrating an example of an adversarial data detection process in the first embodiment. 13 is a flowchart illustrating an example of an extended evaluation target data update process in the first embodiment. 11 is a flowchart illustrating an example of a hostile data determination process according to the first embodiment. FIG. 2 is an explanatory diagram illustrating an example of an overview of an adversarial data detection process in the first embodiment. FIG. 11 is a sequence diagram illustrating an example of an adversarial data detection process in the second embodiment. 13 is a flowchart illustrating an example of a hostile data determination process according to the second embodiment. FIG. 13 is an explanatory diagram illustrating an example of input and output data of the adversarial data detection system in the third embodiment.

Below, an embodiment of the present invention will be described with reference to the attached drawings. In this embodiment, the same components are generally given the same reference numerals, and repeated description will be omitted. It should be noted that this embodiment is merely one example for realizing the present invention, and does not limit the technical scope of the present invention. This embodiment describes a system that determines whether data received from a data sender is data generated by an AE (Adversarial Example) attack, and transmits a processing result to the data receiver according to the determination result.

Figure 1 is a block diagram showing an example of the configuration of an adversarial data detection system. The adversarial data detection system includes, for example, a data transmission terminal 400 that holds evaluation target data, a noise addition server 100 that adds noise to the evaluation target data, an evaluation server 200 that calculates an evaluation value for the evaluation target data, a detection server 300 that determines whether the evaluation target data is data that has been subjected to an AE attack, and a data receiving terminal 500 that receives the evaluation value for the evaluation target data.

Detailed configurations of the data transmission terminal 400, the noise addition server 100, the evaluation server 200, the detection server 300, and the data receiving terminal 500 will be described later. The data transmission terminal 400, the noise addition server 100, the evaluation server 200, the detection server 300, and the data receiving terminal 500 transmit and receive information to and from each other via a network 600 such as the Internet.

In addition, some or all of the devices included in the adversarial data detection system may be integrated. For example, at least some of the noise addition server 100, the evaluation server 200, and the detection server 300 may be integrated, or the data transmission terminal 400 and the data receiving terminal 500 may be integrated.

Figure 2 is a block diagram showing an example of the hardware configuration of the noise addition server 100. The noise addition server 100 is composed of a computer having a processor (CPU) 101, auxiliary storage device 102, memory 103, display device 105, input/output interface 106, and communication interface 107, which are connected to each other by an internal signal line 104 such as a bus.

The processor 101 executes programs stored in the memory 103. The memory 103 includes a ROM, which is a non-volatile, non-temporary storage element, and a RAM, which is a volatile storage element. The ROM stores immutable programs (e.g., BIOS) and the like. The RAM is a high-speed, volatile storage element such as a DRAM (Dynamic Random Access Memory), and temporarily stores programs executed by the processor 101 and data used when the programs are executed.

The auxiliary storage device 102 is, for example, a large-capacity, non-volatile, non-transient storage device such as a magnetic storage device (HDD) or a flash memory (SSD), and stores the programs executed by the processor 101 and data used when the programs are executed. That is, the programs are read from the auxiliary storage device 102, loaded into the memory 103, and executed by the processor 101.

The input/output interface 106 is an interface to which a keyboard, mouse, etc. are connected and which receives input from an operator. The input/output interface 106 is also an interface to which a display device 105, a printer, etc. are connected and which outputs the results of program execution in a format that can be viewed by the operator. The display device 105 displays the results of program execution output from the input/output interface 106.

The communication interface 107 is a network interface device that controls communication with other devices according to a specific protocol. The communication interface 107 may also include a serial interface such as a Universal Serial Bus (USB).

A part or all of the program executed by the processor 101 may be provided to the noise addition server 100 from removable media (CD-ROM, flash memory, etc.) which is a non-transitory storage medium readable by the computer, or from an external computer equipped with a non-transitory storage device connected via the network 600, and stored in the non-volatile auxiliary storage device 102 which is a non-transitory storage medium. For this reason, the noise addition server 100 may have an interface for reading data from removable media. This also applies to the evaluation server 200, the detection server 300, the data transmission terminal 400, and the data receiving terminal 500.

The noise addition server 100 is a computer system configured on one physical computer or on multiple logically or physically configured computers, and may operate in separate threads on the same computer, or may operate on a virtual computer constructed on multiple physical computer resources. The same applies to the evaluation server 200, detection server 300, data transmission terminal 400, and data reception terminal 500.

The processor 101 includes, for example, a data expansion unit 111 and a noise addition unit 112. For example, the processor 101 functions as the data expansion unit 111 by operating according to a data expansion program loaded into the memory 103, and functions as the noise addition unit 112 by operating according to a noise addition program loaded into the memory 103.

The data extension unit 111, for example, adds noise to the evaluation target data to generate extended evaluation target data. The noise addition unit 112 updates the extended evaluation target data by adding noise to the extended evaluation target data.

Figure 3 is a block diagram showing an example of the hardware configuration of the evaluation server 200. The evaluation server 200 is composed of a computer having a processor (CPU) 201, an auxiliary storage device 202, a memory 203, a display device 205, an input/output interface 206, and a communication interface 207, which are connected to each other by an internal signal line 204 such as a bus.

The hardware description of the processor 201, auxiliary storage device 202, memory 203, internal signal line 204, display device 205, input/output interface 206, and communication interface 207 is the same as the hardware description of the processor 101, auxiliary storage device 102, memory 103, internal signal line 104, display device 105, input/output interface 106, and communication interface 107, so the description will be omitted.

The processor 201 includes an evaluation unit 211. For example, the processor 201 functions as the evaluation unit 211 by operating according to an evaluation program loaded in the memory 203.
The evaluation unit 211 calculates evaluation values for the evaluation target data and the extended evaluation target data.

Figure 4 is a block diagram showing an example of the hardware configuration of the detection server 300. The detection server 300 is configured, for example, by a computer having a processor (CPU) 301, auxiliary storage device 302, memory 303, display device 305, input/output interface 306, and communication interface 307, all of which are connected to each other by an internal signal line 304 such as a bus.

The hardware description of the processor 301, auxiliary storage device 302, memory 303, internal signal line 304, display device 305, input/output interface 306, and communication interface 307 is the same as the hardware description of the processor 101, auxiliary storage device 102, memory 103, internal signal line 104, display device 105, input/output interface 106, and communication interface 107, so the description will be omitted.

The processor 301 includes a first detection unit 311 and a second detection unit 312. For example, the processor 301 functions as the first detection unit 311 by operating according to a first detection program loaded into the memory 303, and functions as the second detection unit 312 by operating according to a second detection program loaded into the memory 303.

The first detection unit 311 makes a final determination as to whether the data to be evaluated is hostile data (data subjected to an AE attack). The second detection unit 312 determines whether the data to be evaluated is hostile data or has the possibility of being hostile data, and requests the first detection unit 311 to make a final determination if it has determined that the data is likely to be hostile data. Note that the processor 301 of the first embodiment does not need to include the second detection unit 312.

Figure 5 is a block diagram showing an example of the hardware configuration of the data transmission terminal 400. The data transmission terminal 400 is composed of a computer having a processor (CPU) 401, an auxiliary storage device 402, a memory 403, a display device 405, an input/output interface 406, and a communication interface 407, which are connected to each other by an internal signal line 404 such as a bus.

The hardware description of the processor 401, auxiliary storage device 402, memory 403, internal signal line 404, display device 405, input/output interface 406, and communication interface 407 is the same as the hardware description of the processor 101, auxiliary storage device 102, memory 103, internal signal line 104, display device 105, input/output interface 106, and communication interface 107, so the description will be omitted.

The auxiliary storage device 402 of the data transmission terminal 400 holds the evaluation target data 421. The evaluation target data 421 may be stored in the memory 403 of the data transmission terminal 400.

The hardware configuration example of the data receiving terminal 500 is similar to the hardware configuration example of the data sending terminal 400 in Figure 5, so the explanation is omitted.

Figure 6 is a sequence diagram showing an example of the hostile data detection process. First, the data transmission terminal 400 transmits evaluation target data to the noise-added server 100 (S601). Image data is an example of evaluation target data.

The data expansion unit 111 generates multiple pieces of extended evaluation target data by performing data expansion on the received evaluation target data (S602). An example of the processing of step S602 will be described. For example, a parameter n indicating the number of extended evaluation target data to be generated is set in advance and stored in the memory 103. The data expansion unit 111 generates n different types of noise, for example, and adds each of the n types of noise to one piece of evaluation target data to generate n pieces of extended evaluation target data. The noise generated by the data expansion unit 111 is, for example, a numerical vector with the same number of dimensions as the evaluation target data. In addition, the data expansion unit 111 generates noise according to a predetermined probability distribution, such as a Gaussian distribution.

The noise addition unit 112 and the evaluation unit 211 communicate with each other to calculate an evaluation value for the evaluation target data, calculate an evaluation value for the extended evaluation target data, add noise to the extended evaluation target data generated in step S602, and calculate an evaluation value for the extended evaluation target data with noise added (S603).

For example, the noise addition unit 112 updates each of the n pieces of extended evaluation target data by generating and adding a different noise to each of the n pieces of extended evaluation target data, and the evaluation unit 211 calculates an evaluation value for each of the n pieces of extended data after updating. Note that the noise addition unit 112 may generate noise according to a predetermined probability distribution such as a Gaussian distribution, a uniform distribution, or a Laplace distribution, or may generate noise that intentionally changes the evaluation value calculated by the evaluation unit 211. As noise that intentionally changes the evaluation value, for example, noise generated by a method similar to that used for noise generated when creating adversarial data is used.

In this embodiment, the evaluation unit 211, for example, inputs the updated n pieces of extended evaluation target data into a trained model generated using deep learning, and calculates the class obtained as an evaluation value. Note that the trained model may be a trained model with improved robustness by training data that has been resized, padded, or otherwise processed when performing deep learning. Note that this trained model is, for example, stored in advance in the memory 203.

The update process for each of the n pieces of extended evaluation target data in step S603 and the calculation process for the evaluation value for each of the n pieces of extended evaluation target data after the update are repeated until a predetermined condition is satisfied. The evaluation unit 211 transmits the evaluation value for each piece of extended evaluation target data finally calculated in step S603 to the detection server 300 (S604). Details of the processes in steps S603 and S604 will be described later with reference to FIG. 7.

The first detection unit 311 uses the evaluation value received in step S604 to determine whether the evaluation target data transmitted in step S601 is hostile data (S605). Details of the processing in step S605 will be described later with reference to FIG. 8.

When the first detection unit 311 determines that the evaluation target data is hostile data (S605: Yes), it does not transmit the evaluation value of the evaluation target data to the data sending terminal 400, and transmits a notification (determination result: reject) to the data sending terminal 400 indicating that the transmission of the evaluation value is rejected (S606). When the first detection unit 311 determines that the evaluation target data is not hostile data (S605: No), it transmits the evaluation value of the evaluation target data to the data receiving terminal 500 (S607), and the hostile data detection process ends.

If the first detection unit 311 determines in step S803 that the evaluation target data is hostile data, it may transmit the evaluation value of each of the extended evaluation target data finally calculated in step S603 to the data receiving terminal 500 in step S606 without transmitting a notification (determination result: reject) to the data sending terminal 400 indicating that the transmission of the evaluation value is rejected.

Figure 7 is a flowchart showing an example of the extended evaluation target data update process in steps S603 and S604. The evaluation unit 211 calculates an evaluation value Eval of the evaluation target data D received in step S601 (S701). In other words, the evaluation unit 211 calculates the class of the evaluation target data D obtained by inputting the evaluation target data D into the trained model described above as the evaluation value Eval.

The evaluation unit 211 executes the following steps S702 to S705 for each of i=1, ..., n, starting from i=1. The evaluation unit 211 calculates an evaluation value AEEval _i of D _i included in n pieces of extended evaluation target data {D ₁ , ..., D _n } (S702). The evaluation unit 211 determines whether Eval=AEEval _i (S703).

When the evaluation unit 211 determines that Eval=AEEval _i (S703: Yes), the noise addition unit 112 updates _Di by adding noise to _Di (S704), and the process returns to step S702 to recalculate the evaluation value for the updated _Di. Note that the method of adding noise in step S704 may be the same as or different from the method of adding noise in step S602.

If the evaluation unit 211 determines that Eval=AEEval _i is not satisfied (S703: No), the evaluation unit 211 stores the evaluation value AEEval _i calculated in the most recent step S902 in, for example, the memory 203 (S705), increments the value of i, and returns to step S902.

Note that Eval=AEEval _i indicates that the class of the evaluation target data D and the class of _Di match, that is, the class has not changed even though noise has been added to D when it was generated. On the other hand, if Eval=AEEval _i is not true, the class of the evaluation target data D and the class of _Di do not match, that is, noise has been added to D and the class has changed.

In addition, if the evaluation unit 211 determines in step S705 that Eval = AEEval _i even after the processing of step S704 has been performed a predetermined number of times or more for _Di , it may end the processing of steps S702 to S704 for this _Di (i.e., without calculating a final evaluation value for this _Di ) and increment i, or it may send an error to the data sending terminal 400 and end the hostile data detection processing.

When the process of step S705 is completed for each of i=1, . . . , n, the evaluation unit 211 transmits AEEval ₁ , . . . , AEEval _n stored in the memory 203 to the detection server 300 (S604), and the extended evaluation target data update process ends.

8 is a flowchart showing an example of the hostile data determination process in steps S604 and S605. The first detection unit 311 receives the evaluation values {AEEval ₁ , ..., AEEval _n } (S604). The first detection unit 311 calculates the frequency Num of the most frequent value in the evaluation value group consisting of AEEval ₁ , ..., AEEval _n (S801).

The first detection unit 311 determines whether the value obtained by dividing Num by the number of data n is greater than a predetermined threshold value q (S802). The threshold value q is, for example, stored in advance in the memory 303. If the first detection unit 311 determines that the value obtained by dividing Num by the number of data n is greater than the predetermined threshold value q (S802: Yes), it determines that the evaluation target data is hostile data (S803) and ends the hostile data determination process.

If the first detection unit 311 determines that the value obtained by dividing Num by the number of data n is equal to or less than a predetermined threshold value q (S802: No), it determines that the data to be evaluated is not hostile data (S804) and ends the hostile data determination process.

8, the first detection unit 311 uses the frequency Num of the most frequent value in the evaluation value group consisting of AEEval ₁ , ..., AEEval _n to detect hostile data, but Num is merely one example of an index for detecting hostile data, and an index other than the above-mentioned Num may be used as long as it is an _index indicating the bias of AEEval ₁ , ..., AEEval _{n .} For example, the first detection unit 311 may use as Num the sum of the frequencies of the evaluation values with the highest frequencies of the nth highest in the evaluation value group consisting of AEEval _{1 , ..., AEEval n, or may use the variance regarding the number of data output for each of AEEval 1} , ..., AEEval _n to detect hostile data.

An example of the above-mentioned variance will be described. The first detection unit 311 calculates the frequency of the evaluation values included in the evaluation value group consisting of AEEval ₁ , ..., AEEval _n , and calculates the variance of the calculated frequencies. For example, it is assumed that the number of extended evaluation target data is 10, and the value that can be taken as the evaluation value of the extended evaluation target data is either 0 or 1. Furthermore, it is assumed that the values of AEEval ₁ to AEEval ₁₀ are all 1. In this case, the frequency of evaluation value 0 is 0, and the frequency of evaluation value 1 is 10. At this time, the first detection unit 311 calculates, for example, (average) = (number of extended evaluation target data) / (number of values that can be taken as evaluation values) (i.e., calculates 10/2 = 5), calculates (average - frequency) ^2- for each value that can be taken as an evaluation value (i.e., calculates (5-0) ² = 25 and (5-10) ² = 25), and calculates the sum of the calculated ((average) - frequency) ² divided by the number of extended evaluation target data (i.e., (25 + 25) / 10 = 5) as the variance. If the first detection unit 311 determines that the calculated variance is equal to or less than a predetermined threshold r, it determines that the evaluation target data is not hostile data, and if the calculated variance is greater than the predetermined threshold r, it determines that the evaluation target data is hostile data.

Figure 9 is an explanatory diagram showing an example of an overview of the adversarial data detection process. In the example of Figure 9, an example is described in which the data to be evaluated is an image. When a human looks at the image, he or she recognizes it as an image of a cat, but when the evaluation unit 211 inputs the image into the model, the image is mistakenly recognized as an image of a horse. In other words, the data to be evaluated in Figure 9 is adversarial data.

In this case, the image data to be evaluated looks like an image of a cat to humans, so if noise is added to the image data to be evaluated, the classification result when the image with the added noise is input into the model is likely to be an image of a cat.

The data expansion unit 111 generates multiple images (extended evaluation target data) by adding different noises to the images (S602). The noise addition unit 112 adds noise to each of the multiple images until the evaluation value by the evaluation unit 211 (i.e., the classification result output when the image is input to the model) changes (S603).

The first detection unit 311 calculates an index (Num) that indicates the bias of the evaluation value after the change, and if it determines that the bias is large, it determines that the data to be evaluated is hostile data (because it is believed that when various noises are added to the image of a horse, which is hostile data, it will change into the original image, which is an image of a cat).

In the example of Figure 9, if the threshold value q is 0.5, three of the four evaluation values after the change are images of a cat, and therefore Num/n = 3/4 > 0.5, so the first detection unit 311 determines that the evaluation target data is hostile data.

As described above, the adversarial data detection system of this embodiment can detect adversarial data with high accuracy and small computational complexity through the above-mentioned processing. When adversarial data is input to the model, it outputs an evaluation value different from that of normal data, but it is highly likely that the adversarial data is data created by adding artificially small noise to normal data. Therefore, it can be expected that if noise is continued to be added to the adversarial data, it will tend to return to data that outputs the evaluation value of the original normal data. In other words, if the evaluation value of the expanded evaluation target data after the change is the same value at a high rate, it is considered that the expanded evaluation target data having the same evaluation value has returned to normal data, and the adversarial data detection system of this embodiment uses this property to determine whether the evaluation target data is adversarial data.

Specifically, as described above, the adversarial data detection system of this embodiment continues to add noise to the extended evaluation target data until the evaluation value of the extended evaluation target data changes, and determines whether the evaluation target data is adversarial data based on the bias of the evaluation value after the change. Therefore, even if an attacker has information about the detection method of this embodiment, or has information about a model and generates adversarial data in which the output result of the model does not change even if a small amount of noise is added a small number of times, it is possible to detect with high accuracy whether the evaluation target data is adversarial data, and thus security can be maintained.

In addition, in step S603, the noise adding unit 112 may add noise generated from the same algorithm to each extended evaluation target data, such as generating noise according to a Gaussian distribution each time, or may add noise generated from different algorithms to different extended evaluation target data, such as adding noise according to a Gaussian distribution to _D1 and adding noise according to a uniform distribution to _D2 . In addition, in step S603, the calculation of the evaluation value of the evaluation target data transmitted from the detection server 300 to the data receiving terminal 500 may be performed in advance in step S602, or may be performed immediately before step S613 (it may be performed at any timing after step S601 and before step S613).

In this embodiment, another example of the adversarial data detection process will be described. In the adversarial data detection process of this embodiment, the adversarial data detection system determines whether the evaluation target data is adversarial data based on the evaluation value for the extended evaluation target data that is first generated by adding noise to the evaluation target data (first determination), and if it determines that the evaluation target data is adversarial data, it further executes the determination process (second determination) of steps S603 to S607 described in the first embodiment.

Figure 10 is a sequence diagram showing an example of the adversarial data detection process. The differences from Figure 6 will be mainly explained. Following step S602, the data extension unit 111 transmits the generated extended evaluation target data and the evaluation target data to the evaluation server 200 (S1001).

The evaluation unit 211 calculates an evaluation value for each of the evaluation target data and the extended evaluation target data received in step S1001 (S1002). The evaluation unit 211 transmits the evaluation value calculated in step S1002 to the detection server 300 (S1003). The second detection unit 312 uses the evaluation value received in step S1003 to determine whether the evaluation target data is likely to be hostile data or not (S1004). Details of the processing in step S1004 will be described later with reference to FIG. 11.

If the second detection unit 312 determines that the evaluation target data is hostile data (S1004: Yes), it does not transmit the evaluation value of the evaluation target data to the data transmission terminal 400, and transmits a notification (determination result: reject) to the data transmission terminal 400 indicating that the transmission of the evaluation value is rejected (S1005), and the hostile data detection process ends. If the second detection unit 312 determines that the data is not likely to be hostile data (S1004: No), it transmits a notification (determination result: accept) to the noise addition server 100 indicating that the data will proceed to a second determination (and/or that the evaluation target data is not likely to be hostile data) (S1006), and the processes of steps S603 to S607 (second determination) are performed, and the hostile data detection process ends.

In addition, since the evaluation value of _Di calculated in the first step S702 for each _Di (i = 1, ..., n) is the same as the evaluation value of each extended evaluation target data calculated in step S1002, the processing of the first step S702 for each _Di (i = 1, ..., n) may be omitted.

11 is a flowchart showing an example of the adversarial data determination process (second determination) in step S1004. The second detection unit 312 receives an evaluation value Eval for the evaluation target data and an evaluation value {Eval ₁ , ..., Eval _n } for the extended evaluation target data (S1003). The second detection unit 312 obtains the number Num of extended evaluation target data for which Eval ≠ Eval _i (i = 1, ... n) (S1101). The second detection unit 312 determines that the value obtained by dividing Num by the number n of extended evaluation target data is smaller than a predetermined threshold p (S1102). The threshold p is, for example, stored in advance in the memory 303.

When the second detection unit 312 determines that the value obtained by dividing Num by the number n of data to be subjected to extended evaluation is smaller than a predetermined threshold p (S1102: Yes), it determines that the data to be subjected to evaluation is hostile data (S1103) and ends the hostile data determination process. When the second detection unit 312 determines that the value obtained by dividing Num by the number n of data to be subjected to extended evaluation is equal to or greater than the predetermined threshold p (S1102: No), it determines that the data to be subjected to evaluation may not be hostile data (S1104) and ends the hostile data determination process.

Note that in step S1102, the second detection unit 312 uses the proportion of the number of extended evaluation target data with different evaluation values for detection, but other indices may be used for detection. For example, in step S1102, the second detection unit 312 may count the amount of change in the evaluation value of each extended evaluation target data from the evaluation value of the evaluation target data, and use the average of the amount of change instead of Num.

Since hostile data is likely to be data created by adding artificial minute noise to normal data, it is likely to be data close to the boundary where the evaluation value output by the model changes. In other words, when the evaluation target data is hostile data, the evaluation value of the extended evaluation target data in which noise has been added to the evaluation target data is likely to be different from the evaluation value of the evaluation target data. Therefore, the adversarial data detection system of this embodiment can efficiently detect adversarial data by determining whether the evaluation target data is adversarial data based on the proportion of extended evaluation target data whose evaluation value differs from that of the evaluation target data in the process of FIG. 11.

In the above example, if the first judgment determines that the data to be evaluated is adversarial data, the adversarial data detection system makes a final judgment as to whether the data to be evaluated is adversarial data by the second judgment. However, if the second judgment determines that the data to be evaluated is adversarial data, the adversarial data detection system may make a final judgment as to whether the data to be evaluated is adversarial data by the first judgment.

As described above, the adversarial data detection system of this embodiment can detect adversarial data more safely and with higher accuracy by using two detection methods to capture different properties of the data to be evaluated.

This embodiment describes an example of a use case of the adversarial data detection system. FIG. 12 is an explanatory diagram showing an example of input/output data of the adversarial data detection system. In the example of FIG. 12, the data to be evaluated is a medical image, and the adversarial data detection system determines, based on the medical image, whether the human subject of the medical image is sick, and if so, the type of illness the human subject is suffering from.

In step S601, the data transmission terminal 400 transmits a medical image to the noise addition server 100. In step S605, if the detection server 300 determines that the medical image is hostile data, in step S606, it transmits a report to that effect to the data transmission terminal 400.

If the detection server 300 determines in step S605 that the medical image is not hostile data, in step S607, it transmits the evaluation result for the medical image (whether the subject is sick or not, and if so, the classification result indicating the type of illness) to the data receiving terminal 500.

The present invention is not limited to the above-described embodiments, but includes various modified examples. For example, the above-described embodiments have been described in detail to clearly explain the present invention, and are not necessarily limited to those having all of the configurations described. It is also possible to replace part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment. It is also possible to add, delete, or replace part of the configuration of each embodiment with other configurations.

The above configurations, functions, processing units, processing means, etc. may be realized in part or in whole in hardware, for example by designing them as integrated circuits. The above configurations, functions, etc. may be realized in software by a processor interpreting and executing a program that realizes each function. Information such as the programs, tables, files, etc. that realize each function can be stored in a memory, a recording device such as a hard disk or SSD (Solid State Drive), or a recording medium such as an IC card, SD card, or DVD.

In addition, the control lines and information lines shown are those considered necessary for the explanation, and not all control lines and information lines on the product are necessarily shown. In reality, it can be assumed that almost all components are interconnected.

100 Noise addition server, 200 Evaluation server, 300 Detection server, 400 Data transmission terminal, 500 Data reception terminal, 101 Processor, 102 Auxiliary storage device, 103 Memory, 107 Communication interface, 111 Data expansion unit, 112 Noise addition unit, 201 Processor, 202 Auxiliary storage device, 203 Memory, 207 Communication interface, 211 Evaluation unit, 301 Processor, 302 Auxiliary storage device, 303 Memory, 307 Communication interface, 311 First detection unit, 312 Second detection unit, 401 Processor, 402 Auxiliary storage device, 403 Memory, 407 Communication interface

Claims (12)

1. An adversarial data detection device, comprising:
A processor and a memory,
The memory holds evaluation target data and a model that outputs an evaluation value when the data is input,
The processor,
inputting the evaluation target data into the model to calculate an evaluation value of the evaluation target data;
Adding different noises to the evaluation target data to generate a plurality of extended evaluation target data;
Executing a first determination process based on the plurality of extended evaluation target data;
In the first determination process,
updating the extended evaluation target data by adding noise to each of the plurality of extended evaluation target data until the evaluation value of the extended evaluation target data does not match the evaluation value of the evaluation target data;
An adversarial data detection device that determines whether the evaluation target data is adversarial data based on the bias of the evaluation values of each of the updated extended evaluation target data. 2. The adversarial data detection apparatus of claim 1,
The processor determines that the evaluation target data is hostile data when the frequency of the most frequent evaluation value of each of the updated extended evaluation target data among the number of the multiple extended evaluation target data is greater than a predetermined threshold. 2. The adversarial data detection apparatus of claim 1,
The processor,
inputting each of the plurality of extended evaluation target data into the model, and calculating an evaluation value for each of the plurality of extended evaluation target data;
execute a second determination process for determining whether the evaluation target data is likely to be non-hostile data based on a rate of agreement between the evaluation value of each of the plurality of extended evaluation target data and the evaluation value of the evaluation target data;
The adversarial data detection device executes the first determination process when it determines that the evaluation target data is likely not adversarial data. 4. The adversarial data detection apparatus of claim 3,
The processor determines that the evaluation target data is likely not hostile data when the rate at which the evaluation values of each of the multiple extended evaluation target data match the evaluation value of the evaluation target data is less than a predetermined threshold. 2. The adversarial data detection apparatus of claim 1,
An adversarial data detection device, wherein the noise added in generating the plurality of extended evaluation target data and the noise added in updating the plurality of extended evaluation target data include at least one of noise following a predetermined probability distribution and noise generated to manipulate the evaluation value. 2. The adversarial data detection apparatus of claim 1,
The model is a trained model generated by deep learning, or a trained model obtained by training data that has been subjected to processing including at least one of resizing and padding when deep learning is performed. Adversarial data detection device. An adversarial data detection method by an adversarial data detection device, comprising:
The adversarial data detection device includes a processor and a memory;
The memory holds evaluation target data and a model that outputs an evaluation value when the data is input,
The adversarial data detection method includes:
The processor inputs the evaluation target data into the model and calculates an evaluation value of the evaluation target data;
The processor adds different noises to the evaluation target data to generate a plurality of extended evaluation target data;
The processor executes a first determination process based on the plurality of extended evaluation target data;
In the first determination process,
The processor adds noise to each of the plurality of extended evaluation target data to update the extended evaluation target data until the evaluation value of the extended evaluation target data does not match the evaluation value of the evaluation target data;
The method for detecting hostile data, wherein the processor determines whether the evaluation target data is hostile data based on a bias in the evaluation value of each piece of updated extended evaluation target data. 8. The method of claim 7, further comprising:
The method for detecting adversarial data, wherein the processor determines that the evaluation target data is adversarial data if the frequency of the most frequent evaluation value of each of the updated extended evaluation target data among the number of the multiple extended evaluation target data is greater than a predetermined threshold. 8. The method of claim 7, further comprising:
The processor inputs each of the plurality of extended evaluation target data into the model and calculates an evaluation value for each of the plurality of extended evaluation target data;
The processor executes a second determination process to determine whether the evaluation target data is likely to be non-adversarial data based on a rate at which the evaluation value of each of the plurality of extended evaluation target data matches the evaluation value of the evaluation target data;
The method for detecting adversarial data, wherein the processor executes the first determination process when the processor determines that the evaluation target data is likely not adversarial data. 10. The method of claim 9, further comprising:
The method for detecting hostile data, wherein the processor determines that the evaluation target data is likely not hostile data when the rate at which the evaluation values of each of the plurality of extended evaluation target data match the evaluation value of the evaluation target data is less than a predetermined threshold. 8. The method of claim 7, further comprising:
An adversarial data detection method, wherein the noise added in generating the multiple extended evaluation target data and the noise added in updating the multiple extended evaluation target data include at least one of noise following a predetermined probability distribution and noise generated to manipulate the evaluation value. 8. The method of claim 7, further comprising:
The adversarial data detection method, wherein the model is a trained model generated by deep learning, or a trained model obtained by training data that has been subjected to processing including at least one of resizing and padding when deep learning is performed.

Images