JP7643373B2 - Fault data generation device, fault data generation method, and program - Google Patents

Description

This disclosure relates to a fault data generation device, a fault data generation method, and a program.

In recent years, attention has been focused on troubleshooting based on machine learning and AI (Artificial Intelligence). In ICT (Information and Communication Technology) systems, which continue to grow in size and complexity, it is practically difficult to handle all troubleshooting manually. An approach that pre-learns system behavior and failure patterns based on huge amounts of data, and then uses the constructed machine learning model to quickly and accurately detect anomalies and identify their causes, is becoming essential from the perspective of reducing human costs and providing stable services.

However, depending on the type of fault that occurs, machine learning models may not function as expected, resulting in incorrect inferences (predictions) or oversights. For example, for faults that occur infrequently or for different faults that output similar data, the performance of the machine learning model drops because learning is not performed sufficiently with limited data. Machine learning models are generally versatile, but the performance of the machine learning model may drop significantly, especially in cases where the system exhibits qualitatively different behavior depending on the environmental state (traffic volume, etc.) or the type of fault (including location and scale). Therefore, in order to deal with such "critical faults," it is necessary to clarify what types of faults the machine learning model is vulnerable to, and then to prepare a sufficient amount of training data to deal with those faults.

In response to this, there is a method called "chaos engineering" that uncovers critical failures that have not yet occurred (Non-Patent Document 1). This is an approach that observes the behavior of a system through experiments that artificially insert failures. It has been mainly adopted in microservice architectures, and a large amount of OSS (Open Source Software) and commercial tools have been developed. In recent years, chaos engineering has also been mentioned as a data generation method for machine learning models. It has also been proposed to respond to the above-mentioned failures by, for example, injecting (injecting) all kinds of failures to collect observation data, and pre-training a classifier to identify factors that associate the observation data with the type of failure (Patent Document 1).

C. Rosenthal, L. Hochstein, A. Blohowiak, N. Jones, and A. Basiri, Chaos Engineering: Building Confidence in System Behavior Through Experiments. O'Reilly Media, 2017.[3] Frazier, Peter I. "A tutorial on Bayesian optimization." arXiv preprint arXiv:1807.02811 (2018).

JP 2021-128538 A

However, because there is a huge variety of environmental conditions that can actually be realized and types of faults that can occur, the above problems cannot be solved by simply applying the conventional chaos engineering scheme directly. That is, current chaos engineering schemes implement "random fault occurrence" or "specific fault occurrence specified by the operator." However, when there are a huge number of possible environmental conditions and fault types, random search is inefficient, and issues arise such as difficulty in obtaining data on critical faults and redundant data collection for faults for which sufficient learning data already exists. Furthermore, it is practically difficult for an operator to specify all critical faults in advance. The method of Patent Document 1 assumes comprehensive fault insertion, but this is similarly difficult.

The present invention has been made in consideration of the above points, and aims to discover critical faults and generate learning data for them at the lowest possible cost, i.e., with a small number of experiments or short experiment time, by optimizing fault insertion experiments.

In order to solve the above problem, the invention of claim 1 is a fault data generation device comprising: an insertion fault calculation unit that calculates and determines an optimal fault to be inserted into a system to be verified by using a mathematical algorithm that preferentially searches for faults for which an evaluation function representing the value of generating data by inserting fault data has a value greater than a predetermined value, based on the evaluation function; a system control unit that executes insertion of the determined fault data into the system to be verified based on an instruction from the insertion fault calculation unit; and an inference model unit that infers information about the fault by a machine learning model using observation data obtained from the system to be verified by inserting the fault data as an input , wherein the mathematical algorithm is a Bayesian optimization algorithm or a bandit algorithm that maximizes the evaluation function .

As described above, the present invention has the effect of optimizing fault insertion experiments to discover critical faults and generate learning data for them at the lowest possible cost, i.e., with a small number of experiments or a short experimental time.

1 is a diagram illustrating a relationship between a fault data generation device and a system under verification according to an embodiment, and a functional configuration of the fault data generation device. 1 is a hardware configuration diagram of a fault data generating device according to an embodiment; FIG. 1 illustrates a mathematical algorithm in an embodiment.

The following describes an embodiment of the present invention with reference to the drawings.

[Outline of the present embodiment]
First, an overview of this embodiment will be described.

In this embodiment, an index is introduced to measure the vulnerability of a machine learning model for faults, such as anomaly detection or factor identification. In general, a machine learning model for anomaly detection or factor identification outputs an inference accuracy calculated using a probability distribution or the like in addition to an inference label. If the inference accuracy for the correct label is low, it can be said that the machine learning model is vulnerable to a given fault. Therefore, a vulnerability index based on the inference accuracy can be defined for a machine learning model for anomaly detection or factor identification. Next, in this embodiment, under the assumption that similar faults should have similar vulnerability indices, an efficient search for highly vulnerable faults, i.e., critical faults, is performed using Bayesian optimization. Since a fault insertion experiment requires a certain amount of time to obtain data and the vulnerability function has an unknown shape, the method is suitable as a use case for Bayesian optimization, which excels as an optimization method for black-box functions in which sample acquisition is generally costly. In this way, data on critical faults can be efficiently collected with a small number of experiments, and the machine learning model can be sequentially updated to build a machine learning model for anomaly detection or factor identification that can handle any environmental state or fault type.

[Functional configuration of the fault data generation device]
Next, the functional configuration of the fault data generator 1 will be described with reference to Fig. 1. Fig. 1 is a diagram showing the relationship between the fault data generator and a system under verification according to an embodiment, and the functional configuration of the fault data generator.

In FIG. 1, the fault data generation device 1 is constructed by a computer, and is a device that can track configuration changes of the system 2 to be verified by sequentially updating the machine learning model m by repeating the processes of fault insertion, data collection, and re-learning using the updated machine learning model m, rather than completing the update of the machine learning model m in one go. The system to be verified is a physical system or a virtual system.

As shown in FIG. 1, the fault data generation device 1 has an insertion fault calculation unit 11, a system control unit 12, and an inference model unit 13 including a machine learning model m. Each of these units is a function that is realized by instructions from a processor 101 in FIG. 2, which will be described later, based on a program. The fault data generation device 1 also has a learning database 10, which is constructed in a memory 102 or an auxiliary storage device 103 in FIG. 2, which will be described later.

Of these, the insertion fault calculation unit 11, as a Bayesian optimization engine, calculates and determines the optimal fault to be inserted. Specifically, the insertion fault calculation unit 11 determines the fault to be inserted by a mathematical algorithm that preferentially searches for faults whose evaluation function has a value greater than a predetermined value, based on an evaluation function that represents the value of performing data generation by fault insertion.

Here, the evaluation function used is a vulnerability index to a fault defined based on the inference accuracy of the machine learning model m of the inference model unit 13. In addition, the mathematical algorithm used is a Bayesian optimization algorithm or a bandit algorithm that maximizes the evaluation function.

The system control unit 12 also has a fault insertion engine 12a and a restoration engine 12b. The fault insertion engine 12a obtains a fault (c, x) insertion command from the insertion fault calculation unit 11, and inserts fault (c, x) data into the system 2 under verification. The restoration engine 12b obtains a preservation command from the insertion fault calculation unit 11, and outputs restoration data to the system 2 under verification.

The inference model unit 13 uses the observation data {z ⁱ (c, x)} _i obtained from the system 2 under verification by the fault injection from the fault injection engine 12 a as input data, infers fault-related information using the machine learning model m, and outputs the inference result h(c, x) as output data to the insertion fault calculation unit 11. As a result, the insertion fault calculation unit 11 calculates the vulnerability index V _D (c, x) and the acquisition function, and determines the next fault (c, x) to be inserted.

The learning database 10 stores (accumulates) the observation data {zi(c,x)}i obtained from the system 2 under verification by the restoration of the system 2 under verification by the restoration engine 12b.

After the processing by the insertion fault calculation unit 11, the system control unit 12, and the inference model unit 13 is repeated a certain number of times, the inference model unit 13 receives data for updating the model from the learning database 10, and re-learns and updates the machine learning model m. The updated machine learning model m is appropriately imported into the inference model unit 13 that runs in the actual operation system.

[Hardware Configuration of the Fault Data Generation Device]
Next, a hardware configuration diagram of the fault data generation device 1 will be described with reference to Fig. 2. Fig. 2 is a hardware configuration diagram of the fault data generation device according to the embodiment. Note that the hardware configuration when the verification target system 2 is a physical system is similar to that of the fault data generation device 1, and therefore a description thereof will be omitted.

As shown in FIG. 2, the fault data generation device 1 includes a processor 101, a memory 102, an auxiliary storage device 103, a connection device 104, a communication device 105, and a drive device 106. The hardware components constituting the fault data generation device 1 are connected to each other via a bus 107.

The processor 101 serves as a control unit that controls the entire fault data generation device 1, and includes various computing devices such as a CPU (Central Processing Unit). The processor 101 reads various programs onto the memory 102 and executes them. The processor 101 may also include a GPGPU (General-purpose computing on graphics processing units).

The memory 102 has a primary storage device such as a ROM (Read Only Memory) or a RAM (Random Access Memory). The processor 101 and the memory 102 form what is known as a computer, and the processor 101 executes various programs read onto the memory 102, causing the computer to realize various functions.

The auxiliary storage device 103 stores various programs and various information used when the programs are executed by the processor 101.

The connection device 104 is a connection device that connects an external device (e.g., the display device 110, the operation device 111) to the fault data generation device 1.

The communication device 105 is a communication device for sending and receiving various information between other devices.

The drive device 106 is a device for setting a (non-transient) recording medium 130. The recording medium 130 here includes media that record information optically, electrically, or magnetically, such as a CD-ROM (Compact Disc Read-Only Memory), a flexible disk, or a magneto-optical disk. The recording medium 130 may also include semiconductor memory that records information electrically, such as a ROM (Read Only Memory) or a flash memory.

The various programs to be installed in the auxiliary storage device 103 are installed, for example, by setting the distributed recording medium 130 in the drive device 106 and reading the various programs recorded on the recording medium 130 by the drive device 106. Alternatively, the various programs to be installed in the auxiliary storage device 103 may be installed by downloading them from a network via the communication device 105.

[Processing or Operation of the Embodiment]
Next, the process or operation of this embodiment will be described in detail. Before describing the process or operation, the problem setting will be described.

<Problem Setting>
The problem considered in this embodiment will be formulated below. Here, formulation will be performed for a supervised class classification machine learning model used to identify the cause of a failure and an unsupervised binary classification machine learning model used to detect anomalies. However, this is merely an example and can be applied even in situations where the model does not strictly follow the rules.

First, we describe the formulation for the machine learning model of supervised classification (herein called the “cause specifier”). The fault space F = (C, X) ∋ (c, x) = (c, x ₁ , …, x _n ) is the set of all faults that are expected to occur in the system 2 under verification.

c∈C is the category of failures to be classified by the cause identifier, and represents the type of failure, the location of the failure, etc. Compound failures that combine multiple failures can also be handled by enumerating the combinations.

x∈X is another discrete or continuous multidimensional parameter that defines the state of the verification target system 2, such as the traffic volume at the time of the fault occurrence, or the scale of the fault (delay time in the case of a delay fault, packet loss rate in the case of a packet loss fault, CPU load rate in the case of a CPU fault, etc.). Even if it is known that some fault has occurred during system operation, (c, x) cannot be directly observed. To be precise, there may be a situation in which x can be partially observed, but c cannot be directly observed. Instead, it is possible to observe the observation data z(c,x)=(z ₁ (c,x),...,z _n (c,x)) generated by the target system when a fault (c,x) occurs. Even if the fault is the same, the generated data may have fluctuations, so {z(c,x)|(c,x)∈F} can be regarded as a multidimensional stochastic process parametrized by (c,x) (if (c,x) is fixed, z(c,x) can be regarded as a random variable). Each z _i (c,x) may be any observable numerical quantity, such as various system metrics, the number of occurrences of a particular log, or an evaluation value of service quality.

During operation, a factor specifier is used to infer c, which cannot be directly observed, using the observed data z as input. In other words, the factor specifier D is

という関数とみなせる。ここで|C|は障害のカテゴリー数であり、h(z)は規格化された非負成分からなる|C|次元ベクトルであり、各成分はそのカテゴリーである確率を表す。

Here, |C| is the number of disability categories, and h(z) is a |C|-dimensional vector of normalized non-negative components, each of which represents the probability of being in that category.

For example, in the case of a neural network model, this corresponds to the output value of the softmax function in the output layer. In general, the index argmax _c h _c (z) that gives the maximum value is used as the inference result.

We introduce an index that represents the vulnerability of such a factor specifier D, that is, the possibility that inference will not be performed properly. The vulnerability VD(c,x) of factor specifier D against a fault (c,x) is defined as follows:

で定義する。ここで期待値操作

Here, we define the expected value operation

は確率変数z(c,x)に対するものである。この定義によると正解カテゴリーに対する推論確率が低い障害に対して脆弱性が高いことになる。つまり、要因特定器DはV_D(c,x)が大きくなるような障害(c,x)に対しては、適切な推論がなされない可能性が高く、そのような障害に対する観測データz(c,x)を積極的に提供するのが本実施形態の目的である。なお、脆弱性の具体的な定義式は上記のものに限る必要はなく、

is for the random variable z(c,x). According to this definition, vulnerability is high for faults with a low probability of inferring the correct category. In other words, the cause identifier D is unlikely to make appropriate inferences for faults (c,x) for which _VD (c,x) is large, and the purpose of this embodiment is to actively provide observed data z(c,x) for such faults. Note that the specific definition of vulnerability does not need to be limited to the above,

などとしたり、障害に依存して適当な重みを課したりしてもよい。

Or, appropriate weights may be applied depending on the type of disability.

Next, before explaining how to maximize V _D (c,x), we will also explain an example of formulation for an unsupervised binary classification machine learning model (herein called an "anomaly detector"). Usually, in anomaly detection, learning is performed only with the training data corresponding to the normal state of the verification target system 2, a model is constructed, and for test data, the degree of deviation from the normal state is referred to to determine whether it is normal or abnormal. Specifically, it is implemented with a machine learning model such as a one-class support vector machine or an autoencoder. In this situation, the various concepts mentioned above can be defined as follows. The fault space can be shown as F = (C,X) ∋ (c,x) = (c,x ₁ , ..., x _n ), and in particular, c ∈ {0, 1} is a binary variable that represents normal (0) or abnormal (1) (note that the fault space also includes the normal state (c = 0)). The observed data z(c,x) is defined in the same way as for the factor specifier. The anomaly detector D is

と示すことができ、h(z)は検証対象システム２が異常である確率を表す。一般には、適当な閾値α∈(0、1)を用いて、

where h(z) represents the probability that the verification target system 2 is abnormal. In general, using an appropriate threshold α∈(0, 1),

であれば異常、そうでなければ正常と推論する。状態(c,x)に対する異常検知器Dの脆弱性V_D(c,x)を、

If so, it is inferred to be abnormal, otherwise it is inferred to be normal. The vulnerability V _D (c,x) of an anomaly detector D for state (c,x) is

或いは

or

などと定義できる。ここで

Here,

は引数が真であれば１、偽であれば０を取る指示関数である。

is an indicator function that takes the value 1 if the argument is true and 0 if it is false.

The following explanation uses the case of a factor identifier as an example, but a similar discussion can be applied to an anomaly detector or other machine learning models.

The problem considered in this embodiment is as follows. Now, suppose that a factor identifier D is constructed based on prior data. Since it is impossible to prepare prior data comprehensively, D may be highly vulnerable to a specific fault. Therefore, suppose that an environment is available in which any fault (c, x) can be intentionally inserted and observation data for additional learning can be generated. This can be achieved using a chaos engineering tool or a self-created script. However, in general, the cycle of fault insertion, data acquisition, and restoration to the original state takes a certain amount of time, so comprehensive data generation remains difficult. Then, the problem arises of how to search the fault space in order to efficiently find faults (c, x) that increase the vulnerability index V(c, x), generate additional learning data, and improve the accuracy of the factor identifier.

<Processing or Action>
Next, the processing or operation of this embodiment will be described with reference to Fig. 3. Fig. 3 is a diagram showing an algorithm in this embodiment.

To solve the above problems, the present embodiment takes a Bayesian optimization approach (Reference 1).
<Reference 1> Frazier, Peter I. "A tutorial on Bayesian optimization." arXiv preprint arXiv:1807.02811 (2018).
Bayesian optimization is a general-purpose optimization method used to obtain the maximum value of a black-box function with a high sample acquisition cost. In Bayesian optimization, a black-box objective function is modeled as a stochastic process such as a Gaussian process, and the balance between the magnitude of uncertainty and the magnitude of the expected value (trade-off between exploration and exploitation) is considered, and the utility obtained when a value is acquired at each point is expressed by an acquisition function, so that optimal sampling is sequentially performed. That is, the objective function is maximized while sampling (exploitation) of the surrounding area where the maximum value is expected to be obtained at the current time and sampling (exploration) of the area where the number of samplings is small and the uncertainty is large are performed in a balanced manner. In the case of the problem setting this time, the vulnerability index V _D (c,x) is the objective function to be maximized. The objective of this embodiment is not necessarily maximization, but rather discovery of faults with large V _D (c,x) and generation of data. However, since both exploration and exploitation are attempted in Bayesian optimization as described above, it is possible to widely explore faults with high vulnerability and achieve the objective. If the variables representing the obstacles (c, x) are continuous variables, the usual Bayesian optimization method can be applied as is, but if there are only discrete variables, a multi-armed bandit algorithm is used instead of Bayesian optimization. If there are a mixture of discrete and continuous variables, it is necessary to continuously relax the discrete variables or use a method that combines Bayesian optimization and a multi-armed bandit algorithm, but in either case, it can be implemented using an open source Bayesian optimization tool (Reference 2) or existing algorithms (References 3 and 4).
<Reference 2> Balandat, Maximilian, et al. "BoTorch: A Framework for Efficient Monte-Carlo Bayesian Optimization." NeurIPS. 2020.
<Reference 3> Nguyen, Dang, et al. "Bayesian optimization for categorical and category-specific continuous inputs." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 34. No. 04. 2020.
<Reference 4> Ru, Binxin, et al. "Bayesian optimization over multiple continuous and categorical inputs." International Conference on Machine Learning. PMLR, 2020.
3, basically, Bayesian optimization is applied with the vulnerability index _VD (c,x) of the factor identifier D as the objective function. First, the insertion fault calculation unit 11, the system control unit 12, and the inference model unit 13 perform random sampling _N1 times (an appropriate natural number, which may be dynamically changed for each iteration) (S1).

One sampling is the observation data obtained by inserting a specified fault (c,x) into the target system.

を取得、追加用データセットとして保存の上、脆弱性V_D(c,x)を計算して、検証対象システム２を原状復旧することを指す。z_i(c,x)は確率変数z(c,x)の実現値を表しており、ここではT個のデータが取れたことになる。例えば、データのスクレイピング間隔を１０秒としたとき、障害挿入を10×T秒間継続していれば、T個のデータが取れる。V_D(c,x)の計算においては、例えば、期待値操作を

This refers to obtaining and saving the data set for addition, calculating the vulnerability V _D (c,x), and restoring the target system 2 to its original state. _{z i} (c,x) represents the realized value of the random variable z(c,x), and in this case, T pieces of data have been obtained. For example, if the data scraping interval is 10 seconds, and the fault insertion continues for 10×T seconds, T pieces of data can be obtained. In calculating _{V D} (c,x), for example, the expected value operation is

によるサンプル平均で置き換え、

Replace with the sample mean by

のように求めればよい。ただし、この計算式は例であり、別の方法により算出してもよい。

However, this calculation formula is an example, and calculation may be performed using another method.

Here, the process (S1) will be described in detail with reference to FIG.
S11: The insertion fault calculation unit 11 randomly selects a fault (c, x).
S12: The fault insertion engine 12a inserts fault (c, x) data into the system 2 under verification.
S13: The training database 10 acquires and stores the observed data {z ⁱ (c,x)} _i from the verification target system 2.
S14: The inference model unit 13 acquires the observation data {z ⁱ (c, x)} _i from the system 2 under verification, infers information about faults using the machine learning model m, and outputs the inference result h(c, x) as output data to the insertion fault calculation unit 11. Furthermore, the insertion fault calculation unit 11 calculates the vulnerability index V _D (c, x) and the acquisition function, and determines the fault (c, x) to be inserted next.
S15: The training database 10 acquires and stores the observed data {z ⁱ (c,x)} _i from the verification target system 2.
S16: The restoration engine 12b performs restoration on the verification target system 2.

Next, the insertion fault calculation unit 11, the system control unit 12, and the inference model unit 13 repeat the cycle of calculating the acquisition function and sequentially determining the fault (c, x) to be sampled next _N2 times (an appropriate natural number, which may be dynamically changed for each iteration) according to the Bayesian optimization framework (S2). Various variations of the method of designing the acquisition function have been considered, but any of the methods can be applied to this embodiment.

Here, the process (S2) will be described in detail with reference to FIG.
S21: The insertion fault calculation unit 11 randomly selects a fault (c, x).
S22: The fault insertion engine 12a inserts fault (c, x) data into the system 2 under verification.
S23: The training database 10 acquires and stores the observed data {z ⁱ (c,x)} _i from the verification target system 2.
S24: The inference model unit 13 acquires the observation data {z ⁱ (c, x)} _i from the system 2 under verification, infers information about faults using the machine learning model m, and outputs the inference result h(c, x) as output data to the insertion fault calculation unit 11. Furthermore, the insertion fault calculation unit 11 calculates the vulnerability index V _D (c, x) and the acquisition function, and determines the fault (c, x) to be inserted next.
S25: The training database 10 acquires and stores the observed data {z ⁱ (c,x)} _i from the verification target system 2.
S26: The restoration engine 12b performs restoration on the verification target system 2.
S27: The insertion fault calculation unit 11 calculates the vulnerability index V _D (c, x) and the acquisition function.
S28: The insertion fault calculation unit 11 determines the next fault (c, x) to be inserted based on the calculation in S28.

Finally, the inference model unit 13 retrains the factor identifier using all the observation data obtained during the (N ₁ +N ₂ ) samplings stored in A as the additional data set as training data (S13). When retraining, online learning or sequential learning methods may be used, or batch training may be performed by combining all the pre-training data and the additional training data. The newly obtained factor identifier D' (shown on the left side of S3 in FIG. 3) is expected to be less vulnerable to a specific fault than the previous factor identifier D. Furthermore, the accuracy of the factor identifier can be further improved by repeating the above process using a newly defined vulnerability index V _D' (c,x) for D'. The repetition may be performed under appropriate conditions (SomeConditions). The termination condition may be that no fault is found that makes the vulnerability equal to or greater than a certain value for a sufficient number of trials.

[Application examples]
The following are possible application examples of the embodiment.

The fault insertion and data generation in this embodiment are performed in the verification target system 2 to pre-construct the factor identifier, and then the factor identifier is introduced into the production environment.

The system under verification 2 performs fault insertion and data generation in this embodiment using a factor identifier similar to the one running in the production environment. The factor identifier updated by re-learning is imported into the factor identifier in the production environment as appropriate. If the system under verification 2 is made to follow any changes to the configuration of the production environment, this embodiment will enable data on newly occurring faults to be generated in semi-real time.

A virtual environment such as a container environment that simulates a production environment is used as the above-mentioned verification target system 2. In a virtual environment, existing chaos engineering can be used, making implementation easy, and data acquisition and restoration to the original state can be performed in a short time.

If fault injection experiments are possible in a production environment, this embodiment may be implemented in the production environment. In other words, using a factor identifier running in the production environment, fault injection and data collection according to the present invention are actually repeated to update the factor identifier in semi-real time. Since fault injection in a production environment is common in chaos engineering schemes, this embodiment is also realistic.

[Main Effects of the Embodiments]
As described above, according to this embodiment, the difficulty of inference in a machine learning model is quantified by a vulnerability index, and critical faults that are highly vulnerable and difficult to infer are searched for based on Bayesian optimization. This makes it possible to efficiently discover critical faults with fewer experiments than the random search performed in conventional chaos engineering, and generate data for them. By repeatedly re-learning the machine learning model using these data, it becomes possible to improve the inference accuracy of the machine learning model for all faults.

Reference Signs List 1 Fault data generation device 2 System to be verified 10 Learning database 11 Insertion fault calculation unit 12 System control unit 12a Fault insertion engine 12b Restoration engine 13 Inference model unit m Machine learning model

Claims (6)

an insertion fault calculation unit that calculates and determines an optimal fault to be inserted into the system under verification using a mathematical algorithm that preferentially searches for faults whose evaluation function has a value larger than a predetermined value based on an evaluation function that indicates the value of performing data generation by inserting fault data;
a system control unit that executes insertion of the determined fault data into the system under verification based on a command from the insertion fault calculation unit;
an inference model unit that infers information about the fault using a machine learning model with observation data obtained from the system under verification by inserting the fault data as an input;
having
The fault data generating device , wherein the mathematical algorithm is a Bayesian optimization algorithm or a bandit algorithm that maximizes the evaluation function . The fault data generating device according to claim 1 , wherein the evaluation function uses a vulnerability index to faults that is defined based on an inference accuracy of a machine learning model in the inference model unit. The fault data generation device according to claim 1, wherein the system control unit performs restoration of the verification target system based on an instruction from the insertion fault calculation unit. 4. The fault data generation device according to claim 1, wherein when the inference model unit re-learns a machine learning model using as input observation data obtained from the system to be verified by inserting the fault data, the inference model unit does not complete the update of the machine learning model in one go, but uses the updated machine learning model to repeat the processes of fault insertion, data collection, and re - learning to sequentially update the model. The computer
calculating and determining an optimal fault to be inserted into the system to be verified using a mathematical algorithm that preferentially searches for faults whose evaluation function has a value greater than a predetermined value based on an evaluation function that indicates the value of performing data generation by inserting fault data;
inserting data of the determined fault into the system under test;
Inferring information about the fault using a trained machine learning model using observation data obtained from the system under test by inserting the fault data as an input ;
The fault data generation method , wherein the mathematical algorithm is a Bayesian optimization algorithm or a bandit algorithm that maximizes the evaluation function . A program for causing a computer to execute the method according to claim 5 .

Images