JP7316129B2 - Processing system and processing program - Google Patents

Description

The present invention relates to a processing system and a processing program.

Conventionally, there has been known a technique of analyzing communication logs and evaluating abnormalities (see, for example, Patent Document 1).

Japanese Patent Application Laid-Open No. 2018-160078

However, in the technique of Patent Document 1, since an abnormality is evaluated using one model corresponding to the communication log, it is not possible to perform multifaceted analysis of the communication log, and the evaluation accuracy may decrease. had a nature.

SUMMARY OF THE INVENTION It is an object of the present invention to provide a processing system and a processing program capable of improving the accuracy of evaluation of evaluation target data.

In order to solve the above-described problems and achieve the object, the processing system according to claim 1 relates to evaluation of the evaluation target data performed using a plurality of evaluation models used for evaluation of the evaluation target data. A processing system for performing evaluation-related processing, which is processing to perform evaluation-related processing, comprising: obtaining means for obtaining evaluation-related data; and processing means for performing said evaluation-related processing based on said evaluation-related data obtained by said obtaining means. wherein the processing means performs a first process of associating the plurality of evaluation-related data acquired by the acquisition means based on a predetermined standard, and a plurality of the evaluation-related data associated in the first process. a second process of performing the evaluation-related process based on the evaluation-related data, wherein the plurality of evaluation-related data are a plurality of data of a first attribute used in communication conforming to a predetermined standard, and a plurality of data of a first attribute used in communication conforming to a predetermined standard; and a plurality of data of the second attribute to be used, wherein each of the plurality of data of the first attribute and each of the plurality of data of the second attribute are data communicated in a predetermined communication order, and the processing means associates each of the plurality of data of the first attribute and each of the plurality of data of the second attribute with each other based on the communication order of the evaluation-related data in the first process; and in the second process, the performing the evaluation-related processing based on a plurality of combinations including each of the plurality of data of the first attribute and each of the plurality of data of the second attribute associated in the first processing, and included in each of the plurality of combinations The data is only data of two kinds of attributes including the first attribute and the second attribute.

The processing system according to claim 2 is the processing system according to claim 1 , wherein the acquisition means acquires the evaluation target data as the evaluation related data, and the processing means acquires the data acquired by the acquisition means. A process of evaluating the evaluation target data based on the evaluation target data and the plurality of evaluation models is performed as the evaluation related process.

The processing system according to claim 3 is the processing system according to claim 2 , wherein the processing means evaluates the evaluation target data acquired by the acquisition means using each of the plurality of evaluation models, A process of generating evaluation result information indicating results of each evaluation performed using each of the plurality of evaluation models is performed as the evaluation-related process.

4. The processing system according to claim 4 is the processing system according to claim 2 or 3 , wherein the processing means evaluates a data value included in the evaluation target data, or the evaluation target data is performed as the evaluation-related processing.

The processing program according to claim 5 is a processing program that performs evaluation-related processing that is processing related to evaluation of the evaluation target data performed using a plurality of evaluation models used for evaluation of the evaluation target data. , the computer functions as acquisition means for acquiring evaluation-related data, and processing means for performing the evaluation-related processing based on the evaluation-related data acquired by the acquisition means, and the processing means is configured to perform the evaluation-related processing based on the evaluation-related data acquired by the acquisition means. a first process of associating the plurality of evaluation-related data acquired by the acquisition means based on the above, and performing the evaluation-related process based on the plurality of the evaluation-related data associated in the first process 2 processing, wherein the plurality of evaluation-related data are a plurality of data of a first attribute used in communication conforming to a predetermined standard, and a plurality of data of a second attribute used in communication conforming to a predetermined standard. wherein each of the plurality of data of the first attribute and each of the plurality of data of the second attribute are data communicated in a predetermined communication order, and the processing means performs the evaluation in the first processing each of the plurality of data of the first attribute and each of the plurality of data of the second attribute are associated with each other based on the communication order of the related data, and in the second process, the first attributes associated in the first process; and the plurality of data of the second attribute, the evaluation-related processing is performed based on the plurality of combinations, and the data included in each of the plurality of combinations is the first attribute and the first It is only data of two kinds of attributes including two attributes.

According to the processing system according to claim 1 and the processing program according to claim 5 , by performing evaluation-related processing related to evaluation performed using a plurality of evaluation models, for example, a plurality of evaluation models Since the evaluation target data can be evaluated using the model, it is possible to improve the accuracy of the evaluation of the evaluation target data.
Further, by associating a plurality of evaluation-related data with each other based on a predetermined standard and performing evaluation-related processing based on the associated plurality of evaluation-related data, for example, the communication specification of the evaluation-related data as a predetermined standard In the case of adopting a criterion that is in line with the actual situation, it is possible to appropriately perform the evaluation-related processing in line with the actual situation.

According to the processing system of claim 2 , by performing the process of evaluating the evaluation target data based on the evaluation target data and the plurality of evaluation models, for example, the evaluation accuracy of the evaluation target data can be improved. becomes possible.

According to the processing system of claim 3 , the evaluation target data is evaluated using each of the plurality of evaluation models, and the evaluation result information indicating the results of each evaluation performed using each of the plurality of evaluation models is generated. By performing the generating process, for example, it is possible to appropriately indicate the results of the evaluation of the evaluation target data performed using each of the plurality of evaluation models.

According to the processing system of claim 4 , by performing the processing of evaluating the data value included in the evaluation target data or the processing of evaluating the time included in the evaluation target data, for example , data value, and time, it is possible to perform various evaluations on evaluation target data.

2 is a block diagram functionally conceptually showing the information processing apparatus according to the first embodiment; FIG. It is the figure which illustrated the analysis object data specific information. It is the figure which illustrated the 1st model specific information. 6 is a flowchart of model generation processing; 4 is a flowchart of analysis processing; 4 is a flowchart of evaluation processing; FIG. 7 is a block diagram functionally conceptually showing an information processing apparatus according to a second embodiment; It is the figure which illustrated the 2nd model specific information. 6 is a flowchart of model generation processing; 9 is a flowchart of time difference data generation processing; 4 is a flowchart of analysis processing; FIG. 11 is a block diagram functionally conceptually showing an information processing apparatus according to a third embodiment; It is the figure which illustrated the analysis object data specific information. It is the figure which illustrated the data specific information for learning. It is the figure which illustrated the 3rd model specific information. 6 is a flowchart of model generation processing; 4 is a flowchart of analysis processing;

Embodiments of a processing system according to the present invention will be described in detail below with reference to the accompanying drawings. First, [I] the basic concept of the embodiment will be explained, then [II] the specific contents of the embodiment will be explained, and finally [III] modifications of the embodiment will be explained. However, the present invention is not limited by the embodiment.

[I] Basic concept of the embodiment First, the basic concept of the embodiment will be described. Embodiments relate to a processing system and a processing program. The processing system according to the present invention is a system that performs evaluation-related processing. It is a concept that includes a system realized by implementing a function for performing evaluation-related processing on a system, etc.). A "processing system" comprises, for example, acquisition means and processing means.

Further, "evaluation-related processing" is processing related to evaluation of evaluation target data performed using a plurality of evaluation models used for evaluation of evaluation target data. It is a concept that includes processing, processing for evaluating evaluation target data, and the like.

In addition, "acquisition means" means means for acquiring evaluation-related data, and is a concept including, for example, means for acquiring teacher data for generating a plurality of evaluation models as evaluation-related data, and This is a concept including means for acquiring evaluation target data as evaluation-related data.

"Evaluation-related data" is data that serves as a basis for performing evaluation-related processing, and is a concept that includes, for example, evaluation target data, teacher data, and the like.

In addition, "evaluation target data" is data to be evaluated using a plurality of evaluation models. For example, it is a concept that includes data whose attributes are identifiable, or data whose attributes are unidentifiable. , CAN (Controller Area Network), Ethernet (registered trademark), USB (Universal Serial Bus), and other communication standards. This concept includes communication log data, etc.

In addition, "teaching data" is data for generating multiple evaluation models. For example, it is data used in machine learning to generate multiple models. is the same as the evaluation target data described above, that is, it is a concept that includes communication log data and the like conforming to communication standards such as CAN. It should be noted that the data used as this "teaching data" is, for example, data that is determined in advance as to whether it is normal or abnormal based on the above-mentioned arbitrary communication standards or other predetermined rules. can be used.

In addition, "evaluation model" is a model used for evaluation of evaluation target data, for example, a model generated by machine learning, or any other method other than machine learning (e.g., simulation, or It is a concept that includes models, etc. generated by methods of conducting experiments, etc.). Note that "a plurality of evaluation models used to evaluate evaluation target data" is a concept that includes, for example, a plurality of evaluation models used in processing when evaluating evaluation target data. As a result, in the processing system, a plurality of evaluation models are used in order to evaluate one piece of evaluation target data.

Further, the "processing means" is means for performing evaluation-related processing based on the evaluation-related data acquired by the acquisition means. It is a concept that includes "Attribute of evaluation-related data" is, for example, a concept used as a criterion for dividing evaluation-related data into groups. It is a concept including identification information (that is, “ID”) for identifying content, transmission node, etc.).

Further, the "processing means" means, for example, a means for performing evaluation-related processing based on the associated plurality of evaluation-related data, by associating the plurality of evaluation-related data acquired by the acquisition means with each other based on a predetermined standard. It is a concept that includes The “predetermined standard” is a predetermined standard, and is a concept including, for example, a standard determined from a communication standard or a communication specification (that is, a standard based on the actual situation of communication).

Further, the "processing means", for example, determines whether or not the evaluation-related data acquired by the acquisition means is to be subjected to the evaluation-related processing, based on a predetermined determination model, and performs the evaluation-related processing. This is a concept including means for performing evaluation-related processing based on evaluation-related data determined to be targeted. The “determination model” is a model used for determining whether or not the evaluation-related data acquired by the acquisition means is to be subjected to evaluation-related processing. For example, a model generated by machine learning, or It is a concept that includes models and the like generated by any method other than machine learning (for example, a method of conducting simulations, experiments, or the like).

In addition, the "processing means" is, for example, based on the teacher data acquired by the acquisition means and a predetermined algorithm for machine learning for generating a plurality of evaluation models, by machine learning is a concept including means for performing the process of generating the , as an evaluation-related process. "Predetermined algorithm" is an algorithm for generating a plurality of evaluation models, specifically an algorithm for generating a model for detecting outliers, or any other algorithm, etc. For example, it is a concept that includes the "OneClass SVM algorithm" in machine learning.

Further, "processing means" is a concept that includes means for generating a plurality of evaluation models by, for example, changing the values of some of the plurality of parameters in the above-described predetermined algorithm. , As an example, by changing the value of one of a plurality of known parameters including the "kernel" which is an adjustable parameter of the above-mentioned "OneClass SVM algorithm", a plurality of evaluation It is a concept that includes means for generating a model.

Further, the "processing means" means, for example, evaluating the evaluation target data acquired by the acquiring means using each of a plurality of evaluation models, and calculating the results of each evaluation performed using each of the plurality of evaluation models. It is a concept that includes means for performing, as an evaluation-related process, the process of generating the evaluation result information shown, and also the process of evaluating the data values contained in the evaluation target data, or the evaluation target data This is a concept including means for performing time-related evaluation processing as evaluation-related processing.

In each of the embodiments shown below, a case where "evaluation target data" and "teacher data" are communication log data of the CAN communication standard will be described. In the second embodiment, the case of evaluating the time included in the evaluation target data will be described. In the third embodiment, A case will be described where evaluation is performed on data values contained in evaluation target data, and evaluation is performed by associating a plurality of evaluation-related data with each other.

[II] Specific Contents of Each Embodiment Next, specific contents of each embodiment will be described.

(Embodiment 1)
First, Embodiment 1 will be described. This embodiment is a form in which data values included in evaluation target data are evaluated.

(composition)
First, an information processing apparatus according to this embodiment will be described. FIG. 1 is a block diagram functionally conceptually showing an information processing apparatus according to the present embodiment. The information processing apparatus 1 is a processing system, such as a personal computer, and includes an input unit 11, an output unit 12, a recording unit 13, and a control unit 14, for example.

(Configuration - input section)
The input unit 11 is input means for inputting information. Although the specific configuration of the input unit 11 is arbitrary, it can be configured using a known mouse, keyboard, or the like, for example.

(Configuration - output section)
The output unit 12 is output means for outputting information. Although the specific configuration of the output unit 12 is arbitrary, it can be configured using a known display or the like, for example.

(Construction-recording unit)
The recording unit 13 is recording means for recording programs and various data necessary for the operation of the information processing apparatus 1, and is configured using, for example, a hard disk (not shown) as an external recording device. However, any other recording medium can be used in place of or in conjunction with the hard disk, including magnetic recording media such as flash memories, magnetic disks, or optical recording media such as DVDs and Blu-ray discs.

The recording unit 13 stores, for example, analysis target data identification information, learning data identification information, and first model identification information.

(Configuration - recording section - analysis target data identification information)
The “analysis target data specifying information” of the recording unit 13 is information specifying the analysis target data. "Analysis target data" is the above-mentioned evaluation-related data, specifically, the above-mentioned evaluation target data. communication log data in the device). FIG. 2 is a diagram exemplifying analysis target data identification information. For example, as shown in FIG. 2, the 'analysis target data identification information' includes the item 'row information', the item 'time information', the item 'data ID', the item 'payload information', and information corresponding to each item. is information associated with each other. The item "line information" is line information that uniquely identifies each piece of information (combination of time information, data ID, and payload information) (in FIG. 2, it is a unique numerical value such as "1"). Information corresponding to the item "time information" is time information specifying the time stamp (that is, the time when communication was performed) attached to each information (data ID and payload information) (in FIG. 2, "t1", etc.). It should be noted that "t1" etc. here is a notation for convenience, and in fact, the elapsed time from the timing is used based on an arbitrary timing, that is, each data is transmitted or received by CAN communication is used to specify the timing at which the The information corresponding to the item “data ID” is information specifying attributes of the evaluation-related data, and is, for example, an ID for identifying CAN communication data contents, a transmission node, etc. (in FIG. 2, the CAN communication data frame , such as "ID11"). It should be noted that "ID11" and the like are described for convenience of explanation. The item "payload information" is a data value, and is payload information that specifies the data body transmitted and received in CAN communication. The payload information is arbitrary, and is information corresponding to the data field in the data frame of CAN communication. However, in FIG. 2, for convenience of explanation, this information is described as "D1" or the like. Here, information in which row information, time information, data ID, and payload information are associated with each other corresponds to "analysis target data".

Any method of storing the analysis target data in such analysis target data specifying information may be used. 3, collects communication data of CAN communication (data including at least data ID and payload information), attaches a time stamp to the collected communication data, and records it in the recording unit of the log collection device do. Then, the time-stamped communication data recorded in the log collection device is collected by any method (for example, a method of communicating with the log collection device, or a recording medium such as a memory stick). method, etc.) to the information processing apparatus 1, and the control unit 14 attaches line information to the input communication data with a time stamp, and the attached line information, the time stamp, and the data of the communication data By recording the ID and the payload information of the communication data, it is stored as the data to be analyzed in FIG. Alternatively, the above-described known log collection device (not shown) collects communication data of CAN communication in an in-vehicle system that performs CAN communication, generates analysis target data of FIG. 2, and records it in its own recording unit. , the data to be analyzed recorded in the log collection device is input to the information processing device 1 by an arbitrary method, and the control unit 14 stores the input data to be analyzed (The same applies to other embodiments).

By storing in this way, for the data to be analyzed in FIG. For 't1' to 't9', communication of 'data ID'='ID11' is performed sequentially, and thereafter, at least for time information 't101' to 't104', 'data ID'='ID13'. , which is shown to be performed sequentially, will be described below.

(Configuration - recording unit - learning data identification information)
Returning to FIG. 1, the “learning data specifying information” of the recording unit 13 is information specifying learning data. The “learning data” is the above-mentioned evaluation-related data, specifically, the above-mentioned teacher data, for example, the same type of data as the above-mentioned analysis target data, that is, the CAN communication standard This is communication log data (that is, communication log data of CAN communication). For example, although not shown, the "learning data specifying information" includes the item "line information", the item "time information", the item "data ID", and the item " Payload information" and information corresponding to each item are associated with each other. The information corresponding to each of these items corresponds to "learning data" and is the same as the information with the same name as the data to be analyzed, so detailed description thereof will be omitted.

Any method can be used to store the learning data in the learning data specifying information. configured to collect CAN communication communication data (at least data including data ID and payload information) in the implemented system), attach a time stamp to the collected communication data, and record it in the recording unit of the log collection device It is assumed that Then, for the recorded communication data with time stamps, for example, it is determined by an arbitrary method whether it is normal or abnormal, and only when it is determined to be normal, the normal The communication data determined to be By recording the attached line information, time stamp, data ID of communication data, and payload information of communication data, it is stored as learning data in the learning data specifying information of FIG. do).

In addition, regarding the determination of the communication data here, for example, the administrator may pay attention to the payload information of the communication data and visually check based on the specification of CAN communication in the in-vehicle system. Alternatively, a judgment function for judging normality or abnormality based on CAN communication specifications may be implemented in any device, and judgment may be made using the judgment function. By storing in this manner, communication data determined to be normal is stored as learning data.

In addition, as this learning data, a sufficient amount of communication data, etc., which is determined to be normal for performing machine learning, including communication data based on CAN communication specifications, is stored. will be described below.

(Configuration-recording unit-first model specific information)
Returning to FIG. 1, the "first model specifying information" of the recording unit 13 is information specifying a plurality of evaluation models. Specifically, payload information (that is, Data value) is information that identifies the model to evaluate. FIG. 3 is a diagram exemplifying the first model identification information. For example, as shown in FIG. 3, the 'first model identification information' is information in which the item 'data ID', the item 'model information', and the information corresponding to each item are associated with each other. Information corresponding to the item "data ID" is the same as the information with the same name in FIG. Information corresponding to the item "model information" is model information specifying the evaluation model. Note that the model information is arbitrary, and is information specifying a model generated in a model generation process to be described later, such as "ML11-1" in FIG. 3 for convenience of explanation. Further, in the present embodiment, the “evaluation model” is input with analysis target data, and in the input analysis target data, the payload information of the analysis target data deviates relatively greatly from the learning data. A case will be described in which the payload information is a model that outputs information indicating analysis target data that corresponds to the outlier. The method of storing such first model identification information is arbitrary, but for example, it is stored by executing a model generation process, which will be described later.

(Configuration - control unit)
Returning to FIG. 1, the control unit 14 is control means for controlling the information processing apparatus 1. Specifically, the control unit 14 includes a CPU, various programs interpreted and executed on the CPU (a basic control program such as an OS, (including application programs that are activated on the OS and implement specific functions), and an internal memory such as a RAM for storing programs and various data. In particular, the program according to the embodiment substantially configures each unit of the control unit 14 by being installed in the information processing apparatus 1 via any recording medium or network.

For example, the control unit 14 functionally and conceptually includes an acquisition unit 141 and a processing unit 142 . The acquisition unit 141 is acquisition means for acquiring evaluation-related data. The processing unit 142 is processing means that performs evaluation-related processing based on the evaluation-related data acquired by the acquisition unit 141 . Processing of each part of the control part 14 will be described later.

(process)
Next, processing executed by the information processing device 1 according to the present embodiment will be described. Specifically, model generation processing and analysis processing will be described.

(Processing - Model generation processing)
First, model generation processing will be described. The "model generation process" is roughly a process of generating an evaluation model using learning data, for example, a process of generating and storing the first model identification information in FIG. FIG. 4 is a flowchart of model generation processing (steps are abbreviated as "S" in the following description of each processing). Although the timing of executing this model generation process is arbitrary, for example, after the learning data specifying information for specifying a sufficient amount of learning data is recorded in the recording unit 13 of FIG. It is assumed that activation is started when the user performs a predetermined operation, and the description will start from the point at which activation is started (the same applies to model generation processing in other embodiments).

At SA1 in FIG. 4, the acquisition unit 141 acquires learning data. Specifically, although it is arbitrary, for example, the learning data specified by the learning data specifying information of the recording unit 13 in FIG. 1 is acquired. Here, for example, CAN communication communication data determined to be normal is acquired. Specifically, data of the same type as the analysis target data in FIG. , time information, data ID, and payload information are associated with each other.

At SA2 in FIG. 4, the processing unit 142 classifies the learning data based on the data ID. Specifically, it is arbitrary, but for example, the learning data acquired in SA1 is acquired, and based on the data ID of the learning data, the same data ID is classified into the same group. Here, for example, it is assumed that information corresponding to "data ID" = "ID11", "ID13" and "ID12" (not shown) in FIG. classify the learning data into a group corresponding to "data ID" = "ID11", a group corresponding to "data ID" = "ID12", a group corresponding to "data ID" = "ID13", etc. do. By classifying in this way, the following processing can be performed for each data ID.

At SA3 in FIG. 4, the processing unit 142 acquires data for one unit of processing. Specifically, although it is arbitrary, for example, the learning data of the group corresponding to one data ID that has not yet been acquired among the groups classified in SA2 is acquired. Here, for example, the learning data (that is, line information, time information, data ID, and payload information) of the group corresponding to "data ID" = "ID11" is acquired.

At SA4 in FIG. 4, the processing unit 142 generates a plurality of evaluation models. Specifically, although it is arbitrary, for example, it is assumed that a model generation algorithm (not shown) is recorded in the recording unit 13, and the model is generated using this model generation algorithm. The "algorithm for model generation" is the above-mentioned predetermined algorithm, specifically, an algorithm for generating a model for detecting outliers, for example, "Algorithm of OneClassSVM". The "OneClassSVM algorithm", which is the model generation algorithm, has three different parameters "P1", "P2", and "P3" as a plurality of parameters for which the user can set values. After setting values for these parameters, a model is generated by applying the learning data to the model generation algorithm, and the generated model is used for evaluation. model. It should be noted that "P1", "P2", and "P3" as the names of a plurality of parameters here are names for convenience of explanation.

More specifically, for the processing of SA4, for example, the model generation algorithm of the recording unit 13 is acquired, and arbitrary fixed values are set for two of the three parameters of the acquired model generation algorithm. Set one by one, and for the remaining one parameter of the three parameters, set 10 values in order, and transfer the payload information of the learning data acquired in SA3 to the model generation algorithm are sequentially applied to perform machine learning, one evaluation model corresponding to each of the 10 values is generated, and a total of 10 evaluation models are generated.

It should be noted that which of the three parameters of the model generation algorithm here should be the parameter for setting 10 values, and the values to be set for each parameter (one value each and 10 value) may be determined in advance by the user based on the specifications of the model generation algorithm, may be determined in advance by the user based on experiments or simulations, or may be determined randomly. Alternatively, they may be determined by implementing arbitrary logic for determining them and executing the logic, and processing may be executed according to these determined contents. In the present embodiment, of the above-mentioned "P1", "P2", and "P3", "P1" is a parameter for setting ten values, and "P2" and "P3" are one. A case where each value is set as a parameter will be described.

Here, for example, a "OneClassSVM algorithm" that is a model generation algorithm (not shown) of the recording unit 13 is acquired, and three parameters of the acquired "OneClassSVM algorithm", "P1", "P2", and Two parameters of "P3", "P2" and "P3", are set to arbitrary fixed values one by one, and the remaining one parameter of the three parameters is For "P1", 10 mutually different values "first value to tenth value" are sequentially set, and the payload information of the learning data acquired in SA3 is sent to the model generation algorithm By sequentially applying, one evaluation model corresponding to each of the ten values is generated, and a total of ten evaluation models are generated. More specifically, for example, after setting the “first value” to “P1”, “D1” to “D9” in FIG. to generate an evaluation model specified by "ML11-1" in FIG. 3, which is an evaluation model when "P1" is the "first value", and After setting the "second value" to "P1", machine learning by applying "D1" to "D9" etc. in FIG. to generate an evaluation model specified by "ML11-2" in FIG. 3, which is an evaluation model when "P1" is the "second value". After that, after setting "third value" to "tenth value" in "P1", "D1" to "D9" in FIG. "ML11-3" in FIG. 3, which is a model for evaluation when "P1" is "third value" to "tenth value" by performing machine learning by applying it to "OneClass SVM algorithm" . . . to generate an evaluation model specified by “ML11-10”.

Here, as described above, communication data determined to be normal (that is, payload information) is used as learning data, and mutually different "first values" ~ By setting the "tenth value", an evaluation model specified by "ML11-1" to "ML11-10", which are evaluation models that detect outliers based on mutually different criteria, is generated. It will be.

At SA5 in FIG. 4, the processing unit 142 stores the evaluation model. Specifically, it is arbitrary, but for example, after specifying the data ID that is the unit acquired in SA3 and acquiring the evaluation model generated in SA4, the specified data ID and the acquired evaluation model are associated with each other and stored as data ID and model information in FIG. Here, for example, after specifying "ID11" which is the data ID that is the unit acquired in SA3, and acquiring the specified evaluation model that is the evaluation model generated in SA4, " data ID”=“ID11” and “model information”=“ML11-1” to “ML11-10” are stored.

At SA6 in FIG. 4, the processing unit 142 determines whether or not to end the processing. Specifically, it is arbitrary, but for example, an arbitrary method (for example, each group classified by SA2 It is assumed that information indicating whether or not an evaluation model has been generated by acquiring is recorded in the recording unit 13, and determination is made by a method of determination focusing on this information, etc.), and determination is made based on the determination result . Then, when it is determined that all the learning data of each group classified in SA2 is not acquired and the corresponding evaluation model is generated (that is, in each group classified in SA2, the corresponding evaluation model is If there are still some that have not been generated yet), it is determined that the process is not to end (NO in SA6), and the process proceeds to SA3. Also, when it is determined that all of the learning data for each group classified in SA2 has been acquired and the corresponding evaluation model has been generated (that is, in each group classified in SA2, all the corresponding evaluation models have been generated). ), the process is determined to end (YES at SA6), and the process ends. Here, for example, by repeating the above-described processes, the group corresponding to "data ID" = "ID11", the group corresponding to "data ID" = "ID12", and the "data ID" classified in SA2 ==“ID13” etc., when corresponding evaluation models are generated, it is determined that the process is finished. Then, in this manner, the first model specifying information of FIG. 3 is generated and stored. This completes the model generation process.

(processing-analytical processing)
Next, analysis processing will be described. "Analysis processing" is roughly a process of evaluating analysis target data using an evaluation model. FIG. 5 is a flowchart of analysis processing. Although the timing of executing this analysis process is arbitrary, for example, after the data to be analyzed and the first model specifying information are recorded in the recording unit 13, the user performs a predetermined operation via the input unit 11. will be started, and the description will start from the point at which the startup is started.

At SB1 in FIG. 5, the acquisition unit 141 acquires analysis target data. Specifically, it is arbitrary, but for example, the analysis target data of the analysis target data specifying information of the recording unit 13 in FIG. 1 is acquired. Here, for example, as shown in FIG. 2, analysis target data in which row information, time information, data ID, and payload information are associated with each other is acquired.

At SB2 in FIG. 5, the processing unit 142 classifies the analysis target data based on the data ID. Specifically, although it is arbitrary, for example, the same processing as SA2 in FIG. 4 is performed. Specifically, the analysis target data acquired in SB1 is acquired, and the data ID of the analysis target data is used as a reference to classify the same data ID into the same group. Here, for example, the data to be analyzed acquired in SB1 are divided into a group corresponding to "data ID" = "ID11" in FIG. 2, a group corresponding to "data ID" = "ID12" (not shown), and It is classified into a group or the like corresponding to "data ID"="ID13". By classifying in this way, the following processing can be performed for each data ID.

At SB3 in FIG. 5, the processing unit 142 acquires data for one processing unit. Specifically, although it is arbitrary, for example, the same processing as SA3 in FIG. 4 is performed. Specifically, the analysis target data of the group corresponding to one data ID that has not yet been acquired among the groups classified in SB2 is acquired. Here, for example, the analysis target data (that is, line information, time information, data ID, and payload information) of the group corresponding to "data ID" = "ID11" in Fig. 2 is acquired.

At SB4 in FIG. 5, the processing unit 142 executes evaluation processing. The “evaluation process” is a process of evaluating data to be analyzed and storing the evaluation result. FIG. 6 is a flowchart of evaluation processing.

In SC1 of FIG. 6, the processing unit 142 selects an evaluation model. Specifically, it is arbitrary, but for example, a data ID corresponding to the analysis target data acquired in SB3 of FIG. 5 is specified, and the first model specifying information of FIG. Acquire the model information to be used, and select the evaluation model specified by the acquired model information. Here, for example, "ID11", which is the data ID corresponding to the analysis target data acquired in SB3 of FIG. 5, is specified, and the specified data ID " "ML11-1" to "ML11-10", which are model information corresponding to "ID11", are acquired, and the evaluation model specified by "ML11-1" to "ML11-10" is selected.

In SC2 of FIG. 6, the processing unit 142 evaluates the analysis target data. Specifically, it is arbitrary, but for example, as described above, the "evaluation model" is input with analysis target data, and in the input analysis target data, the payload information of the analysis target data is learning data It is a model that detects whether or not it corresponds to a so-called outlier, which is a relatively large outlier value based on Focusing on one thing, the following processing is performed. Specifically, the analysis target data acquired in SB3 of FIG. 5 is input to all the evaluation models selected in SC1 of FIG. 6, the information output from each evaluation model is acquired, and based on the acquired information Then, an evaluation score is calculated and generated for each piece of analysis target data. The "evaluation score" is evaluation result information, for example, information indicating the degree of possibility that the data to be analyzed is abnormal. It is the information of the index related to discomfort. Note that the method of calculating the evaluation score here is arbitrary, but for example, for each analysis target data, the analysis target data is detected as an outlier with respect to the number of evaluation models to which the analysis target data is input. Then, the ratio of the number of the evaluation models is calculated, and the calculated ratio is used as the evaluation score.

Here, for example, the analysis target data of the group corresponding to "data ID"="ID11" in FIG. 2, which is the analysis target data acquired at SB3 in FIG. information) is input to all the evaluation models specified by “ML11-1” to “ML11-10” in FIG. 3, which are the evaluation models selected in SC1 in FIG. 6, and the information output from each evaluation model is obtained, and an evaluation score is calculated and generated for each piece of analysis target data based on the obtained information. Here, a detailed description will be given by focusing on the analysis target data of "line information" = "1" in "data ID" = "ID11" in Fig. 2 . Specifically, the data to be analyzed with “line information” = “1” is input to the 10 evaluation models specified by “ML11-1” to “ML11-10” in FIG. ” to “ML11-9” identify 9 evaluation models, “payload information” ="D1" corresponding to ``row information'' ="1" is detected as an outlier, and the 9 evaluation models When the model outputs information including that the analysis target data of "row information" = "1" is an abnormal value, for "10" which is the number of evaluation models to which each analysis target data , calculate “90%” by calculating “9 ÷ 10” for the ratio of “9”, which is the number of evaluation models that detected each analysis target data as an outlier, and is the calculated ratio Let “90%” be the evaluation score of the analysis target data with “line information”=“1”. Since each process described here is also performed for the analysis target data such as "line information" = "2" to "9", the analysis target data such as "line information" = "2" to "9" An evaluation score is also calculated for

At SC3 in FIG. 6, the processing unit 142 stores the evaluation result of the analysis target data. Specifically, although it is arbitrary, for example, an evaluation score is acquired for each analysis target data, and the acquired evaluation score is recorded in the recording unit 13 in a state of being associated with the analysis target data corresponding to the evaluation score. Here, for example, "90%", which is the evaluation score of the analysis target data with "row information" = "1", and evaluation scores of the analysis target data such as "row information" = "2" to "9" are acquired. Then, information in which the acquired "90%" and the analysis target data of "row information" = "1" are associated, and evaluation scores such as "row information" = "2" to "9" and "row information ”=“2” to “9”, which are associated with data to be analyzed. Then, the evaluation process is returned.

At SB5 in FIG. 5, the processing unit 142 determines whether or not to end the processing. Specifically, although it is arbitrary, for example, the same processing as SA6 in FIG. 4 is performed. Specifically, whether or not all the analysis target data of each group classified at SB2 in FIG. is recorded in the recording unit 13, and determination is made by a determination method focusing on this information, etc.), and determination is made based on the determination result. Then, if it is determined that all the analysis target data of each group classified at SB2 has not been acquired and evaluated (that is, if there is still some data that has not been evaluated in each group classified at SB2) ), it is determined that the process should not be terminated (NO at SB5), and the process proceeds to SB3. Also, when it is determined that all the analysis target data of each group classified at SB2 has been acquired and evaluated (that is, when all evaluations have been performed for each group classified at SB2), the process is terminated. (YES at SB5), and shifts to SB6. Here, for example, by repeating the above-described processing, a group corresponding to "data ID" = "ID11" in FIG. And when all the groups corresponding to the group corresponding to "data ID"="ID13" in FIG. 2 have been evaluated, the process proceeds to SB6.

At SB6 in FIG. 5, the processing unit 142 outputs the evaluation result. Specifically, it is arbitrary, but for example, the user inputs the display threshold score via the input unit 11, and the display threshold score or more of the evaluation scores that are the evaluation results recorded in SC3 of FIG. An evaluation score is selected and acquired, and information in which the acquired evaluation score is associated with the analysis target data corresponding to the evaluation score is displayed and output on the display of the output unit 12 . The “display threshold score” is information that serves as a criterion for selecting the evaluation score to be displayed. Here, for example, when the user inputs "80%" as the display threshold score via the input unit 11, the evaluation score equal to or higher than the display threshold score among the evaluation scores recorded in SC3 of FIG. Select and acquire the above-mentioned "90%" etc., and the acquired evaluation score "90%" is the analysis target data corresponding to the evaluation score "row information" = "1", "time information ”=“t1”, “data ID”=“ID11”, and “payload information”=“D1” are displayed on the display of the output unit 12 and output. In practice, when there is data with a display threshold score of "80%" or higher in addition to the analysis target data with "line information" = "1", information about the data is displayed. This completes the analysis process.

(Effect of this embodiment)
According to the present embodiment, by performing evaluation-related processing related to evaluation performed using a plurality of evaluation models, for example, analysis target data (evaluation target data) performed using a plurality of evaluation models can be evaluated, it is possible to improve the accuracy of the evaluation of the evaluation target data.

In addition, by performing model generation processing (evaluation-related processing) and analysis processing (evaluation-related processing) for each data ID that is an attribute of analysis target data (evaluation-related data) or learning data (evaluation-related data), for example, Since evaluation-related processing can be performed in units of evaluation-related data having similar attributes, evaluation-related processing can be performed efficiently and appropriately.

In addition, based on learning data (teacher data) and a model generation algorithm (predetermined algorithm for machine learning), by performing a process of generating a plurality of evaluation models by machine learning, for example, a plurality of evaluation models It is possible to appropriately generate a model for

In addition, by changing the values of some of the parameters of the "OneClassSVM algorithm" to generate a plurality of evaluation models, for example, a plurality of evaluation models can be generated more appropriately. becomes possible.

Further, by performing a process of evaluating the evaluation target data based on the analysis target data (evaluation-related data) and a plurality of evaluation models, it is possible to improve the evaluation accuracy of the evaluation target data, for example.

In addition, the data to be analyzed (evaluation-related data) is evaluated using each of the multiple evaluation models, and an evaluation score (evaluation result information) that indicates the results of each evaluation performed using each of the multiple evaluation models is generated. By performing the processing to do so, for example, it is possible to appropriately indicate the results of the evaluation of the evaluation target data performed using each of the plurality of evaluation models.

Further, by performing a process of evaluating the data values included in the analysis target data (evaluation-related data), for example, it is possible to evaluate the data values.

(Embodiment 2)
Next, Embodiment 2 will be described. This embodiment is a form that evaluates the time included in the evaluation target data. Descriptions of the same contents as those of the first embodiment will be omitted as appropriate, and only the features that are characteristic of the present embodiment will be described in detail.

(composition)
First, an information processing apparatus according to this embodiment will be described. FIG. 7 is a block diagram functionally conceptually showing the information processing apparatus according to the present embodiment. The information processing device 2 is a processing system, such as a personal computer, and includes an input unit 21, an output unit 22, a recording unit 23, and a control unit 24, for example. The input unit 21 and the output unit 22 are the same as those of the first embodiment with the same names.

(Construction-recording unit)
The recording unit 23 is recording means for recording programs and various data necessary for the operation of the information processing device 2, and is configured in the same manner as the recording unit 13 of the first embodiment. , learning data specification information, second model specification information, and timing determination model specification information are stored. Note that the analysis target data specifying information and the learning data specifying information are the same as the information with the same name in the first embodiment.

(Configuration-Recording Unit-Second Model Specific Information)
The “second model identification information” of the recording unit 23 is information for identifying a plurality of evaluation models. This is information that identifies the model of FIG. 8 is a diagram exemplifying the second model specifying information. For example, as shown in FIG. 8, the 'second model identification information' is information in which the item 'data ID', the item 'model information', and the information corresponding to each item are associated with each other. Each of these pieces of information is the same as the information with the same name in FIG. In the present embodiment, the "evaluation model" is a value in which the time difference data of the analysis target data is input, and the input time difference data is a value that deviates relatively greatly from the learning data. , a model for detecting whether or not the data corresponds to a so-called outlier, and a model for outputting information indicating analysis target data corresponding to the outlier. The “time difference data” is information corresponding to the difference between continuous time information of the analysis target data, for example, the difference between “t1” and “t2” in FIG. and "t3" and the difference between "t3" and "t4". The method of storing such second model specifying information is arbitrary, but for example, the second model specifying information is stored by executing a model generation process to be described later.

(Configuration - recording unit - model specific information for timing determination)
The “timing determination model identification information” in the recording unit 23 is information that identifies the timing determination model. The "timing determination model" is the above-described determination model. For example, based on the communication timing in CAN communication, for determining the types of a plurality of time difference data in arbitrary units (for example, data ID units). is a model of Regarding the types assumed here, various assumptions are made based on the communication specifications of CAN communication, etc. Any type of "normal communication" in which communication is performed, "normal communication with pause" in which the above-mentioned normal communication is temporarily interrupted and a temporary stop occurs, and "event communication" in which communication is performed at random timing is assumed. The communication specifications will be described below. A specific timing determination model used here and a method of storing the timing determination model specifying information for specifying the model will be described later.

(Configuration - control unit)
Returning to FIG. 7, the control unit 24 is control means for controlling the information processing device 2, and is configured in the same manner as the control unit 14 of the first embodiment. and a processing unit 242 . The acquisition unit 241 and the processing unit 242 are the same as the components with the same names in the first embodiment. Processing of each part of the control part 24 will be described later.

(process)
Next, processing executed by the information processing device 2 according to the present embodiment will be described. Specifically, model generation processing and analysis processing will be described.

(Processing - Model generation processing)
First, model generation processing will be described. The "model generation process" is roughly a process of generating an evaluation model using learning data, for example, a process of generating and storing the second model specifying information in FIG. FIG. 9 is a flowchart of model generation processing.

In SD1 of FIG. 9, the acquisition unit 241 acquires learning data. Specifically, although it is arbitrary, for example, the same processing as SA1 in FIG. 4 is performed. Specifically, for example, the learning data specified by the learning data specifying information of the recording unit 23 in FIG. 8 is acquired. Here, for example, learning data that is the same type of data as the analysis target data in FIG. 2 and in which row information, time information, data ID, and payload information are associated with each other is acquired.

In SD2 of FIG. 9, the processing unit 242 classifies the learning data based on the data ID. Specifically, although it is arbitrary, for example, the same processing as SA2 in FIG. 4 is performed. The learning data acquired in SD1 is acquired, and the data ID of the learning data is used as a reference to classify the same data ID into the same group. Here, for example, it is assumed that information corresponding to "data ID" = "ID11", "ID13" and "ID12" (not shown) in FIG. classify the learning data into a group corresponding to "data ID" = "ID11", a group corresponding to "data ID" = "ID12", a group corresponding to "data ID" = "ID13", etc. do.

At SD3 in FIG. 9, the processing unit 242 acquires data for one processing unit. Specifically, although it is arbitrary, for example, the same processing as SA3 in FIG. 4 is performed. Specifically, for example, the learning data of the group corresponding to one data ID that has not yet been acquired among the groups classified in SD2 is acquired. Here, for example, the learning data (that is, line information, time information, data ID, and payload information) of the group corresponding to "data ID" = "ID11" is acquired.

At SD4 in FIG. 9, the processing unit 242 executes time difference data generation processing. The “time difference data generation process” is a process of generating time difference data and appropriately excluding the generated time difference data. FIG. 10 is a flowchart of time difference data generation processing.

At SE1 in FIG. 10, the processing unit 242 generates a plurality of time difference data. Specifically, although it is arbitrary, for example, time difference data is generated by calculating the difference between continuous time information pieces of the learning data acquired in SD3 of FIG. Here, for example, when "tg1", "tg2", "tg3", "tg4" and the like are stored as continuous time information of the learning data (not shown), "tg2-tg1", " The calculation results such as "tg3-tg2" and "tg4-tg3" are used as time difference data.

Incidentally, as the plurality of time difference data generated here (specifically, the plurality of time difference data having the same data ID), various assumptions are made based on the communication specifications of CAN communication, etc., but the above-mentioned Since "normal communication", "normal communication with pause", and "event communication" are assumed as described above, a plurality of pieces of time difference data corresponding to any of these are generated.

At SE2 in FIG. 10, the processing unit 242 acquires the statistical value and the number. Specifically, it is arbitrary, but for example, a plurality of time difference data generated in SE1 are acquired, a known statistical process is performed on the acquired plurality of time difference data, and the maximum value of the plurality of time difference data, The variance, standard deviation, average value, and median value are obtained as statistical values, the obtained statistical values are obtained, and the number of pieces of time difference data generated in SE1 is counted to obtain the counted number. Here, for example, "tg2-tg1", "tg3-tg2", and "tg4-tg3" as the maximum value, variance, standard deviation, average value, and median value, "Tm", "Tv", " Ts”, and “Tm” are acquired, and “Tnum” is acquired as the number of “tg2-tg1”, “tg3-tg2”, and “tg4-tg3” (the notation of “Tm” etc. are for convenience of explanation).

At SE3 in FIG. 10, the processing unit 242 identifies the type of the time difference data generated at SE1. Specifically, it is arbitrary, but for example, the “timing determination model” specified by the timing determination model specification information of the recording unit 23 in FIG. The number of time difference data is input, and based on the input information, whether the time difference data of the input information corresponds to "normal communication", "normal communication with pause", or "event communication" A case will be described where the model outputs Such a timing determination model is generated by performing arbitrary machine learning (for example, known supervised learning, etc.) in advance, and the timing for specifying the generated timing determination model is The following description assumes that the judgment model is stored in the recording unit 23 of FIG.

More specifically, in SE3 of FIG. 10, for example, the "timing determination model" specified by the timing determination model specifying information of the recording unit 23 in FIG. , and the number thereof, and input the obtained variance and the number to the obtained timing determination model to specify the type indicated by the information output from the timing determination model. Specifically, when "normal communication" is output from the timing determination model, the type of time difference data is specified as "normal communication", and "normal communication with pause" is output from the timing determination model. In this case, the type of time difference data is identified as "normal communication with pause", and when "event communication" is output from the timing determination model, the type of time difference data is identified as "event communication".

Here, for example, the “timing determination model” specified by the timing determination model specification information of the recording unit 23 in FIG. , and input the acquired “Tv” and “Tnum” to the acquired timing determination model to specify the type indicated by the information output from the timing determination model.

Specifically, for example, when the time difference data generated in SE1 corresponds to "normal communication", examples of the time difference data are "1.0", "1.3", " 1.2", "1.2", "1.2", "1.1" ... (the unit "seconds" is omitted, the same applies hereinafter), etc., and the corresponding dispersion and the number are input to the timing determination model, "normal communication" is output from the timing determination model, and "normal communication" is specified. Further, for example, when the time difference data generated in SE1 corresponds to "normal communication with pause", examples of the time difference data are "1.0", "1.3", "1.2 ", "6.2", "1.2", "1.1", etc. are included, and when the variance and the number corresponding to these are input to the timing determination model, the timing determination "Normal communication with pause" is output from the model, and "normal communication with pause" is specified. Further, for example, when the time difference data generated in SE1 corresponds to "normal communication", examples of the time difference data are "5.0", "1.3", "1.2", "4.2", "1.2", "20.1", etc. are included, and when the variance and the number corresponding to these are input to the timing determination model, the timing determination model "Event communication" is output, and "event communication" is specified.

At SE4 in FIG. 10, the processing unit 242 processes all of the time difference data generated at SE1 (that is, data for one unit of processing and based on time information corresponding to one data ID). Determine whether to target or not. Specifically, it is arbitrary, but for example, it is determined based on the type of time difference data specified in SE3. Then, if the type of the specified time difference data is "normal communication", it is determined that all the generated time difference data are to be processed (YES in SE4), and the process returns. In addition, if the type of the specified time difference data is not "normal communication" ("normal communication with pause" or "event communication"), not all the generated time difference data will be processed. It is determined (that is, it is determined that at least part of the time difference data is not to be processed) (NO in SE4), and the process proceeds to SE5. Here, for example, since "normal communication" is specified in SE3, it is determined that all the generated time difference data are to be processed.

At SE5 in FIG. 10, the processing unit 242 processes all of the time difference data generated at SE1 (that is, data for one unit of processing and based on time information corresponding to one data ID). Determine whether or not to exclude from the target. Specifically, it is arbitrary, but for example, it is determined based on the type of time difference data specified in SE3. Then, if the type of the specified time difference data is "event communication", it is determined that all the generated time difference data are to be excluded from the processing targets (YES in SE5), and the process returns. Also, if the type of the identified time difference data is not "event communication" (if it is "normal communication with pause"), it is determined that all the generated time difference data are not excluded from the processing target (i.e. , only part of the time difference data is to be processed) (NO in SE5), and the process proceeds to SE6.

At SE6 in FIG. 10, the processing unit 242 processes part of the time difference data generated at SE1 (that is, data for one unit of processing and based on time information corresponding to one data ID). Exclude data from Specifically, it is arbitrary, but for example, the time difference data generated in SE1 is acquired, and in the acquired time difference data, the time difference data corresponding to the temporary interruption of normal communication in CAN communication is specified. A method (for example, a method for identifying known outliers using a clustering technique such as DBSCAN (Density-based spatial clustering of applications with noise) (a method different from the above-described method using "OneClassSVM"), or A method of simply comparing with a threshold value (for example, a method of identifying time difference data equal to or greater than the threshold value of 1.5 seconds as the time difference data corresponding to the suspension, etc.) Identify time difference data and exclude the identified time difference data. Note that "excluding time difference data" is a concept corresponding to not using the excluded time difference data in subsequent processing. Here, for example, as described above, the time difference data generated in SE1 are "1.0", "1.3", "1.2", "6.2", "1.2", "1 .1”, etc. are included, “6.2” is excluded. After that, the time difference data generation process is returned.

At SD5 in FIG. 9, the processing unit 242 determines whether or not to generate an evaluation model. Specifically, it is arbitrary, but for example, in the time difference data generation process of FIG. 10, it is determined based on whether or not it is determined that all the time difference data are to be excluded from the processing targets. If it is determined in SE5 of FIG. 10 that all the time difference data are to be excluded from the processing targets, it means that there is no time difference data to be processed, so it is determined that the evaluation model is not to be generated. (NO in SD5), and the process ends. Also, if it is determined in SE5 of FIG. 10 that all the time difference data are not to be excluded from the processing targets, there is time difference data to be processed, so an evaluation model is generated. (YES in SD5), and proceeds to SD6. Here, for example, if the time difference data generated in SE1 corresponds to "normal communication" or "normal communication with pause", it is determined that an evaluation model is to be generated. If the data corresponds to "event communication", it will be determined that the evaluation model is not to be generated.

At SD6 in FIG. 9, the processing unit 242 generates a plurality of evaluation models. Although it is specifically arbitrary, for example, it is generated roughly in the same manner as the processing of SA4 in FIG. ' is used in SD6 of FIG. 9, unlike the case of SA4, SD6 of FIG. 9 is generated in the same manner except that the time difference data generated at SE1 of FIG.

More specifically, for example, the time difference data generated in SE1 of FIG. 10 is obtained first. It should be noted that if some of the time difference data is excluded in SE6 of FIG. 10, data other than the excluded time difference data is obtained from among the time difference data generated in SE1. Next, a model generation algorithm (not shown) in the recording unit 23 is acquired, and two of the three parameters of the acquired model generation algorithm are set to arbitrary fixed values one by one, Further, for the remaining one parameter of the three parameters, 10 values are sequentially set, and the time difference data obtained above (that is, the time difference data generated in SE1 in FIG. 10, or , the remaining data partially excluded in the time difference data generated in the SE1) are sequentially applied to the model generation algorithm to generate one evaluation model corresponding to each of the ten values. to generate a total of 10 evaluation models. Then, here, for example, CAN communication is continuously performed at predetermined time intervals (eg, 1.0 to 1.5 second intervals) by each determination and data exclusion of the time difference data generation process described above. It is possible to generate an evaluation model for detecting whether or not an outlier corresponds to an outlier, assuming that the current communication is normal.

Here, for example, when the time difference data generated in SE1 corresponds to "normal communication", examples of the time difference data are "1.0", "1.3", and "1.2". , "1.2", "1.2", "1.1", etc. are obtained, and as another example, the time difference data generated in SE1 corresponds to "normal communication with pause". If there is, as an example of the time difference data, "1.0", "1.3", "1.2", "6.2", "1.2", "1.1"... etc. are included, but since "6.2" is excluded in SE6 as described above, "1.0", "1.3", "1.2", "1.2", " 1.1” and so on. Next, the “OneClass SVM algorithm” that is the model generation algorithm of the recording unit 23 is acquired, and the three parameters “P1”, “P2”, and “P3” of the acquired “OneClass SVM algorithm” are One arbitrary fixed value is set for each of the two parameters "P2" and "P3", and the remaining one parameter of the three parameters "P1" , by sequentially setting "first value to tenth value", which are ten mutually different values, and sequentially applying the above-mentioned acquired time difference data to the model generation algorithm, the ten A model for evaluation corresponding to each value of 1 is generated one by one, and a total of 10 models for evaluation are generated. More specifically, for example, after setting the “first value” to “P1”, the above-mentioned “1.0”, “1. Machine learning is performed by applying ".3", "1.2", "1.2", "1.2", "1.1", etc. to the "Algorithm of OneClass SVM", and "P1 is the "first value", the evaluation model specified by "MT11-1" in FIG. 8 is generated. After that, by setting "second value" to "tenth value" to "P1" and performing the same processing, "MT11-3" to "MT11-10" in FIG. 8 are specified. Generate an evaluation model for

At SD7 in FIG. 9, the processing unit 242 stores the evaluation model. Specifically, although it is arbitrary, for example, the same processing as SA5 in FIG. 4 is performed. Specifically, after specifying the data ID, which is the unit acquired in SD3 of FIG. 9, and acquiring the evaluation model generated in SD6, the specified data ID and the information specifying the acquired evaluation model are associated with each other and stored as the data ID and model information in FIG. Here, for example, after specifying "ID11" which is the data ID that is the unit acquired in SD3, and acquiring the specified evaluation model that is the evaluation model generated in SD6, " data ID”=“ID11” and “model information”=“MT11-1” to “MT11-10” are stored.

At SD8 in FIG. 9, the processing unit 242 determines whether or not to end the processing. Specifically, although it is arbitrary, for example, the same processing as SA6 in FIG. 4 is performed. Specifically, it is determined by an arbitrary method whether or not all of the learning data for each group classified in SD2 has been acquired and the corresponding evaluation model has been generated, and determination is made based on the determination result. If it is determined that the learning data for each group classified in SD2 is not all obtained and the corresponding evaluation model is generated, it is determined that the process is not to be terminated (NO in SD8), and the process proceeds to SD3. Transition. Also, if it is determined that all the learning data of each group classified in SD2 has been acquired and the corresponding evaluation model has been generated, it is determined that the processing is terminated (YES in SD8), and the processing is terminated. Here, for example, the above-described processing is repeatedly performed, and the group corresponding to "data ID" = "ID11", the group corresponding to "data ID" = "ID12", and the group corresponding to "data ID" = "data ID" = " When the corresponding evaluation models are generated for all the groups corresponding to ID13", etc., it is determined that the process is finished. Then, in this manner, the second model specifying information of FIG. 8 is generated and stored. This completes the model generation process.

(processing-analytical processing)
Next, analysis processing will be described. "Analysis processing" is roughly a process of evaluating analysis target data using an evaluation model. FIG. 11 is a flowchart of analysis processing. Although the timing of executing this analysis process is arbitrary, for example, after the data to be analyzed and the second model specifying information are recorded in the recording unit 23, the user performs a predetermined operation via the input unit 21. will be started, and the description will start from the point at which the startup is started.

In SF1 of FIG. 11, the acquisition unit 241 acquires analysis target data. Specifically, although it is arbitrary, for example, the same processing as SB1 in FIG. 5 is performed. Specifically, the analysis target data of the analysis target data specifying information in the recording unit 23 of FIG. 7 is acquired. Here, for example, as shown in FIG. 2, analysis target data in which row information, time information, data ID, and payload information are associated with each other is acquired.

In SF2 of FIG. 11, the processing unit 242 classifies the analysis target data based on the data ID. Specifically, although it is arbitrary, for example, the same processing as SD2 in FIG. 9 is performed. Specifically, the analysis object data acquired in SF1 is acquired, and the same data ID is classified into the same group based on the data ID of the analysis object data. Here, for example, the data to be analyzed acquired in SD1 are divided into a group corresponding to "data ID" = "ID11" in FIG. 2, a group corresponding to "data ID" = "ID12" (not shown), and It is classified into a group or the like corresponding to "data ID"="ID13".

In SF3 of FIG. 11, the processing unit 242 acquires data of one processing unit. Specifically, although it is arbitrary, for example, the same processing as SD3 in FIG. 9 is performed. Specifically, the analysis target data of the group corresponding to one data ID that has not yet been acquired among the groups classified in SF2 is acquired. Here, for example, the analysis target data (that is, line information, time information, data ID, and payload information) of the group corresponding to "data ID" = "ID11" in Fig. 2 is acquired.

In SF4 of FIG. 11, the processing unit 242 executes time difference data generation processing. The time difference data generation process executed here is the same as the time difference data generation process executed at SD4 in FIG. Each step of the time difference data generation process in FIG. 10 is denoted by "G", and only the outline will be described.

In SG1 of FIG. 10, the processing unit 242 generates a plurality of time difference data. Specifically, although it is arbitrary, for example, time difference data is generated by calculating the difference between continuous time information of the analysis target data acquired in SF3 of FIG. In particular, for the time difference data generated here, it is possible to specify which of the time difference data between the analysis target data corresponds to the two analysis target data that are the basis of the time difference data. It is assumed that it is managed by the processing unit 242 in association with two pieces of line information. Here, for example, the calculation results such as "t2-t1", "t3-t2", and "t4-t3" in the time information of FIG. 2 are used as time difference data. For these "t2-t1", "t3-t2", and "t4-t3", etc., "line information" = "2" and "1", "3" and "2", "4" and Assume that it is managed in association with "3".

In SG2 of FIG. 10, the processing unit 242 acquires the statistical value and the number. Specifically, it is arbitrary, but for example, a plurality of time difference data generated in SG1 are acquired, a known statistical process is performed on the acquired plurality of time difference data, and the maximum value of the plurality of time difference data, The variance, standard deviation, average value, and median value are obtained as statistical values, the obtained statistical values are obtained, and the number of pieces of time difference data generated in SG1 is counted to obtain the counted number. Here, for example, "Tam", "Tav", "Tam", "Tav", "Tam", "Tav", " Tas” and “Tam” are obtained, and “Tanum” is obtained as a number such as “t2-t1”, “t3-t2”, and “t4-t3”.

In SG3 of FIG. 10, the processing unit 242 identifies the type of time difference data generated in SG1. Specifically, it is arbitrary, but for example, the “timing determination model” specified by the timing determination model specification information of the recording unit 23 in FIG. The number is acquired, and the acquired variance and number are input to the acquired timing determination model, and the type indicated by the information output from the timing determination model is specified.

Here, for example, the “timing determination model” specified by the timing determination model specification information of the recording unit 23 in FIG. ” is input to the acquired timing determination model to specify the type indicated by the information output from the timing determination model.

In SG4 of FIG. 10, the processing unit 242 processes all of the time difference data generated in SG1 (that is, data in one unit of processing and based on time information corresponding to one data ID). Determine whether to target or not.

In SG5 of FIG. 10, the processing unit 242 processes all the time difference data generated in SG1 (that is, data in one unit of processing and based on time information corresponding to one data ID). Determine whether or not to exclude from the target.

In SG6 of FIG. 10, the processing unit 242 processes part of the time difference data generated in SG1 (that is, data in one unit of processing and based on time information corresponding to one data ID). Exclude data from After that, the time difference data generation process is returned.

In SF5 of FIG. 11, the processing unit 242 determines whether or not to execute the evaluation process. Specifically, although it is arbitrary, for example, the same processing as SD5 in FIG. 9 is performed. Specifically, in the time difference data generation process of FIG. 10, it is determined based on whether or not it is determined that all the time difference data should be excluded from the processing targets. Then, in SG5 of FIG. 10, if it is determined that all the time difference data are to be excluded from the processing targets, it means that there is no time difference data to be processed, so it is determined that the evaluation processing is not to be executed. (NO in SF5), the process ends. If it is determined in SG5 of FIG. 10 that all the time difference data are not to be excluded from the processing target, there is time difference data to be processed, so evaluation processing is executed. (YES in SF5), and shifts to SF6. Here, for example, when the time difference data generated in SG1 corresponds to "normal communication" or "normal communication with pause", it is determined that the evaluation process is to be executed, while the time difference data generated in SG1 corresponds to "event communication", it is determined that the evaluation process is not to be executed.

In SF6 of FIG. 11, the processing unit 242 executes evaluation processing. The evaluation process executed here is the same as the evaluation process executed at SB4 in FIG. is denoted by "H", and only the outline will be described.

At SH1 in FIG. 6, the processing unit 242 selects an evaluation model. Specifically, although it is arbitrary, for example, the same processing as SC1 is performed. Specifically, the data ID corresponding to the data to be analyzed acquired in SF3 of FIG. 11 is specified, the second model specifying information of FIG. 8 is referred to acquire the model information corresponding to the specified data ID, Select the model for evaluation specified by the acquired model information. Here, for example, "ID11", which is the data ID corresponding to the analysis target data acquired in SF3 of FIG. 11, is specified, and the specified data ID " 'MT11-1' to 'MT11-10', which are model information corresponding to 'ID11', are acquired, and an evaluation model specified by 'MT11-1' to 'MT11-10' is selected.

At SH2 in FIG. 6, the processing unit 242 evaluates the analysis target data. Specifically, it is arbitrary, but for example, as described above, the "evaluation model" is input with the time difference data of the analysis target data, and the input time difference data is based on the learning data. It is a model that detects whether or not the time difference data corresponds to a so-called outlier, which is a value that is relatively large, and outputs information indicating the analysis target data that corresponds to the outlier. Focusing on this, the following processing is performed. Specifically, the time difference data generated in SG1 of FIG. 10 (Note that if some time difference data is excluded in SG6 of FIG. data other than the difference data) and two pieces of row information of each two pieces of analysis target data that are the basis of the time difference data are associated with each other, and all the evaluation models selected in SH1 of FIG. Input and acquire information output from each evaluation model, and based on the acquired information, calculate and generate an evaluation score for each piece of analysis target data.

Here, for example, "t2-t1", "t3-t2", and "t4-t3" which are the time difference data generated in SG1 of FIG. "Row information" = "2" and "1" (corresponding to "t3-t2"), which is information in combination with each two pieces of row information of the data to be analyzed (corresponding to "t2-t1") “Line information” = “3” and “2”, “Line information” = “4” and “3” (corresponding to “t4-t3”) are associated with each other, and selected in SH1 of FIG. Input to all evaluation models specified by “MT11-1” to “MT11-10” in FIG. 8, which are evaluation models, acquire information output from each evaluation model, and based on the acquired information , an evaluation score is calculated and generated for each analysis target data. Here, a detailed description will be given by focusing on "t2-t1" (that is, time difference data corresponding to the difference between "line information"="2" and "1" in FIG. 2). Specifically, the time difference data of "t2-t1" is input to 10 evaluation models specified by "MT11-1" to "MT11-10" in FIG. In the nine evaluation models specified by "MT11-9", the time difference data of "t2-t1" is detected as an outlier, and the nine evaluation models detect the time difference data of "t2-t1". (for example, the information of "line information" = "2" and "1", which is information specifying the analysis target data that is the basis of "t2-t1") is output, each analysis target The ratio of "9", which is the number of evaluation models for which the time difference data of each analysis target data is detected as an outlier, to "10", which is the number of evaluation models to which the time difference data of the data is input "90%" is calculated by calculating "9/10", and the calculated ratio "90%" is used as the evaluation score of "t2-t1" (that is, "row information" = " 2” and the evaluation score of the time difference data of the time information of the data to be analyzed specified by “1”). Then, each process described here also calculates the evaluation score for the time difference data of the analysis target data such as "t3-t2" and "t4-t3".

At SH3 in FIG. 6, the processing unit 242 stores the evaluation result of the analysis target data. Specifically, although it is arbitrary, for example, an evaluation score is acquired for the time difference data of each analysis target data, and the acquired evaluation score is associated with the analysis target data corresponding to the evaluation score, and the recording unit 13 to record. Here, the time difference data corresponds to the difference in the time information of the two pieces of analysis target data. , or may be stored in association with one of them. A case of matching will be described as an example. Here, for example, "90% ”, etc., and the acquired “90%” and “row information” = “1” analysis target data (that is, “row information” which is information that specifies the analysis target data that is the basis of “t2-t1” ="2" and "1", which specifies the earlier analysis data ("line information" ="1"). Then, the evaluation process is returned.

At SF7 in FIG. 11, the processing unit 242 determines whether or not to end the processing. Specifically, although it is arbitrary, for example, the same processing as SB5 in FIG. 5 is performed. Then, by repeating the above-described process, the group corresponding to "data ID" = "ID11" in Fig. 2 classified in SF2 in Fig. 11, the group corresponding to "data ID" = "ID12" (not shown), and If all the groups corresponding to the group corresponding to "data ID"="ID13" in FIG. 2 have been evaluated, the process proceeds to SF8.

At SF8 in FIG. 11, the processing unit 242 outputs the evaluation result. Specifically, although it is arbitrary, for example, the same processing as SB6 in FIG. 5 is performed. Specifically, here, for example, when the user inputs "80%" as the display threshold score via the input unit 21, the display threshold score among the evaluation scores that are the evaluation results recorded in SH3 of FIG. Select and acquire the above evaluation score "90%", etc., and set the acquired evaluation score "90%" to the analysis target data corresponding to the evaluation score "row information" = "1" is displayed on the display of the output unit 12 and output. Here, for example, it is configured to output "90%" in a state associated with "line information" = "1" and "2", which are two pieces of data to be analyzed that are the basis of the time difference data. You may Also, if there is information excluded in SG6 of FIG. 10, the excluded information is identified by any method (for example, a list of excluded information is recorded in the recording unit 23, and the list is referenced to specify method, etc.), and the specified excluded information may also be output. This completes the analysis process.

(Effect of this embodiment)
According to the present embodiment, the analysis target data (evaluation related data) or the learning data (evaluation related data) is subjected to model generation processing (evaluation related processing) based on a predetermined timing determination model (determination model). ) and analysis processing (evaluation-related processing), and based on the evaluation-related data determined to be subject to evaluation-related processing, by performing evaluation-related processing, for example, appropriate Evaluation-related processing can be performed using the evaluation-related data.

In addition, by performing a process of evaluating the time included in the evaluation target data, it is possible to perform the evaluation of the time, for example.

(Embodiment 3)
Next, Embodiment 3 will be described. This embodiment is a form in which data values included in evaluation target data are evaluated, and a plurality of evaluation-related data are associated with each other. Descriptions of the same contents as those of the first embodiment will be omitted as appropriate, and only the features that are characteristic of the present embodiment will be described in detail.

(composition)
First, an information processing apparatus according to this embodiment will be described. FIG. 12 is a block diagram functionally conceptually showing the information processing apparatus according to the present embodiment. The information processing device 3 is a processing system, such as a personal computer, and includes an input unit 31, an output unit 32, a recording unit 33, and a control unit 34, for example. The input unit 21 and the output unit 22 are the same as those of the first embodiment with the same names.

(Construction-recording unit)
The recording unit 33 is recording means for recording programs and various data necessary for the operation of the information processing device 3, and is configured in the same manner as the recording unit 13 of the first embodiment. , learning data specifying information, and third model specifying information are stored.

(Configuration-recording unit-analysis target data identification information, learning data identification information)
FIG. 13 is a diagram exemplifying analysis target data identification information, and FIG. 14 is a diagram exemplifying learning data identification information. The analysis target data specifying information and the learning data specifying information recorded in the recording unit 33 of FIG. A case will be described in which data that is being transmitted and received is specified according to a sequence (for example, a sequence in which data corresponding to two data IDs are alternately communicated). Specifically, for example, as shown in FIGS. 13 and 14, data corresponding to “data ID”=“ID11” and data corresponding to “data ID”=“ID21” are alternately displayed according to a predetermined sequence. A case in which data is transmitted and received will be described as an example. In addition, in practice, data with various data IDs other than "data ID" = "ID11" and "ID21" are transmitted and received, but descriptions thereof will be omitted.

(Configuration-recording unit-third model specific information)
The "third model specifying information" of the recording unit 33 is information specifying a plurality of evaluation models. This is information that identifies the model. FIG. 15 is a diagram exemplifying the third model specifying information. For example, as shown in FIG. 15, the 'third model identification information' is information in which the item 'data ID', the item 'model information', and the information corresponding to each item are associated with each other. Each of these pieces of information is the same as the information with the same name in FIG. In the present embodiment, the "evaluation model" is input with a plurality of associated analysis target data, and in the input plurality of associated analysis target data, the payload information of the analysis target data is learning A model that detects whether or not it corresponds to a so-called outlier, which is a relatively large outlier value based on the data for data, and the payload information is a model that detects multiple associated analysis target data that correspond to outliers. A case where the model outputs the information shown will be described. In FIG. 15, the data IDs include "ID11, ID21" indicating that one data ID "ID11" and one data ID "ID21" are associated with each other. stored. The method of storing such third model identification information is arbitrary, but for example, it is stored by executing a model generation process to be described later.

(Configuration - control unit)
Returning to FIG. 12, the control unit 34 is control means for controlling the information processing device 3, and is configured in the same manner as the control unit 14 of the first embodiment. and a processing unit 342 . An acquisition unit 341 and a processing unit 342 are the same as the components with the same names in the first embodiment. Processing of each part of the control part 34 will be described later.

(process)
Next, processing executed by the information processing device 3 according to the present embodiment will be described. Specifically, model generation processing and analysis processing will be described.

(Processing - Model generation processing)
First, model generation processing will be described. The "model generation process" is roughly a process of generating an evaluation model using learning data, for example, a process of generating and storing the third model specifying information in FIG. FIG. 16 is a flowchart of model generation processing.

At SI1 in FIG. 16, the acquisition unit 341 acquires learning data. Specifically, although it is arbitrary, for example, the same processing as SA1 in FIG. 4 is performed. Here, for example, the learning data specified by the learning data specifying information in FIG. 14 (that is, row information, time information, data ID, and payload information) is acquired.

At SI2 in FIG. 16, the processing unit 342 classifies the learning data based on the data ID. Specifically, although it is arbitrary, for example, the same processing as SA2 in FIG. 4 is performed.

At SI3 in FIG. 16, the processing unit 342 associates learning data. Specifically, it is arbitrary, but for example, the above-mentioned predetermined sequence (for example, data corresponding to "data ID" = "ID11" and data corresponding to "data ID" = "ID21" are alternately transmitted and received) sequence, etc.) is recorded in the recording unit 33, a predetermined sequence is identified by referring to this information, and the learning data classified by SI2 are differentiated from each other according to the identified predetermined sequence. Associate between data IDs. The method of associating the learning data is arbitrary, but for example, it is assumed that the transmitted and received data correspond to requests and responses, and a method of associating mutually corresponding request and response data is used. Specifically, a method of associating learning data with the same communication order (order within each data ID) by focusing on the time information of the learning data of each data ID will be described. Here, for example, for "data ID"="ID11" and "ID12" in FIG. The learning data of row numbers "1" and "2" are associated with each other, and similarly, the learning data of row numbers "3" and "4", the learning data of row numbers "5" and "6", The training data of row numbers "7" and "8" are associated with each other. Although not shown in FIG. 14, learning data corresponding to "data ID" = "ID12", "ID22", etc. are also stored, and these data are also alternately transmitted and received according to a predetermined sequence. shall be In this case, the learning data with "data ID" = "ID12" and "ID22" are also associated with each other in the same manner.

At SI4 in FIG. 16, the processing unit 342 acquires data for one processing unit. Specifically, although it is arbitrary, for example, learning data corresponding to a combination of data IDs that have not yet been acquired among the learning data corresponding to the combination of data IDs associated in SI3 is acquired. Here, for example, learning data corresponding to "data ID" = "ID11" and "ID12" associated in SI3 are acquired.

At SI5 in FIG. 16, the processing unit 342 generates a plurality of evaluation models. Specifically, although it is arbitrary, for example, the same processing as the processing of SA4 in FIG. 4 is performed. Specifically, for example, a model generation algorithm (not shown) in the recording unit 33 is acquired, and values are set to the parameters of the acquired model generation algorithm in the same manner as described in SA4 of FIG. A total of 10 evaluation models are generated by sequentially applying the payload information of the learning data acquired in SI4 to the model generation algorithm and performing machine learning. Regarding the payload information to be applied to the model generation algorithm, the payload information associated with each other acquired in SI4 can be obtained by any method (for example, a 16-byte payload that connects each payload information in 8-byte hexadecimal notation information, etc.), and the combined payload information is applied. Here, for example, payload information summarizing "Dg11" and "Dg21" in FIG. 14, payload information summarizing "Dg12" and "Dg22", payload information summarizing "Dg13" and "Dg23", and "Dg14" and payload information summarizing "Dg24" is applied to the "OneClassSVM algorithm", which is an algorithm for model generation, to perform machine learning, and "MM11-1" to "MM11-10" in FIG. 15 are specified Generate an evaluation model for

At SI6 in FIG. 16, the processing unit 342 stores the evaluation model. Specifically, although it is arbitrary, for example, the same processing as the processing of SA5 in FIG. 4 is performed. Here, for example, “data ID”=“ID11, ID21” and “model information”=“MM11-1” to “MM11-10” in FIG. 15 are stored.

At SI7 in FIG. 16, the processing unit 342 determines whether or not to end the processing. Specifically, although it is arbitrary, for example, the same processing as SA6 in FIG. 4 is performed. Then, in this manner, the third model specifying information of FIG. 15 is generated and stored. This completes the model generation process.

(processing-analytical processing)
Next, analysis processing will be described. "Analysis processing" is roughly a process of evaluating analysis target data using an evaluation model. FIG. 17 is a flowchart of analysis processing. The timing of executing this analysis process is arbitrary, but for example, after the analysis target data and the third model specifying information are recorded in the recording unit 33, the user performs a predetermined operation via the input unit 31. will be started, and the description will start from the point at which the startup is started.

In SJ1 of FIG. 17, the acquisition unit 341 acquires analysis target data. Specifically, although it is arbitrary, for example, the same processing as SB1 in FIG. 5 is performed. Here, for example, the analysis target data specified by the analysis target data specifying information of FIG. 13 (that is, line information, time information, data ID, and payload information) is acquired.

In SJ2 of FIG. 17, the processing unit 342 classifies the analysis target data based on the data ID. Specifically, although it is arbitrary, for example, the same processing as SD2 in FIG. 9 is performed.

In SJ3 of FIG. 17, the processing unit 342 associates the analysis target data. Specifically, it is arbitrary, but, for example, similar to SI3 in FIG. 16, the data to be analyzed classified in SJ2 are associated with different data IDs according to a predetermined sequence. Here, for example, for "data ID"="ID11" and "ID12" in FIG. The analysis target data of row numbers "1" and "2" are associated with each other, and similarly, the analysis target data of row numbers "3" and "4", the analysis target data of row numbers "5" and "6", The analysis target data of row numbers "7" and "8" are associated with each other. Although not shown in FIG. 13, analysis target data corresponding to "data ID" = "ID12", "ID22", etc. are also stored, and these "data ID" = "ID12" and " The data to be analyzed with ID22” are also associated with each other in the same manner.

In SJ4 of FIG. 17, the processing unit 342 acquires data of one processing unit. Specifically, it is arbitrary, but for example, similar to SI4 in FIG. Get the target data. Here, for example, analysis target data corresponding to "data ID" = "ID11" and "ID12" associated in SJ3 are acquired.

In SJ5 of FIG. 17, the processing unit 342 executes evaluation processing. The evaluation process executed here is the same as the evaluation process executed at SB4 in FIG. 5, so each step of the evaluation process in FIG.

At SK1 in FIG. 6, the processing unit 342 selects an evaluation model. Specifically, although it is arbitrary, for example, the same processing as SC1 is performed. Specifically, the data ID corresponding to the data to be analyzed acquired in SJ4 of FIG. 17 is specified, the third model specifying information of FIG. 15 is referred to acquire the model information corresponding to the specified data ID, Select the evaluation model specified by the acquired model information. Here, for example, "ID11" and "ID12", which are data IDs corresponding to the analysis target data acquired in SJ4 in FIG. 17, are specified, and the specified data Acquire the model information "MM11-1" to "MM11-10" corresponding to the IDs "ID11" and "ID12", and select the evaluation model specified by "MM11-1" to "MM11-10". select.

At SK2 in FIG. 6, the processing unit 342 evaluates the analysis target data. Specifically, although it is arbitrary, the same processing as SC2 is performed. For example, as described above, the "evaluation model" is input with a plurality of associated analysis target data, and in the input plurality of associated analysis target data, the payload information of the analysis target data is Focusing on the fact that it is a model that detects whether or not it corresponds to a so-called outlier, which is a value that deviates relatively greatly from the data, the following processing is performed. Specifically, the analysis target data acquired in SJ4 in FIG. 17 is input to all the evaluation models selected in SK1 in FIG. 6, the information output from each evaluation model is acquired, and based on the acquired information Then, an evaluation score is calculated and generated for each piece of analysis target data. Note that the payload information of the target data to be input to the evaluation model is combined into one by the same method as for SI5 in FIG. 16, and the combined payload information is input.

Here, for example, the analysis target data corresponding to "data ID"="ID11" and "ID21" in FIG. 3, which are the analysis target data acquired in SJ4 in FIG. Input to all evaluation models specified by "MM11-1" to "MM11-10" in FIG. 15, acquire information output from each evaluation model, and based on the acquired information, each analysis An evaluation score is calculated and generated for the target data.

At SK3 in FIG. 6, the processing unit 342 stores the evaluation result of the analysis target data. Specifically, although it is arbitrary, for example, the same processing as SC3 is performed. For example, an evaluation score is acquired for each piece of analysis target data, and the acquired evaluation score is recorded in the recording unit 13 in a state of being associated with the analysis target data corresponding to the evaluation score. Here, for example, "row information" = "1" in FIG. Information and the like associated with the analysis target data of "1" and "2" are stored. It should be noted that the payload information in this case may be stored in the above-described integrated form, or may be stored in a separated form. Then, the evaluation process is returned.

At SJ6 in FIG. 17, the processing unit 342 determines whether or not to end the processing. Specifically, although it is arbitrary, for example, the same processing as SB5 in FIG. 5 is performed. Then, when all the analysis target data associated in SJ3 of FIG. 17 are acquired and evaluated, the process proceeds to SJ7.

At SJ7 in FIG. 17, the processing unit 342 outputs the evaluation result. Specifically, although it is arbitrary, for example, the same processing as SB6 in FIG. 5 is performed. Specifically, here, for example, when the user inputs "80%" as the display threshold score via the input unit 31, the display threshold score among the evaluation scores that are the evaluation results recorded in SK3 in FIG. Select and acquire the above evaluation score "90%", etc., and set the acquired evaluation score "90%" to the analysis target data corresponding to the evaluation score "row information" = "1" and "2" are displayed on the display of the output unit 12 and output.

(Effect of this embodiment)
According to the present embodiment, a plurality of data to be analyzed (evaluation-related data) or a plurality of learning data (evaluation-related data) are mutually exchanged based on a predetermined standard (a communication specification and the above-described predetermined sequence). and perform model generation processing (evaluation-related processing) and analysis processing (evaluation-related processing) based on the associated multiple evaluation-related data. When a criterion that matches the actual situation is adopted, it becomes possible to appropriately perform the evaluation-related processing in line with the actual situation.

[III] Modifications to the Embodiments Although the embodiments according to the present invention have been described above, specific configurations and means of the present invention are within the scope of the technical ideas of each invention described in the claims. In can be optionally modified and improved. Such modifications will be described below.

(Problem to be solved and effect of invention)
First, the problems to be solved by the invention and the effects of the invention are not limited to the contents described above, and may differ depending on the details of the implementation environment and configuration of the invention. or only part of the effects described above.

(Regarding decentralization and integration)
Also, each electrical component described above is functionally conceptual, and does not necessarily need to be physically configured as illustrated. In other words, the specific forms of distribution and integration of each part are not limited to those illustrated, and all or part of them can be functionally or physically distributed or integrated in arbitrary units according to various loads and usage conditions. can be configured as

(Regarding shapes, numbers, structures, and time series)
Regarding the components illustrated in the embodiments and drawings, the shape, numerical value, structure of multiple components, or interrelationships in chronological order may be arbitrarily modified and improved within the scope of the technical idea of the present invention. can be done.

(Regarding output of evaluation results)
Also, in SB6 of FIG. 5 of the first embodiment, a case has been described in which information in which the evaluation score is associated with the analysis target data corresponding to the evaluation score is output, but the present invention is not limited to this. For example, it may be configured to output any other information in addition to these pieces of information. For example, focusing on the payload information in the analysis target data corresponding to the output evaluation score (that is, the evaluation score equal to or higher than the display threshold score), referring to the analysis target data in FIG. may be configured to specify the number of pieces of analysis target data having and to output the specified number as well.

(About feedback)
In addition, the analysis target data whose evaluation score is "0%" in the analysis processing of each of the above embodiments is displayed, and the user, such as a test engineer, visually recognizes the displayed target data and selects "permit" or the like. The data to be analyzed may be recorded as learning data when a command is input.

(for a given sequence)
16 and SJ3 of FIG. 17 in the third embodiment, data IDs to be associated with each other are stored in the recording unit 33 of FIG. may be recorded by user input, and data may be associated in accordance with this reference information. As an example, reference information such as associating “data ID”=“ID11” and “ID21” and associating “data ID”=“ID11” and “ID22” is recorded. may be configured to associate data with each other. Note that the number of data IDs to be associated with each other here is not limited to two, and three data IDs may be used when two functions are expected to respond to one request based on the use of communication. The data IDs may be associated with each other, or may be arbitrarily changed in consideration of the actual situation of communication.

(About combination)
Also, the techniques of the embodiments and modifications may be combined arbitrarily. For example, one information processing system may be configured to execute all of the processes of Embodiments 1 to 3, or to selectively execute the processes according to instructions from a user.

(Appendix)
The processing system of Supplementary Note 1 is a processing system that performs evaluation-related processing that is processing related to evaluation of evaluation target data performed using a plurality of evaluation models used for evaluation of evaluation target data, It comprises acquisition means for acquiring data, and processing means for performing the evaluation-related processing based on the evaluation-related data acquired by the acquisition means.

The processing system according to Supplementary Note 2 is the processing system according to Supplementary Note 1, wherein the processing means performs the evaluation-related processing for each attribute of the evaluation-related data acquired by the acquisition means.

The processing system according to Supplementary Note 3 is the processing system according to Supplementary Note 1 or 2, wherein the processing means associates the plurality of evaluation-related data acquired by the acquisition means with each other based on a predetermined standard, and the associated The evaluation-related processing is performed based on the plurality of evaluation-related data.

The processing system according to appendix 4 is the processing system according to any one of appendices 1 to 3, wherein the processing means acquires the evaluation-related data acquired by the acquisition means based on a predetermined judgment model. is to be subjected to the evaluation-related processing, and the evaluation-related processing is performed based on the evaluation-related data determined to be subject to the evaluation-related processing.

The processing system according to appendix 5 is the processing system according to any one of appendices 1 to 4, wherein the acquisition means acquires teacher data for generating the plurality of evaluation models as the evaluation-related data. The processing means obtains the plurality of evaluation models by machine learning based on the training data obtained by the obtaining means and a predetermined machine learning algorithm for generating the plurality of evaluation models. The process of generating is performed as the evaluation-related process.

The processing system according to Supplementary Note 6 is the processing system according to Supplementary Note 5, wherein the predetermined algorithm has a plurality of parameters, and the processing means calculates the values of some of the plurality of parameters. By changing, the plurality of evaluation models are generated.

The processing system according to appendix 7 is the processing system according to any one of appendices 1 to 4, wherein the acquisition means acquires the evaluation target data as the evaluation-related data,
The processing means performs a process of evaluating the evaluation target data based on the evaluation target data acquired by the acquisition means and the plurality of evaluation models as the evaluation related process.

The processing system according to Supplementary Note 8 is the processing system according to Supplementary Note 7, wherein the processing means evaluates the evaluation target data acquired by the acquisition means using each of the plurality of evaluation models, and the plurality of evaluation models A process for generating evaluation result information indicating the results of each evaluation performed using each model is performed as the evaluation-related process.

The processing system of Supplementary Note 9 is the processing system according to Supplementary Note 7 or 8, wherein the processing means performs a process of evaluating a data value included in the evaluation target data, or a process included in the evaluation target data. The process of evaluating the time spent is performed as the evaluation-related process.

The processing program of appendix 10 is a processing program that performs evaluation-related processing, which is processing related to evaluation of the evaluation target data, using a plurality of evaluation models associated with the evaluation target data, the program comprising: Acquisition means for acquiring evaluation-related data and processing means for performing the evaluation-related processing based on the evaluation-related data acquired by the acquisition means are caused to function.

(Effect of Supplementary Note)
According to the processing system described in Supplementary Note 1 and the processing program described in Supplementary Note 10, by performing evaluation-related processing related to evaluation performed using a plurality of evaluation models, for example, a plurality of evaluation models Since it is possible to evaluate the evaluation target data that is used, it is possible to improve the accuracy of the evaluation of the evaluation target data.

According to the processing system described in appendix 2, by performing evaluation-related processing in units of attributes of evaluation-related data, for example, evaluation-related processing can be performed in units of evaluation-related data having similar attributes. Processing can be performed efficiently and appropriately.

According to the processing system described in Supplementary Note 3, a plurality of evaluation-related data are associated with each other based on a predetermined criterion, and evaluation-related processing is performed based on the associated plurality of evaluation-related data. If a standard such as a communication specification for evaluation-related data is adopted as the standard, it is possible to appropriately perform the evaluation-related processing in accordance with the actual situation.

According to the processing system described in appendix 4, it is determined whether evaluation-related data is to be subjected to evaluation-related processing based on a predetermined determination model, and is determined to be subject to evaluation-related processing. By performing evaluation-related processing based on the obtained evaluation-related data, for example, it is possible to perform evaluation-related processing using appropriate evaluation-related data.

According to the processing system described in appendix 5, based on teacher data and a predetermined algorithm for machine learning, by performing a process of generating a plurality of evaluation models by machine learning, for example, a plurality of evaluation models can be generated appropriately.

According to the processing system described in appendix 6, by changing the values of some of the parameters to generate a plurality of evaluation models, for example, the plurality of evaluation models can be made more appropriate. can be generated in

According to the processing system described in Supplementary Note 7, by performing a process of evaluating the evaluation target data based on the evaluation target data and the plurality of evaluation models, for example, the evaluation accuracy of the evaluation target data can be improved. It becomes possible.

According to the processing system described in appendix 8, the evaluation target data is evaluated using each of the plurality of evaluation models, and evaluation result information indicating the results of each evaluation performed using each of the plurality of evaluation models is generated. By performing the processing to do so, for example, it is possible to appropriately indicate the results of the evaluation of the evaluation target data performed using each of the plurality of evaluation models.

According to the processing system described in Supplementary Note 9, by performing the process of evaluating the data value included in the evaluation target data or the process of evaluating the time included in the evaluation target data, for example, Since it is possible to perform evaluation regarding data values and evaluation regarding time, it is possible to perform various evaluations on evaluation target data.

1 information processing device 2 information processing device 3 information processing device 11 input unit 12 output unit 13 recording unit 14 control unit 21 input unit 22 output unit 23 recording unit 24 control unit 31 input unit 32 output unit 33 recording unit 34 control unit 141 acquisition Unit 142 Processing unit 241 Acquisition unit 242 Processing unit 341 Acquisition unit 342 Processing unit

Claims (5)

A processing system that performs evaluation-related processing that is processing related to evaluation of evaluation target data performed using a plurality of evaluation models used for evaluation of evaluation target data,
Acquisition means for acquiring evaluation-related data;
a processing means for performing the evaluation-related processing based on the evaluation-related data acquired by the acquisition means;
The processing means
a first process of correlating the plurality of evaluation-related data acquired by the acquisition means based on a predetermined criterion;
a second process of performing the evaluation-related process based on the plurality of evaluation-related data associated in the first process;
The plurality of evaluation-related data includes a plurality of data of a first attribute used in communication conforming to a predetermined standard and a plurality of data of a second attribute used in communication conforming to the predetermined standard,
each of the plurality of data of the first attribute and each of the plurality of data of the second attribute are data communicated in a predetermined communication order;
The processing means
In the first process, each of the plurality of data of the first attribute and each of the plurality of data of the second attribute are associated with each other based on the communication order of the evaluation-related data;
In the second processing, performing the evaluation-related processing based on a plurality of combinations including each of the plurality of data of the first attribute and each of the plurality of data of the second attribute associated in the first processing,
The data included in each of the plurality of combinations is only data of two types of attributes including the first attribute and the second attribute.
processing system. The acquisition means acquires the evaluation target data as the evaluation related data,
The processing means performs, as the evaluation related process, a process of evaluating the evaluation target data based on the evaluation target data acquired by the acquisition means and the plurality of evaluation models.
2. The processing system of claim 1 . The processing means evaluates the evaluation target data acquired by the acquisition means using each of the plurality of evaluation models, and evaluates results of each evaluation performed using each of the plurality of evaluation models. performing a process of generating result information as the evaluation-related process;
3. The processing system of claim 2 . The processing means performs a process of evaluating a data value included in the evaluation target data or a process of evaluating a time included in the evaluation target data as the evaluation-related process.
4. A processing system according to claim 2 or 3 . A processing program for performing evaluation-related processing, which is processing related to evaluation of evaluation target data performed using a plurality of evaluation models used for evaluation of evaluation target data,
the computer,
Acquisition means for acquiring evaluation-related data;
functioning as processing means for performing the evaluation-related processing based on the evaluation-related data acquired by the acquisition means;
The processing means
a first process of correlating the plurality of evaluation-related data acquired by the acquisition means based on a predetermined criterion;
a second process of performing the evaluation-related process based on the plurality of evaluation-related data associated in the first process;
The plurality of evaluation-related data includes a plurality of data of a first attribute used in communication conforming to a predetermined standard and a plurality of data of a second attribute used in communication conforming to the predetermined standard,
each of the plurality of data of the first attribute and each of the plurality of data of the second attribute are data communicated in a predetermined communication order;
The processing means
In the first process, each of the plurality of data of the first attribute and each of the plurality of data of the second attribute are associated with each other based on the communication order of the evaluation-related data;
In the second processing, performing the evaluation-related processing based on a plurality of combinations including each of the plurality of data of the first attribute and each of the plurality of data of the second attribute associated in the first processing,
The data included in each of the plurality of combinations is only data of two types of attributes including the first attribute and the second attribute.
processing program.

Images