JP6976910B2 - Data classification system, data classification method, and data classification device - Google Patents

Description

The present invention relates to a data classification system, a data classification method, and a data classification device.

In order to efficiently manage data such as a large amount of documents, images, and numerical values, a search system that narrows down the data to each category and identifies the target data is useful.
In order to realize such a search system, it is necessary to classify a large amount of data as described above into predefined categories. On the other hand, one data may belong to multiple categories. Therefore, there is a technique for classifying data by assigning a plurality of labels representing categories to one data (multi-label classification problem).

However, even if such a technique is adopted, it is difficult to manually classify when the data to be classified is large. Therefore, there is also a technique for creating a multi-label classifier by machine learning using supervised learning and automatically classifying it.

A classifier refers to a general program that calculates and outputs the classification probability for the label to be assigned by inputting the feature amount of the data to be classified (it can also be created manually by IF statement etc. instead of machine learning). .. The above-mentioned multi-label classifier is a kind of such a classifier, and refers to a classifier that classifies one data with a plurality of labels.

As described above, in order to create a multi-label classifier by supervised learning, the features of the data to be classified and the learning data consisting of the true classification results (labels) according to the features are input to the learning device. Give as. It is difficult to correctly learn all combinations of various features and various labels with a single classifier, and a single classifier correctly classifies only limited features and limited labels. There are many things you can do.

Therefore, in order to create an accurate multi-label classifier, learning of an integrated classifier (ensemble learning) is generally performed by combining a plurality of classifiers. There are many ways to combine classifiers, but the method of taking the majority of the classification results of the classifiers and taking the average value is well known.

Here, due to the nature of ensemble learning, it is necessary to select only classifiers that are useful for improving accuracy when learning an integrated classifier. Therefore, for example, if a classifier that misclassifies is consistent with the true classification result, it is used as a part of the integrated classifier to improve the classification accuracy of the integrated classifier (Patent Document). 1) has been proposed.

However, such ensemble learning presupposes the existence of a large amount of training data including various combinations of features and labels. In reality, there is no large amount of training data, and in many cases only a small amount of biased training data can be used. Also, it is unrealistic to manually create a large amount of training data.

In the absence of training data, it is not possible to automatically select a classifier that is useful for improving the classification accuracy of the integrated classifier. This is a particularly serious problem in the multi-label classification problem in which the number of combinations of features and labels is enormous.

Therefore, as a method of learning a classifier from a small amount of learning data, there is a so-called semi-supervised learning concept in which learning data is increased by using unlabeled data (unlabeled data). As a conventional technique related to such a concept, for example, a user creates a classifier by using domain knowledge without using training data, and integrated classification is performed by combining a large number of simple classifiers that the user can easily create. A method of supplementing the number of training data by learning a vessel (ensemble learning) and using the classification result by an integrated classifier as a substitute for a true label has been proposed (see Non-Patent Document 1).

Japanese Unexamined Patent Publication No. 2015-11686

Alexander J Ratner, Christopher M De Sa, Sen Wu, Daniel Celsam, and Christopher R'e, "Data processing: Creation processing Engineering Processing systems, q 3567-3575, 2016.

When creating learning data for a multi-label classification problem by the method of the above-mentioned conventional technique (Non-Patent Document 1), a user manually creates a large number of classifiers according to combinations of various features and labels. Must.

Therefore, while the user visually confirms the classification result of the integrated classifier, a large number of classifiers are created and sorted by trial and error. As a result, there arises a problem that it becomes difficult to create an integrated classifier with high classification accuracy.
Therefore, an object of the present invention is to provide a technique for efficiently creating a classifier in an integrated classifier and improving the classification accuracy of the integrated classifier.

The data classification system of the present invention that solves the above problems integrates the classification results of a predetermined data set by each of a plurality of classifiers, and obtains training data that defines the correspondence between the feature amount of the data set and the label that is the classification result. The storage device to be stored and the training data are read from the storage device, and for each of the unlabeled data sets in the training data, the feature amount and the classification output by the integrated classifier learned from the classification result by each of the classifiers. Based on the probability, the simultaneous unclassified rate, which is the probability that multiple labels are unclassified for one data set, and the simultaneous classification probability, which is the probability that multiple labels are classified for one data set, are calculated. Processing, processing to calculate the recommendation score by aggregating the multiplication value of the simultaneous unclassified rate and the simultaneous classification probability for each label, and specifying labels in descending order of the recommended score as labels to be additionally created as a classifier. It is characterized by including an arithmetic device for executing a process of outputting the recommendation information of the label to a predetermined device.

Further, the data classification method of the present invention is a storage device that integrates the classification results of a predetermined data set by each of a plurality of classifiers and stores learning data that defines the correspondence between the feature amount of the data set and the label that is the classification result. The information processing system equipped with the above reads the training data from the storage device, and outputs the feature quantity and the integrated classifier learned from the classification result by each of the classifiers for each unlabeled data set in the training data. Based on the classification probability, the simultaneous unclassification rate, which is the probability that multiple labels are unclassified for one data set, and the simultaneous classification probability, which is the probability that multiple labels are classified for one data set, are calculated. The process of calculating, the process of totaling the multiplication value of the simultaneous unclassified rate and the simultaneous classification probability for each label to calculate the recommended score, and the process of additionally creating a classifier, labeled in descending order of the recommended score. Is specified, and the process of outputting the recommendation information of the label to a predetermined device is executed.

Further, the data classification device of the present invention is based on a communication device that performs communication processing with another device via a predetermined network, and a plurality of classifiers provided in the predetermined device by accessing the predetermined device by the communication device. The process of integrating the classification results of a predetermined data set and acquiring the training data that defines the correspondence between the feature amount of the data set and the label that is the classification result, and the characteristics of each of the unlabeled data sets in the training data. Based on the quantity and the classification probability output by the integrated classifier learned from the classification result by each of the classifiers, the simultaneous unclassification rate, which is the probability that multiple labels are unclassified for one data set, and 1. Processing to calculate the simultaneous classification probability, which is the probability that multiple labels are classified for one data set, and processing to calculate the recommendation score by aggregating the multiplication value of the simultaneous unclassified rate and the simultaneous classification probability for each label. , And, as a label to be additionally created, the classifier is provided with a calculation device for specifying labels in descending order of the recommendation score and outputting the recommendation information of the label to a predetermined device. ..

According to the present invention, the creation of a classifier in an integrated classifier is made efficient, and the classification accuracy of the integrated classifier is improved.

It is a figure which shows the example of the network configuration including the data classification system in this embodiment. It is a figure which shows the hardware configuration example of the classifier creation recommendation server of this embodiment. It is a figure which shows the structural example of the unlabeled data of this embodiment. It is a figure which shows the structural example of the classifier management information of this embodiment. It is a figure which shows the structural example of the classification result information of this embodiment. It is a figure which shows the structural example of the learning data information of this embodiment. It is a figure which shows the structural example of the unclassified rate information of this embodiment. It is a figure which shows the structural example of the simultaneous unclassification rate information of this embodiment. It is a figure which shows the structural example of the document group information of this embodiment. It is a figure which shows the structural example of the simultaneous classification probability information of this embodiment. It is a figure which shows the structural example of the recommendation score information of this embodiment. It is a figure which shows the structural example of the user information document group of this embodiment. It is a figure which shows the structural example of the user information classification result of this embodiment. It is a figure which shows the flow example of the learning data generation method in this embodiment. It is a figure which shows the flow example of the classifier creation recommendation method in this embodiment. It is a figure which shows the output example in this embodiment.

--- Network configuration ---
FIG. 1 is a network configuration diagram including the data classification system 100 of the present embodiment. The data classification system 100 shown in FIG. 1 is an information processing system that makes the creation of a classifier in the integrated classifier efficient and improves the classification accuracy of the integrated classifier.

As illustrated in the network configuration of FIG. 1, such a data classification system 100 acquires a classifier (eg, one generated by the operator of the user terminal 101) from the user terminal 101, and integrates classification based on this classifier. Create a vessel 303. The data classification system 100 has the effect of making the multi-label classification of the unlabeled data 201 by the above-mentioned integrated classifier 303 efficient. The purpose of classifying the unlabeled data 201 is to create training data 203 for creating a multi-label classifier.

As an example, the data classification system 100 shown in FIG. 1 can be assumed to be composed of a learning data generation server 102 and a classifier creation recommendation server 103. However, as the minimum configuration of the data classification system 100 of the present embodiment, it may be assumed that a function required for a single device such as only the classifier creation recommendation server 103 is implemented.

Further, as shown in FIG. 1, the data classification system 100 of the present embodiment communicates with an external device such as a user terminal 101, an unlabeled data management server 104, and a document group management server 105 via an appropriate network 406. It is possible, and it is assumed that the unlabeled data 201 and the document group information 206 can be appropriately acquired and read.

As an organization that operates the data classification system 100 of the present embodiment, for example, an entity that analyzes production efficiency in a certain factory and reduces the number of defective products can be assumed.

In the factory managed by this business entity, the explanation written by the data creator for each of the various IoT data generated from the sensors and machine tools installed in the factory (example: welding failure cause analysis data). Is stored in large quantities as unlabeled data 201. The unlabeled data 201 is a data set that has not yet been classified by the integrated classifier 303, that is, labeled.

Even a data analyst who does not know the existence of the above-mentioned IoT data can machine-learn a large amount of unlabeled data 201 for each category in order to discover the IoT data and perform analysis using the IoT data. It is preferable that the IoT data can be narrowed down by category by automatically classifying with.

However, the features of the unlabeled data 201 vary according to various IoT data, and labels expressing various categories are required to realize flexible narrowing down. Further, considering the possibility that one IoT data belongs to a plurality of categories, training data for creating a multi-label classifier is required.

Therefore, as already described, in the learning data generation server 102 of the data classification system 100, the classifier receiving unit 301 that receives the definition of the classifier from the user terminal 101 and the classifiers 302-1 to 302-N (N is arbitrary). The classifier execution unit 302 including the natural number) and the integrated classifier 303 operate, and the integrated classifier 303 is input to the classification result information 203 by the above-mentioned classifiers 302-1 to 302-N for the unlabeled data 201. After training, the training data 204 will be generated as the classification result of the unlabeled data 201 by the integrated classifier 303.

Specifically, the integrated classifier 303 and the training data 204 are added by the data classifier while visually confirming the creation result of the training data 204 according to the method disclosed in Non-Patent Document 1. It can be created by repeating the work of creating it.

The user terminal 101 is a terminal operated by a data classifier who uses the data classification system 100. Such a data classifier browses the display of the user terminal 101, operates the user terminal 101, newly defines a classifier for the unlabeled data 201, and transmits the classifier to the learning data generation server 102.

On the other hand, in order to make the additional creation of the classifier by the above-mentioned data classifier efficient, in the present embodiment, the recommendation information of the label to which the classifier creation recommendation server 103 should additionally create the classifier is concerned. Notify the user terminal 101 of the data classifier.

The classifier creation recommendation server 103 of the present embodiment calculates the unclassified rate information 205 by inputting the learning data 204 and the unclassified rate information 206 by inputting the unclassified rate information 205. Simultaneous unclassified rate calculation unit 305, simultaneous classification probability calculation unit 306 that reads document group 207 from the document group management server 105 and calculates simultaneous classification probability information 208, and simultaneous unclassified rate information 206 and simultaneous classification probability information 208. The recommendation score calculation unit 307 that calculates the recommendation score information 209 by inputting the above, the classifier generation unit 308 that generates a classifier from the simultaneous classification probability 208 and sends it to the learning data generation server 102, and the above-mentioned recommendation score information 209. Is composed of a recommendation execution unit 309 that notifies the user terminal 101 according to the user information classification result 211.

Of the above, the unclassified rate calculation unit 304, upon receiving the training data 204, acquires the label classification probability for the data set (hereinafter referred to as data) included in the training data 204, and the label classification probability is within a predetermined threshold. If it fits in, the label is considered unclassified for the data.

Further, the unclassified rate calculation unit 304 calculates the number of unclassified data of all labels and divides the unclassified number for each label by the number of classified target data (the number of data included in the training data) to obtain the unclassified rate. Is calculated and stored in the unclassified rate information 205.

If the unclassified rate calculated by the unclassified rate calculation unit 304 is below a predetermined threshold specified in advance by the data classifier for all labels, the integrated classifier 303 can classify a sufficient number of unlabeled data 201. , No need to add a classifier. Therefore, at this point, the classifier creation recommendation server 103 may end the process.

On the other hand, when the above-mentioned unclassified rate is lower than a predetermined threshold value for any of the labels, the unclassified rate calculation unit 304 transmits the unclassified information 205 to the simultaneous unclassified rate calculation unit 305.

On the other hand, when the simultaneous unclassified rate calculation unit 305 receives the above-mentioned unclassified rate information 205, in the unclassified rate information 205, for example, the i-th label (synonymous with label i, the same applies hereinafter) and the j-th label (same as the label i). _{It is preferable to calculate the simultaneous unclassified rate u ij} as the ratio of data that is synonymous with label j and the same applies hereinafter, and store this simultaneous unclassified rate u _ij in the simultaneous unclassified rate information 206. ..

Further, the simultaneous classification probability calculation unit 306 reads the document group 207 from the document group management server 105, calculates the co-occurrence probability of the words included in the i-th label and the j-th label, and calculates the calculation result as the simultaneous classification probability. It is preferable to store it in the simultaneous classification probability information 208 as _pij. Specifically, this "word co-occurrence probability" is a known method (eg, Yutaro Fujii, Takuya Yoshimura, Takayuki Ito, Tetsushi Ando, "Automatic classification method for harmful documents using co-occurrence information between multiple words". Proposal ", 10th Information Science and Technology Forum, (FIT2011) Lecture Papers (2011)).

_{Further, in the present embodiment, a method of calculating the simultaneous classification probability pij} is described on the assumption that the unlabeled data 201 does not include any label, but the unlabeled data 201 is actually manually calculated. May contain a small amount of label classification results. In such a case, the simultaneous classification probability may be calculated as the probability that the same data is classified into two labels at the same time from the label classification result or in combination with the label classification result.

The document group 207 held by the document group management server 105 may include in-organizational documents such as the above-mentioned data analysis report of the business entity, documents published on the Internet, and documents of unlabeled data 201. ..

Further, the classifier generation unit 308 automatically creates a classifier that classifies to the j-th label with the probability of the _{simultaneous classification probability pij} indicated by the simultaneous classification probability information 208 when the classifier is classified into the i-th label by another classifier. Then, this classifier is transmitted to the classifier receiving unit 301 of the learning data generation server 102. Specifically, this classifier can be automatically created by an IF statement.

Further, the recommendation score calculation unit 307 _{reads the simultaneous unclassified rate u ij from the simultaneous unclassified rate information 206, reads the simultaneous unclassified probability p ij} from the simultaneous classification probability information 208, and obtains the recommended score for the i-th label Σ _j. It is preferable to calculate as u _ij p _ij and store this recommendation score in the recommendation score information 209.

Further, the recommendation execution unit 309 notifies the user terminal 101 of the recommendation score information 209 and displays it, and recommends the data classifier to additionally create a classifier for classifying the label having a large recommended score value of the recommendation score information 209. do. On the other hand, the data classifier additionally creates a label having a large recommendation score from a classifier that correctly classifies the label. The user terminal 101 distributes the classifier additionally created by the above-mentioned data classifier to the classifier receiving unit 301 of the learning data generation server 102. By adding an appropriate classifier in this way, it becomes possible to classify a larger number of data with a small number of classifiers as the integrated classifier 303.

Further, when the above-mentioned unclassified rate value has already been calculated, the unclassified rate calculation unit 304 in the present embodiment calculates the amount of change in the unclassified rate value due to the additional creation of the classifier for each label. If the amount of change is equal to or less than a predetermined threshold, it is determined that the addition of the classifier to the label is not effective. Further, the unclassified rate calculation unit 304 sends the label name to the recommendation score calculation unit 307 based on this determination, and multiplies a predetermined coefficient (0.8, etc.) at the time of calculating the recommendation score to obtain the label. The recommendation score may be lowered.

It should be noted that there are a plurality of data classifiers, and the user information document group 210 (held by the unlabeled data management server 104) including the description of the data classifier (affiliation, skill, etc.) is used as a part of the unlabeled data 201. It may be possible. In this case, it is possible to classify the explanatory text of such a data classifier by the integrated classifier 303 in the same manner as the other unlabeled data 201, and obtain the user information classification result 211.

Further, when the user information classification result 211 includes a data classifier classified into the same label as the label to be recommended, the recommendation execution unit 309 addresses the label to the user terminal 101 of the data classifier. By notifying the information requesting the addition of the classifier regarding the label, that is, by outputting the recommendation information, it is possible to request the classifier who has knowledge about the label to create the classifier.

When the unlabeled data 201 has various explanatory texts, in order to improve the processing efficiency, the data set including the unlabeled data 201 and the document group 207 is divided into K clusters, and each cluster is not classified. You may calculate the rate. Specifically, the K-means method or the like may be applied as a method for dividing the data set into clusters.

When the unclassified rate is calculated for each cluster as described above, the simultaneous unclassified rate calculation unit 305 calculates the simultaneous unclassified rate described later for each cluster, and calculates the simultaneous unclassified rate _uijk for the kth cluster. It is saved in the simultaneous unclassified rate information 206. Further, the simultaneous classification probability calculation unit 306 calculates the simultaneous classification probability for each cluster, and the simultaneous classification probability _{pij for the kth cluster.
k} is calculated and stored in the simultaneous classification probability information 208. Thereafter, the recommendation score calculation unit 307 calculates as Σ _k Σ _j u _ijk p _ijk in the form of the sum of recommendation score for all clusters.

--- Hardware configuration ---
Further, FIG. 2 shows the hardware configuration of the classifier creation recommendation server 103 that mainly constitutes the data classification system 100 of the present embodiment.

The classifier creation recommendation server 103 of the present embodiment includes a storage device 401 composed of an appropriate non-volatile storage element such as an SSD (Solid State Drive) or a hard disk drive, a memory 404 composed of a volatile storage element such as a RAM, and storage. The program 402 held in the device 401 is executed by reading it into the memory 404 to perform overall control of the device itself, and is connected to a computing device 403 such as a CPU that performs various determinations, calculations, and control processes, and a network 406. A communication device 405 that handles communication processing with other devices (learning data generation server 102, user terminal 101, document group management server 105, etc.) is provided.

In the storage device 401, in addition to the program 402 for implementing the function required as the classifier creation recommendation server constituting the data classification management system 100 of the present embodiment, the unclassified rate information 205 and the simultaneous unclassified are not classified. The rate information 206, the simultaneous classification probability information 208, the recommendation score information 209, and the user information classification result 211 are stored. Details of this information will be described later.

Further, when the above-mentioned arithmetic unit 403 executes the program 402, the unclassified rate calculation unit 304, the simultaneous unclassified rate calculation unit 305, the simultaneous classification probability calculation unit 306, the recommended score calculation unit 307, the classifier generation unit 308, And, the recommendation execution unit 309 is implemented. The details of the functions of these functional units will also be described later.

--- Data structure example ---
Subsequently, the data used by the above-mentioned classifier creation recommendation server 103 and the learning data generation server 102, which constitute the data classification system 100 of the present embodiment, will be described.

FIG. 3 is a diagram showing a configuration example of unlabeled data 201 in the present embodiment. The unlabeled data 201 is a collection of records associated with the value of the feature amount 201b of the unlabeled data using the numerical value for uniquely identifying the unlabeled data or the data ID 201a which is a character string as a key. There is.

Of these, the feature amount 201b is a value indicating the characteristics of the data to be classified or the data created from the data to be classified, and has an arbitrary format such as a character string or a numerical value.

Subsequently, FIG. 4 shows a configuration example of the classifier management information 202 in the present embodiment. The classifier management information 202 of the present embodiment is a classifier indicating a method for executing classification using the classifier using the classifier ID202a, which is a numerical value or a character string for uniquely identifying the classifier, as a key. It is a collection of records in which the execution method 202b and the classification target label 202c indicating the labels of the targets classified by the classifier execution method 202b are associated with each other.

Of these, the classification target label 202c may take a plurality of values depending on the properties of the classifier.
The records of the above-mentioned classifier management information 202 are accumulated by acquiring the values created by the classifier from the user terminal 101 through the classifier receiving unit 301 or by automatically generating them by the classifier creating unit 308.

Subsequently, FIG. 5 shows a configuration example of the classification result information 203 of the present embodiment. The classification result information 203 of the present embodiment is characterized by the unlabeled data 201 using the same data ID 201a as the data ID in the unlabeled data 201 (that is, the classification result regarding the same unlabeled data 201) as a key. It is a collection of records in which the values of the classification probabilities 203-1 to 203-N calculated by each of the quantity 201b and N (N is a natural number) classifier are associated with each other.

Of these, the classification probabilities 203-1 to 203-N are values that indicate the probability that each record of the unlabeled data 201 is classified in the classification target label 202c (held by the classifier management information 202) calculated by each classifier. Is. This value may have a plurality of probability values depending on the number of values of the classification target label 202c.

Subsequently, FIG. 6 shows a configuration example of the learning data 204 of the present embodiment. The learning data 204 of the present embodiment associates the feature amount 201b of the unlabeled data 201 and the classification probability 204c of the unlabeled data 201 for each label calculated by the integrated classifier 203 with the data ID 201a as a key. It is a collection of records. Of these, the classification probability 204c is a vector having the classification probability for each label as an element.

Subsequently, FIG. 7 shows a configuration example of the unclassified rate information 205 of the present embodiment. The unclassified rate information 205 of the present embodiment is a collection of records in which the values of the number of classified data 205b, the number of unclassified data 205c, and the unclassified rate 205d are associated with each value using the label name 205a as a key. ing.

Of these, the number of data to be classified 205b is a numerical value indicating the number of data to be classified into the label of the label name 205a among all the data included in the learning data 204.

Further, the number of unclassified data 205c is a numerical value indicating the number of (unclassified) data for which it has not been determined whether or not to classify the data into the label of the label name 205a among the number of data to be classified 205b.

Further, the unclassified data number 205c and the unclassified rate 205d are numerical values obtained by dividing the unclassified data number 205c calculated by the unclassified rate calculation unit 304 by the classification target data number 205b, and the ratio of the unclassified data is calculated. It is the numerical value shown.

Subsequently, FIG. 8 shows a configuration example of the simultaneous unclassified rate information 206 of the present embodiment. The simultaneous unclassified rate information 206 of the present embodiment is a collection of records in which the values of the label i206a, the label j206b, and the simultaneous unclassified rate 206c are associated with each other.

Of these, the label i206a and the label j206b are both character strings indicating the label names, and the simultaneous unclassified rate information 206 has records for all combinations of these two labels.

Further, the simultaneous unclassified rate 206c is a numerical value indicating the ratio of data in which both labels are unclassified for each label of the label i206a and the label j206b, and is calculated by the simultaneous unclassified rate calculation unit 305.

In the present embodiment, the simultaneous unclassification rate 206c is calculated for the combination of all two labels (label i and label j), but all of the three or more labels, for example, label i, label j, and label k. The simultaneous unclassified rate 206c may be calculated by taking a combination.

Subsequently, FIG. 9 shows a configuration example of the document group 207 of the present embodiment. The document group 207 of the present embodiment is a collection of records including the values of the document ID 207a and the document content 207b.
Of these, the document ID 207a is a numerical value or a character string for uniquely identifying the document. Further, the document content 207b is a character string indicating the content of the document.

As already mentioned, this group of documents 207 and its document content 207b includes in-house documents such as the data analysis report of the above-mentioned entity, documents published on the Internet, and documents of unlabeled data. good.

Subsequently, FIG. 10 shows a configuration example of the simultaneous classification probability 208 of the present embodiment. The simultaneous classification probability information 208 of the present embodiment is a collection of records in which the values of the simultaneous classification probability 208c are associated with the labels i206a and j206b as keys.

Of these, the simultaneous classification probability 208c is a numerical value indicating the probability that each label of the label i206a and the label j206b is simultaneously classified into the same data, and is a value calculated by the simultaneous classification probability calculation unit 306.

Subsequently, FIG. 11 shows a configuration example of the recommendation score information 209 of the present embodiment. The recommendation score information 209 of the present embodiment is a collection of records in which the values of the recommendation score 209b and the recommendation order 209c are associated with each other using the label name 209a as a key.
The recommendation score 209b is a value of the recommendation score calculated by the recommendation score calculation unit 307 for the label described in the label name 209a.

Further, the recommendation rank 209c is a rank for recommending the creation of a classifier determined in descending order of the value of the recommendation score 209b, and the creation of the classifier is recommended to the classifier in order from the label having the smallest value of the recommendation rank 209c through the user terminal 101. Will be done.

Subsequently, FIG. 12 shows a configuration example of the user information document group 210 of the present embodiment. The user information document group 210 of the present embodiment is a collection of records in which the values of the user information document 210b and the user contact 210c are associated with each other using the user ID 210a as a key.
Of these, the user ID 210a is a numerical value or a character string for uniquely identifying the user information document 210b.
Further, the user information document 210b is a document expressing the user's business experience and skill, and is an amorphous document made of natural language or formatted character string data.

Further, the user contact 210c is a character string or a numerical value representing the contact information of the user described in the user information document 210b, and is specifically composed of an e-mail address, a telephone number, or the like of the user.

Subsequently, FIG. 13 shows a configuration example of the user information classification result 211 of the present embodiment. The user information classification result 211 of the present embodiment is a collection of records in which the values of the user contact 210c and the label 211c are associated with each other using the user ID 210a as a key.
Of these, the label 211c is a label output from the integrated classifier 303 as a result of classification by the integrated classifier 303 with the user information document group 210 as an input.

--- Flow example 1 ---
Hereinafter, the actual procedure of the data classification method in the present embodiment will be described with reference to the drawings. Various operations corresponding to the data classification method described below are realized by a program read into a memory or the like by the learning data generation server 102 and the classifier creation recommendation server 103 constituting the data classification system 100, respectively. The program is composed of codes for performing various operations described below.
FIG. 14 is a diagram showing a flow example 1 of the learning data generation method in the present embodiment, and specifically, is a flowchart showing the operation of the learning data generation server 102.
In this flow, the learning data generation server 102 starts processing in response to a classifier addition request from the user terminal 101 or the classifier creation recommendation server 103.

In this case, the classifier receiving unit 301 of the learning data generation server 102 receives the information of the classifier to be added from the user terminal 101 and the classifier creation recommendation server 103, generates a record including the information of the classifier, and generates a record. The record is stored in the classifier management information 202 (S101).

As already described with reference to FIG. 4, the information of the classifier in the above-mentioned record includes the values of the classification execution method 202b and the classification target label 202c. Further, the value of the classifier ID202a is given by incrementing the value each time a record is added.

Next, the classifier execution unit 302 of the learning data generation server 102 reads the value of the classifier execution method 202b from the record newly saved in the classifier management information 202 in the above-mentioned S101, and follows the method described in the value. The classifier is executed to classify the unlabeled data 201, and the result of the classification is stored in the classification result information 203 (S102).

In the case of the image of the classification result information 203 exemplified in FIG. 5, the classifier to be added to the column of each classifier of 203-1 to 203-n (the one newly added to the classifier management information 202 in S101). Item is added, and the value of the classification result by the classifier (eg, "0.80" for the data ID "1") is added for each record of the unlabeled data 201.

Further, the learning data generation server 102 learns the integrated classifier 303 from the above-mentioned classification result information 203 (S103). In this learning method itself, the teacher data when the feature amount 201b in the classification result information 203 is input is performed as each value of the classification results 203-1 to 203-n, and the existing one may be appropriately adopted.

Subsequently, the learning data generation server 102 classifies the unlabeled data 201 by the integrated classifier 303 learned in the above-mentioned S103, saves the result of the classification as the learning data 204 (S104), and ends the process (S104). ).

In addition, the learning data generation server 102 in this step S104 uses the integrated classifier 303 to describe the data classifier (affiliation, skill, etc.) included in the unlabeled data 201, in the same manner as the other unlabeled data 201. It is assumed that the user information classification result 211 is obtained and stored.

--- Flow example 2 ---
FIG. 15 is a diagram showing a flow example of the classifier creation recommendation method in the present embodiment, and specifically, is a flowchart showing the operation of the classifier additional recommendation server 103.

Subsequently, a flow executed by the classifier creation recommendation server 103 when the learning data 204 is updated by the above-mentioned learning data generation server 102 will be described.

In this case, the unclassified rate calculation unit 304 of the classifier creation recommendation server 102 acquires the learning data 204 from the learning data generation server 102, calculates the unclassified rate for each label with respect to the learning data 204, and calculates the unclassified rate for this calculation. The result is stored in the unclassified rate information 205 (S201).

In this case, the unclassified rate calculation unit 304 calculates the number of unclassified data, that is, the number of unclassified data for each label, and divides the unclassified number for each label by the number of data to be classified (the number of data included in the training data). By doing so, the unclassified rate is calculated.

For example, in the number of unclassified data, that is, the number of unclassified data of the "crack" label, the value of "crack" is a predetermined reference value (eg, 0.6) among the vector values of the classification probability 204c for each record of the training data 204. The following) is specified, and the number of corresponding records, that is, the number of data for which the "crack" label is unclassified is counted as "30" or the like.

The unclassified rate of the "crack" label is not determined by dividing the above-mentioned unclassified number "30" by the number of classified data (total number of data included in the training data 204, for example, "100"). Calculate the classification rate as "0.3" or the like.

Subsequently, the learning data generation server 102 determines whether the value of the unclassified rate calculated in S201 exceeds a predetermined threshold value (eg 0.2) specified in advance by the data classifier for any label (S202). ).

As a result of the above determination, when the value of the unclassified rate is not equal to or higher than the above threshold value (S202: n), that is, the value of the unclassified rate calculated in S201 sets a predetermined threshold value predetermined by the data classifier. If it is below all the labels, the integrated classifier 303 has been able to classify a sufficient number of unlabeled data 201, and it is specified that there is no need to add a classifier, and the subsequent processing is terminated.

On the other hand, as a result of the above determination, when the value of the unclassified rate is equal to or higher than the above-mentioned threshold value (S202: y), that is, when the above-mentioned unclassified rate is lower than the predetermined threshold value for any of the labels, the unclassified rate is unclassified. The rate calculation unit 304 transmits the unclassified information 205 to the simultaneous unclassified rate calculation unit 305, _{and causes the simultaneous unclassified rate calculation unit 305 to calculate the simultaneous unclassified rate uij} (S203).

Upon receiving the above-mentioned unclassified rate information 205, the simultaneous unclassified rate calculation unit 305 receives, for example, the i-th label (synonymous with label i, the same applies hereinafter) and the j-th label (label j) in the unclassified rate information 205. both synonymous, hereinafter the same) and is, simultaneously unclassified rate u _ij calculated as a percentage of the data to be unclassified shall store this simultaneous unclassified rate u _ij simultaneously unclassified rate information 206.

For example, the data in which both the "crack" label and the "defective product" label are unclassified can be used as the "crack" and "defective product" values among the vector values of the classification probability 204c of each record of the training data 204. Is specified as a predetermined standard value (example: 0.6 or less), and the number of applicable records, that is, the number of data in which both the "crack" and "defective" labels are unclassified is "28", etc. Count.

Further, both the "cracked" and "defective" labels become unclassified at the same time, and the simultaneous unclassified rate is the above-mentioned unclassified number "28" and the number of classified data (all data included in the training data 204). Number. For example, by dividing by "100"), it is calculated as "0.28".

Further, the simultaneous classification probability calculation unit 306 of the classifier creation recommendation server 103 receives the result of the above determination (S202: y), reads the document group 207 from the document group management server 105, and reads the i-th label (described above). In the case of the example, the co-occurrence probability of the words ("crack" and "defective product") contained in the jth label ("defective product" in the case of the above example) is calculated (S204). The calculation result is stored in the simultaneous classification probability information 208 as the simultaneous classification probability _pij.
It is assumed that each of the above-mentioned processes S203 and S204 is executed asynchronously.

Then, the recommendation score calculation unit 307 of the classifier creation recommendation server 103, reads the simultaneous unclassified rate u _ij from simultaneous unclassified rate information 206, reads the simultaneous unclassified probability p _ij from simultaneous classification probability information 208, all of the It calculates a recommendation score for label i as _{Σ j u ij p ij (S205} ), and stores the recommendation score to recommendation score information 209.

For example, regarding the combination of the label i being "cracked" and the label j being "defective product", the value "0.30" of the simultaneous unclassified rate 206c shown by the record of the simultaneous unclassified rate information 206 and the simultaneous classification probability information 208 are A calculation to obtain "0.24" by multiplying the value "0.8" of the simultaneous classification probability 208c shown is executed for all combinations whose label i is "cracked", and the multiplication value which is the execution result is aggregated. , Calculate the recommendation score.

Subsequently, the classifier generation unit 308 of the classifier creation recommendation server 103 automatically creates a classifier that classifies the label i into the jth label with a probability of the _{simultaneous classification probability pij, as shown by the above-mentioned simultaneous classification probability information 208.} (S206), this classifier is transmitted to the classifier receiving unit 301 of the learning data generation server 102.

Further, if a user classified into the same label as the label to be recommended described in the recommendation score information 209 is included in the user information classification result 211, the recommendation execution unit 309 addresses the user's contact information. The recommendation information (screen 1000 in FIG. 16) recommending the additional creation of the classifier is transmitted (S207), and the process is terminated.
It is assumed that the above-mentioned S206 and S207 are executed asynchronously.

The above-mentioned data classifier browses the above-mentioned recommendation information on the user terminal 101, and performs work on a classifier that correctly classifies the label (label with a large score) indicated by the recommendation information. The user terminal 101 distributes the classifier additionally created by the above-mentioned data classifier to the classifier receiving unit 301 of the learning data generation server 102. By adding an appropriate classifier in this way, it becomes possible to classify a larger number of data with a small number of classifiers as the integrated classifier 303.

Although the best mode for carrying out the present invention has been specifically described above, the present invention is not limited to this, and various modifications can be made without departing from the gist thereof.
According to such an embodiment, the creation of a classifier in the integrated classifier is made efficient, and the classification accuracy of the integrated classifier is improved.

The description herein reveals at least the following: That is, in the data classification system of the present embodiment, when the calculation device calculates the simultaneous unclassification rate, if the classification probability of the label included in the training data falls within a predetermined threshold value, the calculation device is used with respect to the data set. _{It is determined that the label is unclassified, the simultaneous unclassified rate uij} is calculated as the ratio of data in which both the predetermined label i and the label j are unclassified in the training data, and the simultaneous classification probability is calculated. _{At the same time, the simultaneous classification probability pij} is calculated as the co-occurrence probability of the words corresponding to the label names of the label i and the label j included in the predetermined document group, and the recommendation score regarding the label i is calculated. _Is calculated as Σ _j u _ij p ij.

According to this, the calculation of the simultaneous classification probability and the simultaneous classification probability is made efficient, and by extension, the creation of the classifier in the integrated classifier is made more efficient, and the classification accuracy of the integrated classifier is improved. Will be.

Further, in the data classification system of the present embodiment, the arithmetic unit acquires a classifier additionally created by a predetermined user based on the recommendation information from the input device, and adds the added classifier to the plurality of classifiers. To generate a classifier group, and further execute a process of recreating a new learning data in which the classification results of each classifier in the classifier group are integrated and an integrated classifier learned by the training data. For each unlabeled data set, the labels included in the new training data are unclassified based on the feature quantity and the classification probability output by the integrated classifier trained by the new training data. The unclassified rate is calculated as the ratio of the data set to be, and the recommendation information is output, the integrated classifier and the training data are recreated, and the unclassified rate is calculated until the unclassified rate falls below a predetermined standard. May be repeated.

According to this, labeling for each data set is made efficient without omission, and by extension, the creation of the classifier in the integrated classifier is made more efficient, and the classification accuracy of the integrated classifier is improved. Become.

Further, in the data classification system of the present embodiment, in the arithmetic device, the value of the unclassified rate is the same or increased before and after the new learning data and the re-creation of the integrated classifier accompanying the addition of the classifier. In this case, the process of multiplying the recommendation score of the label targeted by the recommendation information by a predetermined coefficient that lowers the rank of the recommendation score between the labels may be further executed.

According to this, it becomes possible to appropriately eliminate the adverse effects of the addition of the classifier, which in turn makes the creation of the classifier in the integrated classifier more efficient and improves the classification accuracy of the integrated classifier. Will be made to.

Further, in the data classification system of the present embodiment, the arithmetic unit refers to the simultaneous classification probability of the predetermined label i specified as the target of the recommendation information, and the simultaneous classification probability with the label i is equal to or higher than the predetermined reference. The label j is specified, and when the label i is given, a classifier that classifies the label j with the probability of the simultaneous classification probability is automatically generated, and the automatically generated classifier is used as the plurality of classifiers. It may be said that the process to be added is further executed.

This makes it possible to automatically generate a classifier with few omissions based on keywords with a high co-occurrence probability, that is, the relationship between labels, which in turn makes it more efficient to create a classifier in an integrated classifier. , The classification accuracy of the integrated classifier will be improved.

Further, in the data classification system of the present embodiment, when the arithmetic unit calculates the simultaneous classification probability, in addition to the document group, a sentence included in the feature amount of the training data and a plurality of labels attached to the training data. It may be said that the simultaneous classification probability is calculated from the above.

According to this, even if there is no document group prepared in advance (eg, technical documents in the company, etc.), it is possible to calculate the simultaneous classification probability based on the existing learning data, and by extension, the integrated classification. The creation of the classifier in the vessel will be made more efficient, and the classification accuracy of the integrated classifier will be improved.

Further, in the data classification system of the present embodiment, the storage device describes a description indicating that the user is involved in a predetermined event and information of each contact for each user who can be the main body of the additional creation of the classifier. The user information document group is further stored, and the arithmetic unit collates the word corresponding to the label name of the specified label with the user information document group, identifies the user who includes the word in the description, and identifies the user. The recommendation information may be output to the contact information of the user.

According to this, it becomes possible to notify the above-mentioned recommendation information to a suitable user who should be encouraged to create a classifier, which in turn makes the creation of the classifier in the integrated classifier more efficient, and the integrated classifier. It will improve the classification accuracy of.

Further, in the data classification system of the present embodiment, the arithmetic unit clusters the feature amount of the data set included in the learning data and the feature amount of the document group, and when calculating the simultaneous unclassification rate, the data set of the data set is calculated. _{Of these, for those belonging to the predetermined cluster k, the simultaneous unclassification rate uijk} is calculated as the ratio at which both the predetermined label i and the label j are unclassified, and in the calculation of the simultaneous classification probability, among the document group. _{, The simultaneous classification probability pijk} is calculated as the co-occurrence probability of the words corresponding to the label names of the label i and the label j included in the document group belonging to the predetermined cluster k, and the recommendation score is calculated with respect to the label i. wherein in which the recommendation score calculated as _{Σ k Σ j u ijk p ijk} , may be.

According to this, it is possible to recommend a classifier according to the label to be added for each cluster such as the process type of the production line, which in turn makes it more efficient to create a classifier in the integrated classifier. The classification accuracy of the integrated classifier will be improved.

100 Data classification system 101 User terminal 102 Learning data generation server (predetermined device)
103 Classifier creation recommendation server (data classification device)
104 Unlabeled data management server 105 Document group management server 201 Unlabeled data 202 Classifier management information 203 Classification result 204 Learning data 205 Uncategorized rate information 206 Simultaneous unclassified rate information 207 Document group 208 Simultaneous classification probability information 209 Recommended score information 210 User information document group 211 User information classification result 301 Classifier receiving unit 302 Classification device Execution unit 303 Integrated classifier 304 Unclassified rate calculation unit 305 Simultaneous unclassified rate calculation unit 306 Simultaneous classification probability calculation unit 307 Recommended score calculation unit 308 Classification device Generation unit 309 Recommendation execution unit 401 Storage device 402 Program 403 Computing device 404 Memory 405 Communication device

Claims (10)

A storage device that integrates the classification results of a predetermined data set by each of a plurality of classifiers and stores training data that defines the correspondence between the feature amount of the data set and the label that is the classification result.
1 The process of calculating the simultaneous unclassification rate, which is the probability that multiple labels are unclassified for one data set, and the simultaneous classification probability, which is the probability that multiple labels are classified for one data set, said simultaneous unclassification. The process of calculating the recommendation score by totaling the multiplication value of the rate and the simultaneous classification probability for each label, and specifying the labels in descending order of the recommendation score as labels to be additionally created as a classifier, and recommending the label. An arithmetic unit that executes the process of outputting information to a predetermined device, and
A data classification system characterized by including. The arithmetic unit is
When the classification probability of the label included in the training data falls within a predetermined threshold in the calculation of the simultaneous unclassification rate, it is determined that the label is unclassified for the data set, and the label is determined in the training data. _{The simultaneous unclassified rate u ij} is calculated as the ratio of data in which both the label i and the label j are unclassified.
_{In calculating the simultaneous classification probability, the simultaneous classification probability pij} is calculated as the co-occurrence probability of the words corresponding to the label names of the labels i and j included in the predetermined document group.
In calculating the recommendation score, is to calculate the recommendation score for label i as Σ _j u _ij p _ij,
The data classification system according to claim 1. The arithmetic unit is
A classifier additionally created by a predetermined user based on the recommendation information is acquired from an input device, the added classifier is added to the plurality of classifiers to generate a classifier group, and each classification in the classifier group is generated. Further execution of the process of recreating the new learning data in which the classification results of the vessels are integrated and the integrated classifier learned by the learning data is further executed.
In the new training data, for each unlabeled data set, the label included in the new training data is based on the feature amount and the classification probability output by the integrated classifier trained by the new training data. The unclassified rate is calculated as the ratio of the unclassified data set, and the recommendation information is output, the integrated classifier and the training data are recreated, and the unclassified rate is calculated until the unclassified rate falls below a predetermined standard. It repeats the calculation,
The data classification system according to claim 1. The arithmetic unit is
If the value of the unclassified rate is the same or increases before and after the new training data associated with the addition of the classifier and the re-creation of the integrated classifier, the recommendation score of the label targeted for the recommendation information is , Further performs a process of multiplying a predetermined coefficient that lowers the rank of the recommendation score between the labels.
The data classification system according to claim 3, wherein the data classification system is characterized by the above. The arithmetic unit is
When the label i specified as the target of the recommendation information is referred to the simultaneous classification probability, the label j whose simultaneous classification probability with the label i is equal to or higher than the predetermined standard is specified, and the label i is given. A classifier that classifies to the label j with the probability of the simultaneous classification probability is automatically generated, and a process of adding the automatically generated classifier to the plurality of classifiers is further executed.
The data classification system according to claim 1. The arithmetic unit is
In calculating the simultaneous classification probability, the simultaneous classification probability is calculated from the sentence included in the feature amount of the learning data and a plurality of labels attached to the learning data in addition to the document group.
The data classification system according to claim 2, wherein the data classification system is characterized by the above. The storage device is
For each user who can be the main body of the additional creation of the classifier, a description indicating that the user is involved in a predetermined event and a user information document group describing each contact information are further stored.
The arithmetic unit is
The word corresponding to the label name of the specified label is collated with the user information document group, the user who includes the word in the description is specified, and the recommendation information is output to the contact of the user. Is a thing,
The data classification system according to claim 1. The arithmetic unit is
The feature amount of the data set included in the training data and the feature amount of the document group are clustered.
_{In the calculation of the simultaneous unclassified rate, the simultaneous unclassified rate u ijk} is calculated as the ratio at which both the predetermined label i and the label j are unclassified for the data set belonging to the predetermined cluster k.
_{In calculating the simultaneous classification probability, the simultaneous classification probability pijk} is set as the co-occurrence probability of the words corresponding to the label names of the label i and the label j included in the document group belonging to the predetermined cluster k among the document groups. Calculate and
In calculating the recommendation score, and calculates the recommendation score for label i as _{Σ k Σ j u ijk p ijk} ,
The data classification system according to claim 2, wherein the data classification system is characterized by the above. An information processing system equipped with a storage device that integrates the classification results of a predetermined data set by each of a plurality of classifiers and stores learning data that defines the correspondence between the feature amount of the data set and the label that is the classification result.
1 Processing to calculate the simultaneous unclassification rate, which is the probability that multiple labels are unclassified for one data set, and the simultaneous classification probability, which is the probability that multiple labels are classified for one data set.
The process of calculating the recommendation score by aggregating the multiplication value of the simultaneous unclassified rate and the simultaneous classification probability for each label.
A process of specifying labels in descending order of the recommendation score as labels to be additionally created as a classifier and outputting the recommendation information of the label to a predetermined device.
A data classification method characterized by performing. A communication device that performs communication processing with other devices via a predetermined network,
The predetermined device is accessed by the communication device, the classification results of the predetermined data set by each of the plurality of classifiers provided in the predetermined device are integrated, and the correspondence between the feature amount of the data set and the label as the classification result is determined. Based on the process of acquiring the specified training data, the feature amount for each unlabeled data set in the training data, and the classification probability output by the integrated classifier learned from the classification result by each of the classifiers. The process of calculating the simultaneous unclassified rate, which is the probability that multiple labels are unclassified for one data set, and the simultaneous classification probability, which is the probability that multiple labels are classified for one data set, said simultaneous unclassified. The process of calculating the recommendation score by totaling the multiplication value of the classification rate and the simultaneous classification probability for each label, and specifying the labels in descending order of the recommendation score as labels to be additionally created as a classifier, An arithmetic device that executes the process of outputting recommended information to a predetermined device, and
A data classification device characterized by comprising.

Images