JP7665391B2 - Information processing device and information processing method - Google Patents

Description

The present invention relates to an information processing device and information processing method for processing various types of data.

A diagnostic system is known that diagnoses diagnostic data obtained from a diagnostic target according to predetermined rules, and separately diagnoses the diagnostic data according to rules learned by machine learning using the diagnostic data, compares the respective diagnostic results to extract discrepancies, and uses the extracted discrepancies to update the predetermined rules (Patent Document 1).

International Publication No. 2017/081984

In the above conventional technology, if the diagnostic results do not match, the predetermined rules are changed to make the diagnostic results based on the predetermined rules match the diagnostic results based on the rules learned by machine learning. Therefore, in the above conventional technology, while the diagnostic results based on the predetermined rules are appropriate, if the diagnostic results based on the rules learned by machine learning are inappropriate, there is a risk that the predetermined rules that have produced appropriate diagnostic results will be changed to ones that produce only inappropriate diagnostic results. This poses the problem of a decrease in the data processing accuracy of the diagnostic system.

The problem that the present invention aims to solve is to provide an information processing device and an information processing method that can prevent a decrease in data processing accuracy by changing the rules for classifying input data.

The present invention solves the above problem by, when classifying input data, which is text data written in a natural language , using first rules generated from processing reference data, which is text data indicating classification criteria for classifying the input data according to importance, and second rules learned by machine learning using training data, which is text data including input data associated with an importance of the input data different from the first rules, calculating a similarity indicating the degree of overlap between the processing reference data and reference data for updating the first rules, which includes at least one of the input data and the training data , and which is the frequency of occurrence of words that overlap with words that appear in the processing reference data used when generating the first rules , updating the first rules using reference data whose similarity is equal to or greater than a predetermined value, classifying the input data using the updated first rules and the second rules , and further including in the updated first rules new rules generated using co-occurring words that are extracted from the reference data and appear with the words included in the first rules at a frequency equal to or greater than a predetermined frequency .

According to the present invention, it is possible to prevent a decrease in data processing accuracy by changing the rules for classifying input data.

1 is a block diagram showing an information processing system including an information processing device according to the present invention. 2 is an example of an image displayed on the display device of FIG. 1 . 2 is a flowchart showing an example of a processing procedure in the information processing system of FIG. 1 .

The following describes an embodiment of the present invention with reference to the drawings.

[Information Processing System]
Fig. 1 is a block diagram showing an information processing system 1 according to the present invention. The information processing system 1 shown in Fig. 1 can be used when acquiring product defect information in the market from, for example, text posted on websites or social networking services (SNS), postings on electronic bulletin boards on an intranet within a company, and product repair reports sent to manufacturers by retailers that sold the products, and automatically classifying the acquired information according to its importance.

The information processing system 1 is a system that includes at least one database and an electronic computer that processes input data. As shown in FIG. 1, the information processing system 1 includes a first database 11, a second database 12, a third database 13, a fourth database 14, a communication device 15, a display device 16, and an information processing device 17. The first database 11 stores input data that is input to the information processing device 17 and processed, the second database 12 stores learning data used for machine learning executed by the information processing device 17, the third database 13 stores processing criterion data that indicates a predetermined criterion for processing (e.g., classifying) the data input to the information processing device 17, and the fourth database 14 stores corpus data used in processing in the information processing device 17.

The input data stored in the first database 11 is not particularly limited as long as it is text data written in a natural language. Examples of text data include text posted to websites or SNS, postings on intranet bulletin boards, information entered into an in-house system, and product repair reports sent to the manufacturer by the dealer that sold the product. The input data in this embodiment particularly includes a quality report (technical report) sent from the dealer that sold the vehicle to the automobile company that manufactured the vehicle. The quality report includes at least information on defects in the market of the sold vehicle, such as the location where the defect occurred, the state of the defect, the cause of the defect and countermeasures, the maintenance state and repair history of the vehicle where the defect occurred, the date and time of the defect and the date and time of response, etc.

The data stored in the first database 11 also includes reference data for updating the rules generated by the classification rule generation unit 24, which will be described later, in the rule update unit 26. The reference data is data that can be used to update the first rules, which will be described later, and includes data previously input and classified by the input data classification unit 27 and the classification results, data classified by another information processing device having the same classification function as this embodiment, and classification standards of other countries that correspond to Japanese classifications for classifying vehicle defects according to importance.

The learning data stored in the second database 12 is not particularly limited as long as it is data that can be used for machine learning executed by the information processing device 17, and is, for example, text data including documents such as laws, regulations, and guidelines related to vehicles and safety that are published by the Ministry of Land, Infrastructure, Transport and Tourism, etc. The learning data also includes input data previously processed by the information processing device 17. In this case, the learning data includes data previously input and processed by the information processing device 17 and at least the processing results at that time, and these pieces of information are associated and stored in the second database 12. For example, when input data including vehicle defect information is classified according to the classification criteria described below or according to rules learned by machine learning in the information processing device 17, the input data, the classification results of the input data according to the importance of the defect, and the classification method are associated and stored in the second database 12 as learning data.

The processing criteria data stored in the third database 13 is data that specifies the criteria used when processing input data in the information processing device 17, and is, for example, data that indicates the classification criteria used when classifying data input to the information processing device 17. The processing criteria data is text data such as sentences written in natural language and formulas written in mathematical symbols, and examples of such data include documents related to laws and safety standards published by the Ministry of Land, Infrastructure, Transport and Tourism and other organizations, operating procedures for classifying defects in products such as vehicles according to their importance, product warnings, and product usage manuals.

The corpus data stored in the fourth database 14 is corpus data used for processing in the information processing device 17, particularly for learning some rules by machine learning. A corpus is a structured collection of materials that is the subject of input data analysis, and is a collection of written materials and recorded language materials. The type of corpus is not particularly limited, and examples include a raw corpus that is simply a collection of documents, a balanced corpus that is a balanced collection of various types of documents, a tagged corpus that is assigned parts of speech and syntactic structures, and a parallel corpus that matches text in a certain language with its translated text. For example, if the input data is Japanese, the Balanced Corpus of Contemporary Written Japanese (BCCWJ) or the Corpus of Spoken Japanese (CSJ) can be used, and if the input data is English, the Corpus of Contemporary American English (COCA) can be used.

The first database 11 is connected to the input device 11a by known means such as a wired or wireless LAN in a state in which data can be sent and received, and the user of the information processing system 1 inputs data to be processed by the information processing device 17 to the first database 11 using the input device 11a. The input device 11a is a device by which the user of the information processing system 1 inputs data to be processed by the information processing device 17 to the first database 11, and is, for example, a keyboard, microphone, or touch panel connected to a display. The input device 11a may be provided together with the first database 11, or may be provided in a location away from the first database 11 and/or the information processing device 17 depending on the convenience of the user of the information processing system 1.

As shown in FIG. 1, the first database 11 is connected to the communication device 15 by a known communication means such as a wired or wireless LAN, and the information processing device 17 acquires input data stored in the first database 11 via the communication device 15. The communication device 15 is a device for transmitting and receiving data between the information processing device 17 and an external device, and is not particularly limited as long as it is a communication device capable of communicating with other devices via a network such as the Internet. Note that while there is one communication device 15 shown in FIG. 1, the number of communication devices 15 is not particularly limited, and multiple communication devices 15 may be used. When multiple communication devices 15 are used, the connection relationship between each device included in the information processing system 1 and the communication device 15 is not particularly limited.

In the information processing system 1 of FIG. 1, like the first database 11, the second database 12, the third database 13, and the fourth database 14 are also connected to the communication device 15. The information processing device 17 acquires learning data from the second database 12, acquires processing reference data from the third database 13, and acquires corpus data from the fourth database 14 via the communication device 15. These databases do not need to be provided together with the information processing device 17, and at least one of the first database 11, the second database 12, the third database 13, and the fourth database 14 may be provided in a location away from the information processing device 17, such as the server 12a shown in FIG. 2.

Furthermore, the information processing device 17 is also connected to the display device 16, and the results of processing the input data performed by the information processing device 17 are displayed on the display device 16. The display device 16 is a device for notifying the user of the processing status of the data input to the information processing device 17, and examples thereof include a liquid crystal display and a projector. The display device 16 in FIG. 1 is connected to the information processing device 17 in a state in which data can be exchanged, but the connection method is not particularly limited, and known communication means such as wired or wireless LAN can be used. The display device 16 may be provided together with the information processing device 17, or may be connected to the communication device 15 and provided at a location away from the information processing device 17 according to the convenience of the user of the information processing system 1.

The information processing device 17 is a device for processing input data acquired from the first database 11 using learning data acquired from the second database 12, processing reference data acquired from the third database 13, and corpus data acquired from the fourth database 14, and for notifying the user of the processing results using the display device 16. The information processing device 17 realizes the processing of input data and the notification of the processing results to the user by the processor 18. The processor 18 includes a ROM (Read Only Memory) 182 in which a program is stored, a CPU (Central Processing Unit) 181 which is an operating circuit for functioning as the information processing device 17 by executing the program stored in the ROM 182, and a RAM (Random Access Memory) 183 which functions as an accessible storage device.

[Input data processing section]
The program used in the information processing device 17 of this embodiment includes an input data processing unit 2, which is a functional block for implementing the processing of input data and the notification of the processing result to the user by the information processing device 17. The input data processing unit 2 has a function of acquiring necessary data from each of the databases 11 to 14, classifying the input data according to a certain rule, and notifying the user of the classification result via the display device 16. As shown in Fig. 1, the input data processing unit 2 includes a data acquisition unit 21, a word occurrence extraction unit 22, a co-occurrence word extraction unit 23, a classification rule generation unit 24, a similarity calculation unit 25, a rule update unit 26, an input data classification unit 27, a display image generation unit 28, and a data output unit 29. Each unit is extracted and shown in Fig. 1 for convenience.

The information processing device 17 shown in FIG. 1 has all of the above functional blocks, but it is not necessary for a single information processing device 17 to have all of the functional blocks, and some of the above functional blocks may be provided in other devices included in the information processing system 1, or in another information processing device not shown. For example, in the information processing system 1 of FIG. 1, the display image generation unit 28 may be provided in the display device 16. In this case, the functions of the display image generation unit 28 will be executed using the CPU, ROM, and RAM of the display device 16.

In addition, it is not necessary to execute all the processes of each functional block on a single information processing device, and the functions of each functional block may be realized across multiple information processing devices connected in a state in which data can be exchanged. For example, in the information processing system 1 of FIG. 1, some of the processes executed by the occurrence word extraction unit 22 may be executed on the server 12a, and the remaining processes may be executed on the information processing device 17. In this case, some of the processes for realizing the functions of the occurrence word extraction unit 22 are performed using the CPU, ROM, and RAM of the server 12a. As another example, for the co-occurrence word extraction unit 23 and the classification rule generation unit 24, some of the processes executed by each functional block may be executed on the server 12a, and the remaining processes may be executed on the information processing device 17.

The functions performed by each functional block of the input data processing unit 2 are explained below.

The data acquisition unit 21 has a function of acquiring, as necessary, the input data stored in the first database 11, the learning data stored in the second database 12, the processing reference data stored in the third database 13, and the corpus data stored in the fourth database 14. Furthermore, in this embodiment, the data acquisition unit 21 acquires data stored in the first database 11 as reference data. The data acquired by the data acquisition unit 21 is used in other functional blocks described below.

The word extraction unit 22 has a function of extracting words that appear in the data acquired by the data acquisition unit 21. In particular, the word extraction unit 22 has a function of extracting words used in text data contained in the input data and reference data of the first database 11, the learning data of the second database 12, the processing reference data of the third database 13, and the corpus data of the fourth database 14. When the text data processed by the word extraction unit 22 is an Indo-European language in which words are written with spaces between them (word segmentation), such as English, German, or French, the word extraction unit 22 extracts the words separated by spaces as they are, and attaches tags such as parts of speech to each word.

The word occurrence extraction unit 22 performs stemming on the extracted words as necessary, and deletes stop words from the extracted words. Stemming is a process for regarding a certain word and its derivatives as all having the same feature. For example, for the English verb "run", if inflections such as "ran" and "running" appear in the text data, all of these inflections are considered to have the same feature as "run". A stop word is a word that appears in documents on any topic, and is considered to have no relation to the type of topic. Examples include basic words such as "the", "is", "have", "take", and "get". These processes can be performed, for example, by using a module such as "termextract" in the programming language "Python".

On the other hand, when the text data to be processed by the word extraction unit 22 is a language that does not separate words, such as Japanese, Chinese, or Thai, the word extraction unit 22 performs morphological analysis on the text data. A morpheme is the smallest linguistic unit that has meaning, and in this embodiment, morphological analysis refers to determining word divisions in the text data and determining the part of speech and the like for each divided word. For example, a sentence such as "The engine is making a strange sound and not working" can be divided into "The engine is making a strange sound and not working" by morphological analysis, and the reading, root form, part of speech, conjugation form, and the like of each divided word are determined. Stop words such as particles and auxiliary verbs may be deleted from each of these divided words as necessary. Such processing can be performed using a morphological analysis tool such as "MeCab", "JUMAN", or "ChaSen".

In place of morphological analysis, an n-gram technique may be used. An n-gram is a technique that breaks down a string into n-character units (n is a natural number) and analyzes the string based on the combination of adjacent characters.

The occurrence word extraction unit 22 also has a function of searching for compound words from the extracted words. The occurrence word extraction unit 22 detects compound words that combine two to five words into one, for example. For example, if the word "exhaust/gas" is extracted, the compound word "exhaust gas" is detected, and if the word "adaptive/cruise/control" is extracted, the compound word "adaptive cruise control" (ACC) is detected.

Then, the word extraction unit 22 stores the words extracted from the input data, the words extracted from the learning data, the words extracted from the processing reference data, and the words extracted from the corpus data individually in a storage unit such as the RAM 183, and outputs them to other functional blocks as necessary.

Here, if the learning data processed by the word extraction unit 22 is input data previously processed by the information processing device 17, the association between the input data and the processing result is maintained for the extracted words. For example, if the learning data is input data including vehicle defect information classified according to the classification criteria described below, the association between the classification result according to the importance of the defect and the classification method is maintained for the words extracted by the word extraction unit 22. Similarly, when the processing reference data is processed by the word extraction unit 22, the association between the extracted words and the classification criteria is maintained. For example, if the processing reference data is an operating procedure manual for classifying defects of products such as vehicles according to the importance, the association between the extracted words and the importance is maintained. Note that when already structured corpus data is used, the process of extracting words from the corpus data by the word extraction unit 22 can be omitted as necessary.

The co-occurrence word extraction unit 23 has a function of extracting words that frequently occur together with each word extracted by the occurrence word extraction unit 22. For each word extracted by the occurrence word extraction unit 22, the co-occurrence word extraction unit 23 extracts words that occur frequently together with a certain word in the same text data as words that frequently occur together with that word. The occurrence frequency is the ratio of the number of times a word that occurs together with a certain word to the total number of extracted words. In this specification, words that frequently occur together with a certain word in the same text data are also referred to as "co-occurrence words".

When co-occurrence words are extracted from the words extracted by the word occurrence extraction unit 22, for example, a co-occurrence network is generated. A co-occurrence network is a diagram that visualizes the frequency of occurrence in text data of a certain word and its co-occurrence word, the distance between them when they occur, etc. The distance between a certain word and its co-occurrence word means how many words are separated between the word and its co-occurrence word. Such a co-occurrence network can be generated using a program such as "KH Coder" or "Text Mining Studio".

When extracting co-occurrence words from the generated co-occurrence network, the co-occurrence word extraction unit 23 extracts, as co-occurrence words, words that appear together with a certain word and have a frequency of occurrence equal to or greater than a predetermined value. For example, among words that appear together with a certain word in the same text data, the top 5 to 30 words with respect to frequency of occurrence are extracted as co-occurrence words. Alternatively or in addition to this, the co-occurrence word extraction unit 23 extracts, as co-occurrence words, words that appear together with a certain word and are spaced apart from each other by a predetermined distance or less. For example, among words that appear together with a certain word, words that are spaced apart from each other by 1 to 10 words are extracted as co-occurrence words. The predetermined value of the frequency of occurrence and the predetermined distance can be selected as appropriate values according to the type and amount of data from which the word extraction unit 22 has performed word extraction processing. In addition, the number of co-occurrence words extracted by the co-occurrence word extraction unit 23 may be different for each word, and the same word may be extracted as a co-occurrence word for different words.

When the co-occurrence network is generated by the co-occurrence word extraction unit 23, information other than text data may be used. In particular, when the words extracted from the learning data by the occurrence word extraction unit 22 are associated with previous processing results, the co-occurrence word extraction unit 23 can extract co-occurrence words for each group according to the previous processing results. For example, when the words extracted from the learning data by the occurrence word extraction unit 22 are associated with classifications according to the severity of vehicle defects, the co-occurrence word extraction unit 23 can extract co-occurrence words for each classification according to the severity of the defects. Then, the co-occurrence word extraction unit 23 stores the co-occurrence words extracted from the input data, the co-occurrence words extracted from the learning data, the co-occurrence words extracted from the processing reference data, and the co-occurrence words extracted from the corpus data in a storage unit such as the RAM 183 in correspondence with the original data processed by the occurrence word extraction unit 22, and outputs them to other functional blocks as necessary. At this time, the extracted co-occurrence words may be listed for each word.

The classification rule generation unit 24 has a function of generating rules for processing input data using at least one of the words extracted by the occurrence word extraction unit 22 and the co-occurrence words extracted by the co-occurrence word extraction unit 23. The rules for processing input data are, in particular, rules for classifying input data. The classification rule generation unit 24 expresses the text data included in the input data, learning data, processing reference data, and corpus data acquired by the data acquisition unit 21 as vectors using at least one of the words extracted by the occurrence word extraction unit 22 and the co-occurrence words extracted by the co-occurrence word extraction unit 23. This text data expressed as a vector becomes the rule used when processing input data in the information processing device 17.

The classification rule generating unit 24 can generate multiple rules by selecting which data to use from the input data, the learning data, the processing reference data, and the corpus data, and also whether to use the words extracted by the occurrence word extracting unit 22 or the co-occurrence words extracted by the co-occurrence word extracting unit 23. In this embodiment, among these rules, the first rule is a rule in which the processing reference data is expressed as a vector using the words extracted by the occurrence word extracting unit 22, and the second rule is a rule in which the co-occurrence words of the learning data extracted by the co-occurrence word extracting unit 23 are combined and the resulting combination of co-occurrence words is expressed as a vector. In this embodiment, the learning data, which is the source data of the second rule, includes at least past input data classified by the first classification unit 27a or the second classification unit 27b described later.

The first rule is a vector representation of a predetermined rule, such as a document published by a government agency, a work procedure manual, a product warning label, or a user manual, while the vector generated by the second rule changes depending on the learning data used. Therefore, by processing input data in the information processing device 17 of this embodiment and feeding back the processed input data as learning data after associating it with the processing result, the second rule can be automatically improved and the input data can be processed more appropriately using the second rule. In other words, the second rule is a rule learned by machine learning using the learning data. Below, we will explain how to generate the first and second rules.

The processing standard data that is the source data for the first rule is documents made public by public agencies, work instructions for classifying defects in products such as vehicles according to their importance, product warnings or usage manuals, etc., and are predetermined rules. Examples of documents made public by public agencies include automobile recall and defect information made public as a database by the Ministry of Land, Infrastructure, Transport and Tourism. Examples of work instructions include standards for classifying information on vehicle defects received from dealers according to three levels of importance: high, medium, and low. Such work instructions state, for example, that if the reported defect corresponds to "all the doors of the vehicle cannot be opened from the inside," the defect is classified as "high importance," if the defect corresponds to "some doors of the vehicle cannot be opened or unlocked from the inside," the defect is classified as "medium importance," and if the defect corresponds to "some doors of the vehicle cannot be opened or unlocked from the outside," the defect is classified as "low importance." For example, if the product is a hairspray that uses gas, a warning or instruction manual for the product would be something like "Do not place the product near open flames, as there is a risk of explosion or damage" written on the spray can.

To generate the first rule, the classification rule generation unit 24 obtains the words extracted from the processing reference data as described above from the word occurrence extraction unit 22. In this embodiment, it is assumed that at this stage, the word occurrence extraction unit 22 has performed stemming processing, stop word removal, and compound word search. The extracted words are listed for each text data or sentence. The extracted words are then arranged horizontally as word labels, and a one-row matrix is created for each text data or sentence, for example, by assigning 1 to words that occur and 0 to words that do not occur. This matrix with word labels is also called a word vector. This processing is then performed for each text data or sentence, and the generated word vectors are compiled into one, which becomes the first rule.

For example, in the case of the above-mentioned operating procedure manual, for the phrase "All the doors of the vehicle do not open from the inside", which is classified as "high importance", the morphological analysis function of the word extraction unit 22 can extract words by dividing it into "vehicle/all/of/doors/do ...

Similarly, for "Some of the doors of the vehicle cannot be opened or unlocked from the inside" classified as "medium importance", the morphological analysis function of the occurring word extraction unit 22 extracts words by dividing them into "Some of the doors of the vehicle cannot be opened from the inside, or cannot be unlocked", removes "of", "ga", "kara", "," and "or" as stop words, and extracts "cannot be opened" as the compound word "cannot be opened" and "cannot be unlocked" as the compound word "cannot be unlocked". In this way, the words "vehicle/some/door/inside/cannot be opened/unlocked" can be extracted from the processing reference data "Some of the doors of the vehicle cannot be opened or unlocked from the inside". Then, from these extracted words, a word vector shown in the "medium" row of Table 1 below is generated for classifying the input data as "medium importance".

Similarly, for "Some of the doors of the vehicle cannot be opened from the outside or cannot be unlocked" classified as "low importance", the morphological analysis function of the occurring word extraction unit 22 extracts words by dividing them into "Some of the doors of the vehicle cannot be opened from the outside, or cannot be unlocked", removes "of", "ga", "kara", "," and "or" as stop words, and extracts "cannot open" as the compound word "cannot open" and "cannot unlock" as the compound word "cannot unlock". In this way, the words "vehicle/some/door/outside/cannot open/cannot unlock" can be extracted from the processing reference data "Some of the doors of the vehicle cannot be opened from the outside or cannot be unlocked". Then, from these extracted words, a word vector shown in the "low" row of Table 1 below is generated for classifying the input data as "low importance".

In this way, even when the processing criteria data is processed by the word occurrence extraction unit 22, the association between the extracted words and the classification criteria is maintained, so that each word vector can be associated with the classification criteria.

In response to this, in order to generate the second rule, the classification rule generation unit 24 acquires the co-occurrence words extracted from the learning data from the co-occurrence word extraction unit 23. When generating the second rule, the classification rule generation unit 24 extracts combinations of words with strong co-occurrence relationships from the acquired list of co-occurrence words. For example, the classification rule generation unit 24 extracts combinations of words that are in the top 5 most frequently occurring words among words that appear together with a certain word in the same text data, and that are 5 words or less away from each other. The conditions for this extraction can be set as appropriate within a range in which the number of combinations extracted does not exceed the processing limit of the information processing device 17. Then, the extracted words are arranged horizontally as word labels, and a word vector is created for each combination by assigning 1 to the words extracted as a combination and 0 to the other words. This process is performed for each combination, and the generated word vectors are compiled into one to become the second rule.

In this embodiment, the extracted co-occurrence words are listed for each word, and the acquired co-occurrence words are extracted for each group according to the previous processing result. By extracting co-occurrence words for each group according to the previous processing result, it is possible to match each word vector with the processing result. For example, if the co-occurrence words used to obtain the second rule are extracted for each group according to the severity of a vehicle defect, it is possible to match the generated word vector with the severity of the defect. In other words, when the input data includes information about a vehicle defect, the second rule can be a rule for classifying the data according to the severity of the vehicle defect.

For example, when the learning data includes data that was previously input to the information processing device 17 and classified as "high importance" by the first classification unit or the second classification unit, the classification rule generation unit 24 acquires co-occurrence words of the data classified as "high importance" from the co-occurrence words extracted by the co-occurrence word extraction unit 23. When the co-occurrence words of the data classified as "high importance" are (door, front, right side, left side, strange sound, all, does not open), the classification rule generation unit 24 extracts "door, strange sound, all, does not open" as an example of a combination of co-occurrence words. Then, the words included in this combination are arranged horizontally as word labels, and a one-row matrix is created by assigning 1 to the words included in the combination and 0 to the other words. If the combination is "door, front, strange sound, all, does not open", a one-row matrix shown in the "high" row of Table 2 below, that is, a word vector, is generated. The input data is classified as "high importance" by this word vector.

Similarly, when data previously classified as "medium importance" by the first classification unit or the second classification unit is included in the training data, the classification rule generation unit 24 obtains co-occurrence words of the data classified as "medium importance" from the co-occurrence words extracted by the co-occurrence word extraction unit 23. When the co-occurrence words of the data classified as "medium importance" are (vehicle, both sides, door, strange sound, behind, does not open), the classification rule generation unit 24 extracts "vehicle, door, strange sound, behind, does not open" as an example of a combination of co-occurrence words. Then, from this combination, the word vector shown in the "medium" row in Table 2 below is generated. The input data is classified as "medium importance" based on this word vector.

Similarly, when data previously classified as "low importance" by the first classification unit or the second classification unit is included in the training data, the classification rule generation unit 24 obtains co-occurrence words of the data classified as "low importance" from the co-occurrence words extracted by the co-occurrence word extraction unit 23. When the co-occurrence words of the data classified as "low importance" are (vehicle, door, difficult to open, front, strange sound, rear), the classification rule generation unit 24 extracts "door, difficult to open, strange sound, rear" as an example of a combination of co-occurrence words. Then, from this combination, the word vector shown in the "low" row of Table 2 below is generated. The input data is classified as "low importance" based on this word vector.

In this way, co-occurring words are extracted for each classification result of the learning data, so that each word vector can be associated with a classification criterion. Here, we have described a process in which the first and second rules are used to classify data according to the severity of the vehicle malfunction when the input data contains information about a vehicle malfunction. However, the classification criterion is not limited to the severity of the vehicle malfunction, and may be the location where the vehicle malfunction occurred, the maintenance status of the vehicle where the malfunction occurred, etc. Furthermore, the method of generating the first and second rules described above is merely one example, and other methods may be used.

The similarity calculation unit 25 has a function of calculating the similarity between the data on which the rules (i.e., one or more word vectors) generated by the classification rule generation unit 24 are based and the reference data for updating the rules generated by the classification rule generation unit 24 by the rule update unit 26. In particular, the similarity calculation unit 25 of this embodiment has a function of calculating the similarity between predetermined data on which the first rule is based (i.e., processing reference data) and the reference data for updating the first rule. The reference data is acquired from the first database 11 by the data acquisition unit 21. In addition, in this embodiment, the reference data may include at least one of the input data and the learning data.

The similarity calculated by the similarity calculation unit 25 is an index showing the degree to which a certain text data overlaps with another text data, and may take a positive value in the range of 0 to 1, for example. The higher the similarity value, the more similar the text data to be compared are, and the lower the value, the more dissimilar the text data to be compared are. For example, the frequency of occurrence of words that appear in the processing reference data and overlap with words that appear in the reference data is calculated as the similarity. The frequency of occurrence of such overlapping words can be calculated by comparing the word vector generated from the processing reference data with the word vector generated from the reference data. Specifically, the word labels of the word vector generated from the processing reference data are compared with the word labels attached to the reference data, and overlapping words are extracted. Then, the frequency of occurrence of the extracted overlapping words is calculated as the similarity.

Here, when generating a word vector from the processing reference data, the same method as the method of generating the word vector of the first rule described above can be used, and when generating a word vector from the reference data, the same method as the method of generating the word vector of the first rule described above can be used. For example, if the reference data includes input data such as "The front door of the vehicle cannot be opened from the inside" and "All the doors cannot be opened from the outside", word vectors such as data 1 and data 2 in Table 3 below are generated through morphological analysis, removal of stop words, and extraction of compound words. Also, if the reference data includes learning data or data classified by another information processing device having a classification function similar to that of this embodiment, the co-occurrence word extraction unit 23 may extract co-occurrence words from the reference data and generate a word vector using the extracted co-occurrence words, as in the method of generating a word vector of the second rule described above. For example, if the co-occurrence words of the data classified as "high importance" are (vehicle, door, inside, does not open), the classification rule generation unit 24 extracts "door, inside, does not open" as an example of a combination of co-occurrence words, and generates a word vector such as data 3 in Table 3 below from this combination.

The similarity in this embodiment is, for example, the similarity between a sentence of text data included in the specified processing reference data and a sentence of text data included in the reference data. The similarity is calculated from a word vector generated from the specified processing reference data and a word vector generated from the reference data. Specifically, the word vector generated from the specified processing reference data is compared with the word vector generated from the reference data, and the word label of the word vector is compared with an input value of 0 or 1 indicating the occurrence of a word corresponding to the word label. Next, it is determined whether the input value of 0 or 1 indicating the occurrence of a corresponding word in a certain word label differs or matches between the two word vectors. Then, the number of input values that match or differ is calculated as the similarity.

If the number of differences between the input values of 0 or 1 for word labels in two word vectors is small, then the two word vectors and the sentences corresponding to them are similar. On the other hand, if the number of differences between the input values of 0 or 1 for word labels in two word vectors is large, then the sentences corresponding to the two word vectors have many differences and are not similar. When calculating the similarity between a word vector generated from specified processing standard data and a word vector generated from reference data, the cosine similarity between the word vectors may be calculated.

Alternatively or in addition, the similarity in this embodiment may be the similarity between words in text data included in the predetermined processing reference data and words in text data included in the reference data. When the meaning of a word is taken into consideration, co-occurring words of the words are compared. In other words, when a word and another word are similar in meaning, the co-occurring words of the word and the co-occurring words of the other words are also similar. In this embodiment, by using a co-occurring word list extracted from corpus data as an index, it is possible to compare the co-occurring words of a word and the co-occurring words of other words in the corpus data and calculate the similarity in meaning between the words. For example, the Simpson-Jaccard coefficient can be used to compare the co-occurring words. The Simpson-Jaccard coefficient is an index obtained by arithmetically averaging the Jaccard coefficient and the Simpson coefficient, and is expressed by the following formula (1).

Here, _w1 and _w2 in equation (1) represent words, Jaccard( _w1 , _w2 ) is the Jaccard coefficient, and Simpson( _w1 , _w2 ) is the Simpson coefficient.

Specifically, the word labels of the word vectors generated from the processing reference data are compared with the word labels of the word vectors generated from the reference data to extract overlapping words. Next, for the extracted overlapping words, a co-occurrence word list extracted from the corpus data is obtained. The co-occurrence word list is generated by the co-occurrence word extraction unit 23. Then, the Simpson-Jaccard coefficient is calculated using the overlapping words and the co-occurrence word list extracted from the corpus data.

As an example, the similarity between the word label [part, inside] of a word vector generated from the processing reference data and the word label [left, inside] of a word vector generated from the reference data is calculated as the Simpson-Jaccard coefficient. If it is known from the co-occurrence word list extracted from the corpus data that the similarity between [part] and [left] is 0.7, and the similarity between [inside] and [inside] is 0.9, the Simpson-Jaccard coefficient in this case is (0.7 + 0.9) / 2 = 0.8, and a similarity of 0.8 is calculated as the average similarity. Note that the similarity between completely matching words is calculated as 1.

In this way, by considering the similarity of the meanings of the word labels (i.e., the similarity between words) in addition to the similarity of the input values of the word vectors (i.e., the similarity between sentences in the text data), the similarity between the specified processing reference data and the reference data can be calculated more accurately. For example, if the word vector generated from the processing reference data is [door, inside, does not open] and the word vector generated from the reference data is [door, outside, does not open], the number of differences in the input values of the word vectors is 1, so the similarity of the two word vectors as sentences of the text data is calculated to be high. However, because the word labels [inside] and [outside] have opposite meanings, when the meanings of the words are also taken into account, the similarity of the two word vectors is calculated to be low.

As another method of calculating similarity, the similarity calculation unit 25 may compare the word labels of the word vectors generated from the processing reference data with the word labels of the word vectors generated from the reference data, and calculate the similarity between all words appearing in the text data of the processing reference data and all words appearing in the text data of the reference data. The similarity between words is calculated using the above-mentioned Simpson-Jaccard coefficient or the like. In this case, a co-occurrence network may be generated using the word labels of the processing reference data and the word labels of the reference data to calculate the similarity. By extracting co-occurrence words using the co-occurrence network, it is possible to extract words that are highly similar to each other among words that appear in two pieces of text data.

Furthermore, the similarity calculation unit 25 may extract sentences (i.e., word vectors) of the text data of the reference data that are similar to sentences (i.e., word vectors) of the processing reference data based on the extracted co-occurring words. In addition, the similarity of the extracted reference data word vectors with the word vectors of the processing reference data may be calculated. This makes it possible to efficiently extract word vectors with high similarity when at least one of the reference data or the processing reference data includes multiple data.

The rule update unit 26 has a function of updating the rules for classifying input data in the input data classification unit 27 using the reference data. Specifically, the rule update unit 26 has a function of updating the first rule using reference data whose similarity calculated by the similarity calculation unit 25 is equal to or greater than a predetermined value. The predetermined value can be set to an appropriate value depending on the method of calculating the similarity. For example, when the number of matches between the processing reference data, the word vector, and the word vector value (0 or 1) of the reference data is calculated as the similarity, the predetermined value is set to 3, and reference data whose number of matches is 3 or more is used. Alternatively, when the number of differences between the processing reference data, the word vector, and the word vector value (0 or 1) of the reference data is calculated as the similarity, reference data whose number of differences is 3 or less may be used. Furthermore, when the Simpson-Jaccard coefficient of the word label of the processing reference data and the word label of the reference data is calculated as the similarity, the predetermined value is set to 0.6, and reference data whose similarity is 0.6 or more is used.

In addition, when the similarity between a sentence of text data included in the specified processing reference data and a sentence of text data included in the reference data, and between a word of text data included in the specified processing reference data and a word of text data included in the reference data are calculated as the similarity, reference data in which either one of the similarities is equal to or greater than a specified value may be used, or only reference data in which both similarities are equal to or greater than a specified value may be used. For example, the rule update unit 26 may use reference data in which the number of differences between the word vectors of the processing reference data and the word vectors of the reference data (0 or 1) is 3 or less, and the Simpson-Jaccard coefficient of the word labels of the processing reference data and the word labels of the reference data is 0.6 or more.

Updating the first rule using reference data means, for example, extracting from the reference data that is similar to the processing reference data, which is the data on which the first rule is based, and adding the extracted reference data to the processing reference data in association with it. Specifically, the word vectors of the reference data are compared with the word vectors of the processing reference data, and from the word vectors of the reference data, those that have a similarity to the word vectors of the processing reference data that is equal to or greater than a predetermined value are extracted. Then, in the first rule, the word vectors of the extracted reference data are inserted below the word vectors of the processing reference data.

For example, suppose that the word vector shown in the "High" row of Table 1 has been generated for the classification criterion "None of the vehicle doors can be opened from the inside" that classifies input data as "high importance," and then the word vector shown in Table 4 below corresponding to the data "None of the vehicle doors can be opened" is extracted from the word vectors of the reference data. "High importance" is associated with this word vector as the importance of the defect.

In this case, the rule update unit 26 inserts the word vector shown in Table 4 below, below the word vector in the row for "high", in the first rule, as shown in Table 5 below. The inserted word vector is a word vector added by the rule update unit 26, and is inserted with, for example, "(high)" in parentheses to indicate that it is associated with "high".

In this way, the rule update unit 26 can increase the number of word vectors that make up the first rule, thereby increasing the number of input data that can be classified by the first rule while suppressing a decrease in data processing accuracy caused by changing the rules for classifying input data.
In addition, the rule update unit 26 does not necessarily need to increase the number of word vectors that make up the first rule. For example, the rule update unit 26 may replace the word vectors that make up the first rule with word vectors of reference data that have a similarity greater than or equal to a predetermined value.

The rule update unit 26 also has a function of learning the second rule using data classified by the first classification unit 27a (described later) according to not only the first rule but also the updated first rule. Specifically, the data input to the first classification unit 27a is associated with the classification results and stored as learning data in the second database 12. This feedback changes the co-occurrence words extracted by the co-occurrence word extraction unit 23, and the second rule generated by the classification rule generation unit 24 is trained to classify according to a predetermined classification criterion.

Furthermore, the rule update unit 26 may generate new rules using word vectors of reference data whose similarity is equal to or greater than a predetermined value. In particular, new rules may be generated using co-occurrence words that appear with a frequency equal to or greater than a predetermined frequency together with words included in the first rule, extracted from reference data whose similarity is equal to or greater than a predetermined value. Specifically, the similarity calculation unit 25 generates a co-occurrence network using word labels of the processing reference data and word labels of the reference data, and when co-occurrence words are extracted, combines the co-occurrence words to generate word vectors similar to the word vectors of the processing reference data. This makes it possible to further increase the number of first rules.

The input data classification unit 27 has a function of classifying data input by the input device 11a according to rules. In particular, the input data classification unit 27 of this embodiment includes a first classification unit 27a that classifies the input data according to the first rule updated by the function of the rule update unit 26, and a second classification unit 27b that classifies the input data according to the second rule. As described above, the first rule associates a word vector with a classification criterion, and the second rule associates a word vector with a previous processing result. Therefore, the first classification unit 27a compares the word vector of the input data with the word vector of the first rule, and if the similarity is equal to or greater than a predetermined value, the input data can be classified according to the associated classification criterion. Similarly, the second classification unit 27b compares the word vector of the input data with the word vector of the second rule, and if the similarity is equal to or greater than a predetermined value, the input data can be classified according to the associated previous processing result.

In the input data classification unit 27 of this embodiment, if the word vector of the input data matches the word vector constituting the first rule, the data is classified according to the first rule. In contrast, if the word vector of the input data does not match the word vector constituting the first rule, the data is classified according to the second rule. For example, when the input data includes information on a vehicle malfunction, the input data is classified into three or more groups (for example, three groups of high importance, medium importance, and low importance) according to the importance of the vehicle malfunction. If the word vector of the input data matches the first rule, the data is classified by the first classification unit 27a, and if the word vector of the input data does not match the first rule, the data is classified by the second classification unit 27b. This makes it possible to avoid classifying data twice according to the updated first rule and the second rule, and to shorten the processing time in the information processing device 17.

The display image generating unit 28 has a function of generating an image to be displayed on the display device 16 to inform the user of the processing status of the input data. The image generated by the display image generating unit 28 may indicate the progress of the processing or the processing result, and may include, for example, at least one of the similarity calculated by the similarity calculating unit 25, the reference data whose similarity is equal to or greater than a predetermined value, the updated first rule, and the updated part of the updated first rule.

Figure 2 is a diagram showing an example of an image displayed on the display device 16 of Figure 1. The image shown in Figure 2 is an image displayed on the display device 16 when the input data, which includes information about a vehicle defect, is classified into three groups, namely, high importance, medium importance, and low importance, according to the importance of the vehicle defect. Figure 2 (A) is a first display image 3 showing the classification result of the input data, Figure 2 (B) is a second display image 4 showing the update result of the first rule and the calculation result of the similarity, and Figure 2 (C) is a third display image 5 showing the result when co-occurring words are extracted and an overview of the extraction process.

In the first display image 3 in FIG. 2(A), which shows the classification result, "input data" is displayed in the upper left of the image. Multiple pieces of input data may be input, and in that case, all data can be checked by scrolling the screen using the triangular icon shown in the lower right of the input data. The classification result of the input data is displayed in the upper right of the image, and it is displayed that the input data in FIG. 2(A) has been classified as "medium importance." In the lower left of the image, "rule-based model" is displayed as the "classification method," indicating that classification has been performed by the first classification unit 27a according to the updated first rule, and the applied rule is displayed in the lower right of the image. Note that if classification has been performed by the second classification unit 27b, "machine learning model" is displayed in the lower left of the image.

In the second display image 4 in FIG. 2(B), which shows the update result and the similarity, the extracted "reference data" is displayed in the upper left of the image, and the extracted sentence is displayed below that as "extracted sentence". If co-occurring words are extracted, the combination of co-occurring words is displayed as a "keyword list" in the upper right of the image. For the "reference data" and "keyword list", all data can be checked by scrolling the screen using the triangular icon. At the bottom of the image, the classification result of the input data is displayed in the lower left as the "rule update result", and the first rule that was applied is displayed to the right of that. Then, in the lower right of the image, a sentence similar to the applied first rule that was extracted from the reference data is displayed. In the case of FIG. 2(B), "There is a strange noise coming from the left rear door" has a higher similarity than "There is a strange noise coming from the right front door", so "There is a strange noise coming from the left rear door" is added to the first rule.

In the third display image 5 in Figure 2 (C), which shows the results of co-occurrence word extraction, "reference data" is displayed on the right side of the image, and the generated co-occurrence network is displayed below that. In addition, on the left side of the image, a "keyword list" is displayed showing the combinations of extracted co-occurrence words and "extraction conditions." The "extraction conditions" displayed include "word occurrence count," which indicates the frequency of word occurrence, and "co-occurrence threshold," which indicates the distance between words. Note that the displayed extraction conditions are merely an example, and other conditions can be displayed as necessary.

The data output unit 29 has a function of outputting the image data generated by the display image generation unit 28 to the display device 16. This allows an image to be displayed on the display device 16, and the user of the information processing system 1 to be notified of the processing status of the input data.

[Processing in Information Processing System]
The procedure for information processing by the information processing device 17 will be described with reference to Fig. 3. Fig. 3 is an example of a flowchart showing information processing in the information processing system 1 of the present embodiment. The process described below is executed by the processor 18 of the information processing device 17 at predetermined time intervals.

First, in step S1, the data acquisition unit 21 acquires necessary data from each of the first database 11 to the fourth database 14. In the following step S2, the occurrence word extraction unit 22 extracts words used in the data acquired by the data acquisition unit 21 using a morphological analysis tool or the like. For example, from the data "All doors do not open from the inside", the words "vehicle/all/doors/inside/do not open" are extracted through morphological analysis, stop word deletion, and compound word extraction. In the following step S3, the co-occurrence word extraction unit 23 generates a co-occurrence network and extracts co-occurrence words, particularly for words extracted from the learning data, the processing reference data, and the corpus data. In the following step S4, the classification rule generation unit 24 generates first rules using the words extracted from the processing reference data, and generates second rules using the co-occurrence words extracted from the learning data.

Next, in step S5, the function of the similarity calculation unit 25 is used to calculate the similarity between the specified data on which the first rule is based and the reference data. Specifically, the word vectors of the processing reference data are compared with the word vectors of the reference data, and the number of overlaps of the word vectors, the similarity of the meanings of the words corresponding to the word labels, etc. are calculated as similarities. In the following step S6, the function of the rule update unit 26 is used to update the first rule using reference data having a similarity equal to or greater than a predetermined value, and a new rule is added. In the following step S7, it is determined whether the input data can be classified according to the updated first rule.

If the word vectors of the input data match the word vectors of the first rule and it is determined that the data can be classified according to the updated first rule, the process proceeds to step S8a, where the first classification unit 27a classifies the data according to the updated first rule. On the other hand, if the word vectors of the input data do not match the word vectors of the first rule and it is determined that the data cannot be classified according to the updated first rule, the process proceeds to step S8b, where the second classification unit 27b classifies the data according to the second rule.

The results of these classification processes are processed in the following step S9, and an image for display on the display device 16 is generated by the function of the display image generation unit 28. The generated image is output to the display device 16 by the function of the data output unit 29 in the following step S10. The image output from the data output unit 29 is then displayed on the display device 16, and the user of the information processing system 1 is notified of the processing results. Furthermore, in step S11, the classification results of the first classification unit in step S8a are fed back to the learning data by the function of the rule update unit 26, and the second rule is learned.

[Embodiments of the invention]
As described above, the information processing device 17 of the present embodiment includes a similarity calculation unit 25 that calculates a similarity between predetermined data used when generating a first rule for classifying input data and reference data for updating the first rule, a rule update unit 26 that updates the first rule using the reference data in which the similarity calculated by the similarity calculation unit 25 is equal to or greater than a predetermined value, and an input data classification unit 27 that classifies the input data using the updated first rule and a second rule that is different from the first rule and has been learned by machine learning using learning data. This makes it possible to prevent a decrease in data processing accuracy caused by changing the rule for classifying input data, while avoiding double classification of data using the updated first rule and the second rule, thereby shortening the processing time in the information processing device 17.

Furthermore, according to the information processing device 17 of this embodiment, the similarity is the similarity between a sentence of text data included in the specified data and a sentence of text data included in the reference data, and/or the similarity between a word of text data included in the specified data and a word of text data included in the reference data. This makes it possible to more accurately calculate the similarity between the specified processing reference data and the reference data.

Furthermore, according to the information processing device 17 of this embodiment, the learning data includes data that was previously input and classified by the input data classification unit. This makes it possible to update the first rule and learn the second rule using the past input data, thereby further improving classification accuracy.

Furthermore, according to the information processing device 17 of this embodiment, the reference data includes at least one of the input data and the learning data. This makes it possible to update the first rule and learn the second rule using the input data, thereby further improving classification accuracy.

Furthermore, according to the information processing device 17 of this embodiment, the rule update unit 26 learns the second rule using data classified by the input data classification unit 27 according to the updated first rule. This promotes learning of the second rule.

Furthermore, according to the information processing device 17 of this embodiment, the updated first rules include new rules generated by the rule update unit 26 using the reference data whose similarity is equal to or greater than a predetermined value. This makes it possible to further increase the number of first rules.

Furthermore, according to the information processing device 17 of this embodiment, the updated first rules include new rules generated using words that appear with a frequency equal to or greater than a predetermined frequency together with the words included in the first rules, which are extracted from the reference data. This makes it possible to further increase the number of first rules.

In addition, the information processing device 17 of this embodiment further includes a display device 16 that displays at least one of the similarity, the reference data for which the similarity is equal to or greater than a predetermined value, the updated first rule, and the updated portion of the updated first rule. This makes it possible to notify the user of the information processing system 1 of the processing status of the input data.

Furthermore, according to the information processing method of this embodiment, when input data is classified using a first rule generated from predetermined data and a second rule different from the first rule and learned by machine learning using learning data, the similarity between the predetermined data and reference data for updating the first rule is calculated, the first rule is updated using the reference data whose similarity is equal to or greater than a predetermined value, and the input data is classified using the updated first rule and the second rule. This makes it possible to prevent a decrease in data processing accuracy caused by changing the rules for classifying input data, while avoiding double classification of data using the updated first rule and the second rule, thereby shortening the processing time in the information processing device 17.

1... Information processing system 11... First database (input data)
11a...input device 12...second database (learning data)
12a... Server 13... Third database (processing reference data)
14...Fourth database (corpus data)
15: Communication device 16: Display device 17: Information processing device 18: Processor 181: CPU
182...ROM
183...RAM
2... Input data processing section 21... Data acquisition section 22... Appearance word extraction section 23... Co-occurrence word extraction section 24... Classification rule generation section 25... Similarity calculation section 26... Rule update section 27... Input data classification section 27a... First classification section 27b... Second classification section 28... Display image generation section 29... Data output section 3... First display image 4... Second display image 5... Third display image

Claims (6)

a similarity calculation unit that calculates a degree of overlap between processing reference data, which is text data indicating a classification standard for classifying input data according to importance and which is used when generating a first rule for classifying the input data, which is text data written in a natural language, and reference data for updating the first rule, which includes at least one of the input data and learning data, which is text data including the input data associated with the importance, the similarity indicating a degree of overlap between the processing reference data and reference data for updating the first rule, the similarity being an appearance frequency of words that overlap with words that appear in the reference data among words that appear in the processing reference data used when generating the first rule;
a rule update unit that updates the first rule by using the reference data in which the similarity calculated by the similarity calculation unit is equal to or greater than a predetermined value;
an input data classification unit that classifies the input data using the updated first rule and a second rule that is different from the first rule and that has been learned by machine learning using the learning data ,
An information processing device, wherein the first rule updated by the rule update unit includes new rules extracted from the reference data and generated using co-occurring words that appear together with the words included in the first rule with a frequency equal to or greater than a predetermined frequency . 2. The information processing device according to claim 1, wherein the similarity is the similarity between a sentence of the text data included in the processing reference data and a sentence of the text data included in the reference data, and/or the similarity between a word of the text data included in the processing reference data and a word of the text data included in the reference data. The information processing device according to claim 1 , wherein the rule update unit learns the second rule by using data classified by the input data classifying unit in accordance with the updated first rule. The information processing device according to claim 1 , wherein the updated first rule includes a new rule generated by the rule update unit using the reference data whose similarity is equal to or greater than a predetermined value. 5. The information processing device according to claim 1, further comprising a display device that displays at least one of the similarity, the reference data for which the similarity is equal to or greater than a predetermined value, the updated first rule, and an updated portion of the updated first rule. An information processing method executed by an information processing device classifies input data , which is text data written in a natural language , using a first rule generated from processing reference data, which is text data indicating classification criteria for classifying the input data according to importance , and a second rule learned by machine learning using learning data, which is text data different from the first rule and includes the input data associated with the importance ,
The information processing device includes:
calculating a similarity indicating a degree of overlap between the processing reference data and reference data for updating the first rule, the reference data including at least one of the input data and the learning data , the similarity being an occurrence frequency of words that appear in the processing reference data used when generating the first rule and that overlap with words that appear in the reference data;
updating the first rule using the reference data whose similarity is equal to or greater than a predetermined value;
classifying the input data using the updated first rule and the second rule ;
An information processing method, wherein the updated first rule includes a new rule extracted from the reference data and generated using co-occurring words that appear together with the words included in the first rule with a frequency equal to or greater than a predetermined frequency .

Images