JP6720764B2 - Text analysis device and program - Google Patents

Description

The present invention relates to a text analysis device and a program.

Conventionally, various techniques for analyzing text have been developed.

Patent Document 1 describes that morphological analysis and syntactic analysis are performed on a Japanese sentence, and emotion information is extracted from the result using a word dictionary in which emotion elements are registered and an emotion information conversion rule. There is.

Non-Patent Document 1 describes that a convolutional neural network, which is a type of machine learning method, is used to perform part-of-speech assignment, chunk identification, proper name extraction, and semantic role assignment on text.

Non-Patent Document 2 describes that a recursive neural network, which is a type of machine learning method, is used to perform reputation classification in units of partial branches of syntax trees.

Japanese Patent Publication No. 6-82377

Ronan Collobert, Jason Weston, Leon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. 2011. Natural Language Processing (Almost) from Scratch. Journal of Machine Learning Research 12, pp. 2461-2505. Richard Socher, Alex Perelygin, Jean Y. Wu, Jason Chuang, Christopher D. Manning, Andrew Y. Ng, and Christopher Potts. 2013. Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank. In Proceedings of 2013 Conference on Empirical Methods in Natural Language Processing, pp. 1632-1642

When performing text analysis, unless the classification information in word units is incorporated into the machine learning model, the accuracy of text analysis is insufficient. Further, when the syntax tree corpus to which the classification information is added is used instead of using the classification information for each word, the cost is increased by the amount of the corpus construction.

An object of the present invention is to provide an apparatus and a program that can efficiently perform text analysis using classification information in word units.

The invention according to claim 1 is one or more around the entry word with the classification information including a label of any of positive, negative, neutral, and emotions from the unlabeled text. Is a snippet extracting means for extracting the word as a snippet, and an acquiring means for pre- learning with the first neural network using the words in the extracted snippet to acquire the classification information in word units , wherein the words in the snippet are word Convert to a distributed expression, perform a convolution operation on the sequence of the word distributed expression, update the weights of the word distributed expression and the word convolution layer by the error backpropagation method, and acquire as the classification information for each word, acquisition Means and the classification information for each word acquired by the pre-learning by the acquisition means to initialize the weight of the second neural network, and the labeled text is used as teacher data for learning by the second neural network. , Is a learning means for learning to output the classification of the text when any text is input, and initializes the weights of the word distributed expression and the word convolutional layer using the classification information of the word unit, and the labeled Extracting a word from the text as a snippet, converting the word in the extracted snippet into a word distributed expression, performing a convolution operation on the sequence of the word distributed expression, by the error backpropagation method, the initialized word And a learning means for updating the weight of the distributed expression and the word convolution layer .

According to the second aspect of the present invention, the computer is provided with 1 of the surroundings of the headword with the classification information including the label of any of positive, negative, neutral, and emotions from the unlabeled text. A snippet extracting means for extracting one or a plurality of words as a snippet, and an acquiring means for pre-learning with the first neural network using the words in the extracted snippet to acquire the classification information in word units. Converts a word to a word distributed expression, performs a convolution operation on the sequence of the word distributed expression, updates the word distributed expression and the weight of the word convolutional layer by an error backpropagation method, and acquires the word-based classification information. The weighting of the second neural network is initialized using the acquisition unit and the word-based classification information acquired by the pre-learning in the acquisition unit, and the labeled neural network is used as the teacher data for the second neural network. Is a learning means for learning to output the classification of the text when any text is input, and initializes the weights of the word distributed expression and the word convolutional layer using the classification information of the word unit, A word is extracted as a snippet from the labeled text, the words in the extracted snippet are converted into a word distributed expression, a convolution operation is performed on the sequence of the word distributed expression, and an error backpropagation method is used to initialize the word. It is a program that functions as a learning means for updating the word distributed expression and the weight of the word convolution layer.

According to the invention described in claims 1 and 2 , it is possible to obtain the classification information in units of words in advance. Further, it is not necessary to use the syntax tree corpus to which the classification information is added. Further, it is possible to perform highly accurate text analysis as compared with the case where the classification information in units of words is not used.

It is a functional block diagram of an embodiment. It is a configuration block diagram of an embodiment. It is a data example of the dictionary database with classification information of the embodiment. It is an example of data of the text (without label) database of the embodiment. It is a data example of a text database with a sentence/document label of the embodiment. It is a processing flowchart (1) of embodiment. It is a processing flowchart (2) of embodiment. It is a processing flowchart (3) of embodiment. It is an input/output explanatory view of embodiment. It is a system block diagram of a modification.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.

FIG. 1 is a functional block diagram of the text analysis device according to this embodiment. The text analysis device includes a word classification learning unit 10 and a sentence/document classification learning unit 12.

The word classification learning unit 10 is a module that performs machine learning for classification in units of words in advance. The word classification learning unit 10 includes a word extraction unit 103, a snippet extraction unit 104, a word distributed expression conversion unit 105, a word convolution unit 106, a maximum feature extraction unit 107, and a dictionary classification association unit 108.

The word extracting unit 103 reads out text from the text (without label) database 101 and extracts words from the text. Here, the “label” is information based on some classification such as positive (positive), negative (negative), and neutral (neutral). Therefore, the text (without label) is text that is not classified in this way.

The snippet extraction unit 104 extracts a text set including a headword from the text extracted by the word extraction unit 103, and extracts L words (L is, for example, 2, 3, 4,...) Around the headword. Extract the snippet. Here, the "snippet" is a text fragment composed of a dictionary entry word and its surrounding L words in the text. The headword is read from the dictionary database with classification information 102. For example, if the text is "I was told to him that I am very sad", the headword is "sad", and L=2, then 2 before and after the headword "sad" in the text. Extract words as snippets,
"To/Very/Sad/Things/"
Becomes

The word distributed expression conversion means 105 converts the words in the extracted snippet into word distributed expressions. Here, the “word distributed expression” is to represent the meaning of a word by an n-dimensional real number. It can also be called a word vector.

The word convolution unit 106 performs a convolution operation (convolution) with a filter width h1 and a feature map number k1 on the word distributed expression string.

The maximum feature extraction means 107 extracts the maximum value of the k1 feature maps obtained by the calculation by the word convolution means 106.

The dictionary classification associating unit 108 calculates a loss function between the k1 maximum features and the dictionary classification, back-propagates the loss to the word convolutional layer, and updates the weight of the word convolutional layer. Also, the word distributed expression is updated by back-propagating the loss to the word distributed expression. Then, the updated word distributed expression and the weight of the word convolutional layer are stored as the classification information for each word. The dictionary classification associating unit 108 functions as an acquiring unit that acquires classification information in units of words in the embodiment.

On the other hand, the sentence/document classification learning unit 12 is a module that classifies sentences/documents by machine learning using the result of machine learning performed in advance by the word classification learning unit 10. The sentence/document classification learning unit 12 includes a word extraction unit 122, a word distributed expression conversion unit 123, a word convolution unit 124, a sentence/document convolution unit 125, a maximum feature extraction unit 126, and a sentence/document label association unit 127.

The word extracting means 122 selects a labeled text from the text/document labeled text database 121 and extracts a word from the selected text.

The word distributed expression conversion unit 123 converts the words in the snippet into word distributed expressions.

The word convolution unit 124 performs a convolution operation with a filter width h1 and a feature map number k1 on a sequence of word distributed expressions.

The sentence/document convolution unit 125 performs a convolution operation with a filter width h2 and a feature map number k2 on the word convolution result.

The maximum feature extraction means 126 extracts the maximum value for k2 feature maps.

The sentence/document label associating unit 127 calculates the loss function between the k2 maximum features and the label, and propagates the loss back to the sentence/document convolution layer to update the weight of the sentence/document convolution layer. Also, the loss is back-propagated to the word convolutional layer to update the weight of the word convolutional layer. Furthermore, the loss is back-propagated to the word distributed expression to update the word distributed expression. The sentence/document label associating unit 127 functions as an updating unit in the embodiment.

In this embodiment, the “module” means a logically separable component such as software and hardware. Therefore, the module in this embodiment means not only the module in the computer program but also the module in the hardware configuration. The modules may correspond to the functions in a ratio of 1:1, but one module may be configured by one program, or a plurality of modules may be configured by one program. Further, the plurality of modules may be executed by one processor or computer, or may be executed by a plurality of processors or computers in a distributed or parallel environment. In the processing by each module, the target information is read from the storage device, processed by a processor such as a CPU, and then the processing result is output to the storage device and written. The storage device includes an HDD, a RAM, a register in the CPU, and the like.

FIG. 2 is a block diagram showing the configuration of the text analysis device according to this embodiment. The text analysis device is realized by a computer, and specifically includes a CPU 20, a ROM 22, a RAM 24, an input/output interface (I/F) 26, a communication interface (I/F) 28, and a storage device 30.

The CPU 20 reads a processing program stored in the ROM 22, HDD, SSD or the like, and executes various kinds of processing by using the RAM 24 as a working memory. The input/output I/F 26 is a keyboard, a mouse, a display, a touch panel, or the like. The communication I/F 28 connects to a network such as the Internet. The CPU 20 functions as the word extraction means 103, the snippet extraction means 104, the word distributed expression conversion means 105, the word convolution means 106, the maximum feature extraction means 107, and the dictionary classification association means 108 in FIG. 1, and the word extraction means 122, It functions as a word distributed expression conversion unit 123, a word convolution unit 124, a sentence/document convolution unit 125, a maximum feature extraction unit 126, and a sentence/document label association unit 127. The storage device 30 is configured by a hard disk or the like and stores various data and functions as a text (unlabeled) database 101, a classification information-added dictionary database 102, and a text/document labeled text database 121.

The storage device 30 may be provided in a server computer or a cloud computer on the network. That is, the text (unlabeled) database 101, the classification information-added dictionary database 102, and the text/document-labeled text database 121 are configured by a computer different from the text analysis device, and data can be transmitted/received via the communication I/F 28. May be configured.

FIG. 3 shows an example of data stored in the dictionary database 102 with classification information. The ID, the label, and the entry word are stored in association with each other. For example,
The label with ID=1 is "joy" and the headword is "happy"
The label with ID=2 is "angry" and the headword is "angry"
The label with ID=3 is "sorrow" and the headword is "sad"
Etc.

FIG. 4 shows an example of data stored in the text (unlabeled) database 101. The ID and the text are stored in association with each other. For example,
The text with ID=1 was "I told him something very sad."
The text with ID=2 is "I'm glad I got a present unexpectedly."
The text with ID=3 is "very much happy"
Etc.

FIG. 5 shows an example of data stored in the text database 121 with sentence/document labels. The ID, label, and text are stored in association with each other. For example,
The label with ID=1 is "Positive" and the text is "I am very happy to buy a new smartphone!!"
The label with ID=2 is "negative" and the text is "I am sad because I failed at work today".
The label with ID=3 is "Neutral" and the text is "Now moving by train."
Etc.

6, 7 and 8 show process flowcharts of this embodiment. FIG. 6 and FIG. 7 show a process of preliminarily machine learning the word unit classification in the word classification learning unit 10, and FIG. 8 shows a process of machine learning the sentence/document classification in the sentence/document classification learning unit 12. is there. A neural network is used for machine learning, and weights are optimized by using an error backpropagation method. A first neural network is used as a neural network for performing machine learning for word-based classification in advance, and a second neural network is used for machine learning as sentence/document-based classification.

In FIG. 6, the first text is extracted from the text group stored in the text (without label) database 101 (S101). In the example data shown in FIG. 4, the first text is ID=1, "I was told very sadly to him."

Next, words are extracted from the text (S102). A known morphological analysis method can be used for extracting the word.

Next, it is determined whether or not all the texts in the text (without label) database 101 have been processed (S103), and if all the texts have not been processed (NO in S103), the next text is selected and the same process is performed. The process is repeated (S104).

When all texts have been processed (YES in S103), the first entry word is selected from the dictionary database 102 with classification information (S105). In the example of the data shown in FIG. 3, the first entry word is “happy” with ID=1.

Next, the text including the headword is extracted (S106). If the entry word is "happy", the text containing it is "I'm glad I got a present unexpectedly." and "I'm very happy."

Next, snippets of L words around the headword are extracted. For example, if L=2, in the text "I'm glad I received a gift unexpectedly", "Give me / Te / glad /." is extracted as a snippet, and in the text "There are so many things I'm glad." Is extracted as a snippet.

Next, the headword in the snippet is replaced with the unknown word. For example, if the unknown word is XX,
"Give me/te/happy/." → "Give me/te/XX/."
"Happy/Koto/Ga" → "XX/Koto/Ga"
To replace.

Next, it is determined whether or not all headwords in the dictionary database 102 with classification information have been processed (S109), and if all headwords have not been processed (NO in S109), the next headword is selected. The same process is repeated (S110). For example, in the headword "sad", "I was told that he was very sad" was extracted as the text, and "ni/very/sad/koto/wa" was extracted as the snippet of the surrounding L words. Then, replace the headword in the snippet with an unknown word,
“To/Very/Sad/Things/To” → “To/Very/XX/Things/To”
To replace.

When all headwords have been processed (YES in S109), the process proceeds to the process in FIG. In FIG. 7, a snippet is selected from the extracted snippet group (S201).

Next, the words in the snippet are converted into a word distributed expression (S202). As an initialization process, an n-dimensional word distributed expression is constructed by randomly initializing each word with a uniform distribution.

Next, a convolution operation of the filter width h1 and the number of feature maps k1 is performed on the sequence of word distributed expressions (S203).

Next, the maximum value is extracted from the k1 feature maps (S204).

Next, a loss function between the k1 maximum features and the dictionary classification is calculated (S205).

Next, the loss calculated in S205 is back-propagated to the word convolution layer, and the weight of the word convolution layer of the first neural network is updated (S206).

Next, the loss calculated in S205 is back-propagated to the word distributed expression to update the word distributed expression of the first neural network (S207).

Next, it is determined whether all the extracted snippets have been processed (S208), and if all the snippets have not been processed (NO in S208), the next snippet is selected and the same processing is repeated (S209).

When all snippets have been processed (YES in S208), the updated word distributed expression of the first neural network and the weight of the word convolution layer are stored in the RAM 24 or the storage device 30, and the pre-learning ends (S210).

For example, if the word distribution expression is three-dimensional, "happy"=[1.1, 0.1, -0.2]
"Sad" = [0.2, 0.3, 1.5]
"Get" = [0.7, 0.2, -0.1]
Suppose that

In the calculation of the convolutional layer, the i-th result ci is expressed by the following equation with x as a word string.
ci=f(w·x _i:i+h−1 )
w ∈ R ^hk

Here, h is a convolution window width, k is the number of feature maps, w is a weight (weight matrix), and x _{i: i+h−1} is a partial word string from the i th to the i+h−1 th.

The weight matrix w of the word convolution process with the window width h=1 is randomly initialized. For example,
w=[0.1, -0.1, -0.2]
Is initialized to. Note that if h>1, processing is performed for a combination of dimensions of word distributed expressions for two or more words.

When the word learning process is performed and the first dimension of the word distributed expression has positive correlation and the third dimension has negative correlation, the weights of the first and third dimensions of the weight matrix w are updated by learning. .. For example, for the above initial values,
w=[2.0, -0.1, -1.8]
Will be updated.

Since the weight matrix w updated as described above has learned the relationship between each dimension of the word distribution expression and the positive/negative, compared with the case where the sentence/document classification learning unit 12 separately initializes at random. This improves performance and shortens learning time.

After the pre-learning is completed, the processing of FIG. 8 is executed using the weight matrix w obtained by the pre-learning. In other words, the weight matrix w obtained by pre-learning with the first neural network is given as the initial value of the weight matrix w of the second neural network.

In FIG. 8, weights are first initialized (S301). The weights stored in the RAM 24 or the storage device 30 are used to initialize the word distribution expression and the word convolution layer weights.

Next, the labeled text is selected from the text/document labeled text database 121 (S302).

Next, a word is extracted from the labeled text (S303), and the word in the snippet is converted into a word distributed expression (S304).

Next, the convolutional operation with the filter width h1 and the characteristic map number k1 is performed on the column of the word distributed expression (S305), and the convolutional operation with the filter width h2 and the characteristic map number k2 is further performed on the word convolution result (S306). ). The convolution operation of the filter width h2 and the number of feature maps k2 corresponds to the sentence/document convolution.

Next, the maximum value is extracted from the k2 feature maps (S307), and the loss function between the k2 maximum features and the label is calculated (S308).

Next, the calculated loss is back-propagated to the sentence/document convolution layer, and the weight of the sentence/document convolution layer of the second neural network is updated (S309).

Next, the calculated loss is back-propagated to the word convolution layer, and the weight of the word convolution layer of the second neural network is updated (S310).

Next, the calculated loss is back-propagated to the word distributed expression to update the word distributed expression of the second neural network (S311).

Next, it is determined whether or not all the labeled texts have been processed (S312), and if all the labeled texts have not been processed (NO in S312), the next labeled text is selected and the same processing is repeated. (S313).

By the above processing, the word distributed expression, the word convolutional layer weight, and the sentence/document convolutional layer weight are updated using the labeled text database 121 as teacher data.

FIG. 9 shows an example of input and output of the learned text analysis device. If you input any text to the learned text analysis device, for example, "I am very happy to buy a new smartphone!", a "positive" label will be output and classified. In this case, the text parser may act as a sentence/document reputation classifier.

As described above, in the present embodiment, the word (unlabeled) database 101 and the classification information-added dictionary database 102 are used to previously learn word-by-word classification information by the first neural network, and words obtained by this pre-learning Since the second neural network processes the sentence/document units by using the unit classification information as a partial structure, it is possible to improve the accuracy of the sentence/document unit processing as compared with the case where the word unit classification information is not used. You can

Further, in the present embodiment, since the word-based classification information is learned and acquired in advance by the first neural network, the syntax tree corpus to which the word-based classification information is added is unnecessary, and the corpus is constructed. Cost is eliminated.

Although the embodiment of the present invention has been described above, the present invention is not limited to this, and various modifications can be made. These modifications will be described below.

<Modification 1>
In the embodiment, the text (unlabeled) database 101, the classification information-added dictionary database 102, and the sentence/document-labeled text database 121 are stored in the storage device 30, but they may be stored in different storage devices. Of course, they may be stored in different server computers on the network.

Further, in the present embodiment, the CPU 20 functions as the word extracting means 103, the snippet extracting means 104, the word distributed expression converting means 105, the word convolutional means 106, the maximum feature extracting means 107, and the dictionary classification associating means 108 in FIG. , The word extraction unit 122, the word distributed expression conversion unit 123, the word convolution unit 124, the sentence/document convolution unit 125, the maximum feature extraction unit 126, and the sentence/document label association unit 127 have been described. The module may be shared by a plurality of CPUs.

Further, the word extraction means 103 and the word extraction means 122, the word distributed expression conversion means 105 and the word distributed expression conversion means 123, the word convolution means 106 and the word convolution means 124, the maximum feature extraction means 107 and the maximum feature extraction means 126 are the same. Or, since they have similar functions, they may be commonly used.

<Modification 2>
In the embodiment, L words around the entry word are extracted as snippets and L=2, but L=1, L=3, L=4, or the like may be used. In short, one or more words before and after the entry word stored in the dictionary database with classification information 102 may be extracted to extract the snippet. Generally, it is considered that L, 2, 3, 4,... Improves the analysis accuracy than 1, but if the number of snippets increases, the processing becomes complicated and time is required. Therefore, it is considered that about two words before and after the headword are appropriate, but not limited to this. You may learn about various L and optimize L from a viewpoint of accuracy and processing time.

<Modification 3>
Although the analysis target of the embodiment is a text of a sentence/document, it is also possible to input voice and convert it into text, and set the text as the analysis target. In this case, the text analysis device can function as a text-to-speech analysis device.

<Modification 4>
In the embodiment, both the word classification learning unit 10 and the sentence/document classification learning unit 12 are installed in one computer, but these may be installed in different computers.

FIG. 10 shows the system configuration in this case. A computer that implements the word classification learning unit 10 and a computer that implements the sentence/document classification learning unit 12 are connected to the network 100. A text (no label) database 101, a classification information-added dictionary database 102, and a text/document labeled text database 121 are also connected to the network 100. The word classification learning unit 10 includes a word extraction unit 103, a snippet extraction unit 104, a word distributed expression conversion unit 105, a word convolution unit 106, a maximum feature extraction unit 107, and a dictionary classification association unit 108. The sentence/document classification learning unit 12 includes a word extraction unit 122, a word distributed expression conversion unit 123, a word convolution unit 124, a sentence/document convolution unit 125, a maximum feature extraction unit 126, and a sentence/document label association unit 127.

The word classification learning unit 10 supplies the updated word distributed expression and the weight of the word convolution layer obtained by the pre-learning to the sentence/document classification learning unit 12 via the network 100. The sentence/document classification learning unit 12 initializes using the supplied word distributed expression and the weight of the word convolution layer, and learns using the sentence/document labeled text database 121.

The word classification learning unit 10 may store the updated word distributed expression and the weight of the word convolutional layer obtained by pre-learning in its own storage device, or a storage device on the network, for example, a sentence/document label. It may be stored in the attached text database 121. In this case, the sentence/document classification learning unit 12 reads the updated word distributed expression and the weight of the word convolution layer that have been pre-learned together with the text from the text database 121 with the sentence/document label via the network 100.

The computer in which the word classification learning unit 10 is implemented executes the preprocessing for the sentence/document classification learning process executed by the sentence/document classification learning unit 12 and the text analysis process after the learning. In a sense, the computer can also function as a text analysis process.

<Modification 5>
In the embodiment, the neural network (first neural network and second neural network) is implemented by the CPU 20 and the processing program, but instead of this, the neural network may be implemented by dedicated hardware (ASIC). Alternatively, it may be mounted using an FPGA. Even when the FPGA is used, a part of it may be implemented by software, or may be implemented as a complex of hardware and software.

10 word classification learning unit, 12 sentence/document classification learning unit, 101 text (no label) database, 102 classification information database, 103 word extraction means, 104 snippet extraction means, 105 word distributed expression conversion means, 106 word convolution means, 107 maximum feature extraction means, 108 dictionary classification correspondence means, 121 sentence/document labeled text database, 122 word extraction means, 123 word distributed expression conversion means, 124 word convolution means, 125 sentences/document convolution means, 126 maximum feature extraction Means, 127 sentence/document label associating means.

Claims (2)

From the text of unlabeled, positive, snippets extracted negation, either neutral, or one or more words of headword around with classification information containing any label emotions as snippets extracted Means and
It is an acquisition means for pre- learning with the first neural network using the words in the extracted snippet to acquire the classification information in word units,
Convert the words in the snippet into a word distribution representation,
Performing a convolution operation on the sequence of word distribution expressions,
An updating unit that updates the weights of the word distributed expression and the word convolutional layer by an error backpropagation method and acquires the classification information on a word-by-word basis;
The weighting of the second neural network is initialized by using the classification information of the word unit acquired by the pre-learning by the acquiring unit, and the labeled text is used as the teacher data to learn by the second neural network, A learning means for learning to output the classification of the text when the text is input,
Initialize the word distribution expression and the weight of the word convolutional layer using the classification information for each word,
Extract words from the labeled text as snippets,
Convert the words in the extracted snippet into a word distribution expression,
Performing a convolution operation on the sequence of word distribution expressions,
An error backpropagation method is used to update the initialized word distribution expression and the weight of the word convolutional layer, and learning means,
A text analysis device equipped with . Computer,
Snippet extraction that extracts one or more words around a headword with classification information that includes any of positive, negative, neutral, or emotional label from unlabeled text as a snippet Means and
It is an acquisition means for pre-learning with the first neural network using the words in the extracted snippet to acquire the classification information in word units,
Convert the words in the snippet into a word distribution representation,
Performing a convolution operation on the sequence of word distribution expressions,
An updating unit that updates the weights of the word distributed expression and the word convolutional layer by an error backpropagation method and acquires the classification information on a word-by-word basis;
The weighting of the second neural network is initialized by using the classification information of the word unit acquired by the pre-learning by the acquiring unit, and the labeled text is used as the teacher data to learn by the second neural network, A learning means for learning to output the classification of the text when the text is input,
Initialize the word distribution expression and the weight of the word convolutional layer using the classification information for each word,
Extract words from the labeled text as snippets,
Convert the words in the extracted snippet into a word distribution expression,
Performing a convolution operation on the sequence of word distribution expressions,
An error backpropagation method is used to update the initialized word distribution expression and weight of the word convolution layer, learning means,
A program to function as.

Images