JP7720766B2 - Machine learning device, natural language processing device, and program - Google Patents

Description

The present invention relates to a machine learning device, a natural language processing device, and a program.

For example, it would be effective to automatically analyze the trends in opinions of people or society as a whole by automatically analyzing the large number of sentences posted on social networking services (SNS). However, it is unrealistic to manually analyze tens of millions to hundreds of millions of sentences or more, and there is a strong demand for accurate automatic analysis.

Conventional technology automatically labels each phrase or word contained in a sentence. For example, attempts have been made to use deep learning models to extract sections of opinion objects and opinion parts from within a sentence.

Non-Patent Document 1 describes BERT (Bidirectional Encoder Representations from Transformers), a mechanism for analyzing text written in natural language using a deep learning model.

Non-Patent Document 2 describes a "sequence labeling" technique that uses machine learning to label sequences of sentence components.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova, BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, Proceedings of NAACL-HLT 2019, pages 4171-4186, Association for Computational Linguistics, 2019. Takatomo Ishikawa, Machine Learning for Natural Language Processing "5. Sequence Labeling", [online], Internet <URL: https://www.slideshare.net/Takatymo/ss-64274683, July 22, 2016>.

However, when labeling sentence components using a deep learning analysis model, conventional technology has the problem of not always correctly segmenting sentences into phrases to be labeled. In other words, there is a problem with errors in segmentation into phrases, which is a process that precedes labeling. In the task of extracting opinion target sections or opinion sections within a sentence, errors can occur in the breaks between opinion target sections or opinion sections. Such errors in section extraction can also result in incorrect labeling. In other words, in the task of extracting opinion target sections or opinion sections, some expressions that should be included in those sections may be missing, or incorrect expressions that should not be included in those sections may be extracted as being included in those sections. Furthermore, errors such as those described above in labeling can also lead to problems such as poor analysis accuracy in subsequent processes. For example, this can also lead to poor accuracy when statistically aggregating opinion trends on social media.

The present invention was developed based on the above-mentioned problem recognition, and aims to provide a machine learning device, natural language processing device, and program that enable more accurate extraction of the location of the subsequence to be analyzed contained in a word sequence (specific examples of this subsequence are the above-mentioned opinion target section and opinion section).

[1] In order to solve the above problem, a machine learning device according to one aspect of the present invention includes a word embedding unit that inputs a word string and outputs a word embedding sequence corresponding to the word string, a sequence labeling unit that inputs the word embedding sequence output from the word embedding unit and outputs a label sequence corresponding to the word embedding sequence, a phrase break position prediction unit that inputs the word embedding sequence output from the word embedding unit and outputs phrase break position information corresponding to the word embedding sequence, and supplies a word string to be input to the word embedding unit and a correct label sequence corresponding to the label sequence output by the sequence labeling unit, and a learning data supply unit that supplies correct phrase break position information corresponding to the phrase break position information output by the phrase break position prediction unit, wherein the sequence labeling unit adjusts the parameters of an internal model by performing backpropagation based on the error between the label sequence and the correct label sequence, the phrase break position prediction unit adjusts the parameters of the internal model by performing backpropagation based on the error between the phrase break position information and the correct phrase break position information, and the word embedding unit adjusts the parameters of the internal model by backpropagating the error from the sequence labeling unit and the error from the phrase break position prediction unit.

[2] In another aspect of the present invention, in the machine learning device described above, the phrase break position prediction unit outputs phrase break position information corresponding to the input word embedding sequence by using a fully connected regression model based on all word embeddings included in the input word embedding sequence.

[3] Another aspect of the present invention is that in the above-mentioned machine learning device, the sequence labeling unit outputs at least a label indicating that a predetermined subsequence included in the original input word sequence is an opinion subject, and a label indicating that the predetermined subsequence is an opinion.

[4] Another aspect of the present invention is that in the above-mentioned machine learning device, the phrase break position prediction unit outputs, as the phrase break position information, numerical values representing the start and end positions of the subsequence, and the correct phrase break position information supplied by the learning data supply unit is numerical information representing the correct start and end positions of the subsequence.

[5] Another aspect of the present invention is a natural language processing device that includes a word embedding unit that inputs a word string and outputs a word embedding sequence corresponding to the word string, and a sequence labeling unit that inputs the word embedding sequence output from the word embedding unit and outputs a label sequence corresponding to the word embedding sequence, wherein at least the model internal to the word embedding unit has been trained by the machine learning device described in any one of claims 1 to 4.

[6] Another aspect of the present invention is a word embedding unit that inputs a word string and outputs a word embedding sequence corresponding to the word string; a sequence labeling unit that inputs the word embedding sequence output from the word embedding unit and outputs a label sequence corresponding to the word embedding sequence; a phrase break position prediction unit that inputs the word embedding sequence output from the word embedding unit and outputs phrase break position information corresponding to the word embedding sequence; and a sequence labeling unit that supplies a word string to be input to the word embedding unit, and also supplies a correct label sequence corresponding to the label sequence output by the sequence labeling unit, and further supplies the phrase break position information output by the phrase break position prediction unit. and a learning data supply unit that supplies correct phrase break position information corresponding to the information, wherein the sequence labeling unit adjusts the parameters of an internal model by performing backpropagation based on the error between the label sequence and the correct label sequence, the phrase break position prediction unit adjusts the parameters of the internal model by performing backpropagation based on the error between the phrase break position information and the correct phrase break position information, and the word embedding unit adjusts the parameters of the internal model by backpropagating the error from the sequence labeling unit and the error from the phrase break position prediction unit.

This invention makes it possible to improve the accuracy of assigning label sequences (tag sequences) to word sequences.

1 is a block diagram showing a schematic functional configuration of an opinion analysis device according to an embodiment of the present invention. FIG. 2 is a diagram showing in more detail the configuration of a model for predicting a phrase break position in the embodiment. 2 is a schematic diagram showing an example of a text input to the opinion analysis device according to the embodiment and an example of labeling performed on the input text. FIG. 4 is a schematic diagram showing an example of a word sequence corresponding to an input text example (FIG. 3) in the embodiment and a correct answer of a tag sequence corresponding to the sequence. FIG. FIG. 4 is a schematic diagram showing another example of a word sequence corresponding to the input text example (FIG. 3) and a correct answer of a tag sequence corresponding to the word sequence in the embodiment. 10 is a schematic diagram showing an example of correct answer data for training a phrase break position prediction model in the embodiment. FIG. 10 is a flowchart showing a processing procedure when the opinion analysis device according to the embodiment learns a model. 10 is a flowchart showing a procedure when the opinion analysis device according to the embodiment performs a sequence labeling process using a trained model. FIG. 2 is a block diagram showing an example of the internal configuration of the opinion analysis device according to the embodiment.

Next, one embodiment of the present invention will be described with reference to the drawings. The device according to this embodiment is an opinion analysis device for analyzing opinions contained in input text. Specifically, the opinion analysis device of this embodiment assigns tags (labels) to elements (phrases) that make up the input text to distinguish them as opinion target sections, opinion sections, or sections that are neither.

The opinion analysis device of this embodiment is characterized by the inclusion of a regression model for predicting the division points (phrase breaks) of the extracted section. This regression model specific to this embodiment may be referred to below as the "phrase break position prediction model." The phrase break position prediction model, together with a model for performing sequence labeling (sequence labeling model), shares a distributed representation in a lower layer. The opinion analysis device of this embodiment performs machine learning on an internally stored model and is also referred to as a "machine learning device." The opinion analysis device of this embodiment also uses a trained model to provide a label sequence corresponding to a given unknown word sequence and is also referred to as a "natural language processing device."

In this embodiment, in order to perform machine learning on the phrase break position prediction model, a loss function is used that gives the error between the phrase break position predicted by the phrase break position prediction model and the correct phrase break position in terms of cross-entropy.

Figure 1 is a block diagram showing the general functional configuration of an opinion analysis device according to this embodiment. As shown in the figure, the opinion analysis device 1 includes a word embedding unit 11, a sequence labeling unit 12, a sequence labeling loss function calculation unit 17, a phrase break position prediction unit 22, a phrase break position loss function calculation unit 27, and a training data supply unit 30. Each of these functional units can be implemented, for example, by a computer and a program. Each functional unit also includes storage means as needed. Storage means is, for example, program variables or memory allocated by program execution. Non-volatile storage means such as a magnetic hard disk drive or solid-state drive (SSD) may also be used as needed. At least some of the functions of each functional unit may also be implemented as dedicated electronic circuits rather than programs. The functions of each unit are described below.

The word embedding unit 11 receives a word string as input and outputs a word embedding representation string corresponding to the word string. The word embedding unit 11 includes an internal model capable of machine learning. The model included in the word embedding unit 11 is also called a "word embedding layer." The model included in the word embedding unit 11 is implemented, for example, using a neural network. In this embodiment, BERT is used as the deep learning model for the opinion target extraction device. BERT itself is an existing technology and is described in the aforementioned Non-Patent Document 1, etc. The BERT used by the word embedding unit 11 may be a general model pre-trained in Japanese. In this embodiment, the BERT held by the word embedding unit 11 can be further trained. The input sentence is divided into words using standard morphological analysis processing. In other words, the input sentence is equivalent to a word string. The word string corresponding to the input sentence is input to the word embedding unit 11 and converted into a word embedding representation string.

The sequence labeling unit 12 inputs the word embedding sequence output by the word embedding unit 11 and outputs a label sequence corresponding to the word embedding sequence. In other words, the sequence labeling unit 12 assigns a label to each word. Specific examples of labels will be explained later. The sequence labeling unit 12 internally includes a model capable of machine learning. The model included in the sequence labeling unit 12 is called a "sequence labeling model."

In this embodiment, the sequence labeling unit 12 outputs at least a label indicating that a specific subsequence included in the original input word sequence is an opinion subject, and a label indicating that the specific subsequence is an opinion. Specific examples of the labels assigned by the sequence labeling unit 12 will be described later.

The sequence labeling loss function calculation unit 17 calculates the error between the label sequence output by the sequence labeling unit 12 and the correct label sequence. The error calculated by the sequence labeling loss function calculation unit 17 is the basis for the error backpropagation performed by the sequence labeling unit 12 to adjust the parameters.

The phrase break position prediction unit 22 inputs the word embedding sequence output from the word embedding unit 11 and outputs phrase break position information corresponding to the word embedding sequence. The phrase break position prediction unit 22 has an internal model capable of machine learning. The model provided by the phrase break position prediction unit 22 is called the "phrase break position prediction model."

The phrase break position prediction unit 22 may output phrase break position information corresponding to the input word embedding sequence by using a fully connected regression model based on all word embeddings contained in the input word embedding sequence.

The phrase break position loss function calculation unit 27 calculates the error between the phrase break position information output by the phrase break position prediction unit 22 and the correct phrase break position information. The error calculated by the phrase break position loss function calculation unit 27 is the basis for the error backpropagation performed by the phrase break position prediction unit 22 to adjust the parameters.

When performing machine learning of the model, the word embedding unit 11 adjusts the parameters of the internal model based on both the backpropagation of errors from the sequence labeling unit 12 and the backpropagation of errors from the phrase break position prediction unit 22.

The learning data supply unit 30 supplies learning data for training the model stored internally in the opinion analysis device 1. Specifically, the learning data supply unit 30 supplies word sequences to be input to the word embedding unit 11. The learning data supply unit 30 also supplies correct label sequences that correspond to the label sequences output by the sequence labeling unit 12 in correspondence with the above word sequences. Furthermore, the learning data supply unit 30 supplies correct phrase break position information that corresponds to the phrase break position information output by the phrase break position prediction unit 22 in correspondence with the above word sequences.

The phrase break position prediction unit 22 outputs, as phrase break position information, numerical values representing the start and end positions of subsequences of a word string corresponding to the input sentence. The number of subsequences may be one or more. The correct phrase break position information supplied by the training data supply unit 30 is numerical information representing the correct start and end positions of the above subsequence. The start and end positions are numerical information representing the ordinal number of the word. The correct start and end positions of a subsequence are basically integers. The start and end positions predicted and output by the phrase break position prediction unit 22 are not necessarily integers. Typically, the start and end positions predicted and output by the phrase break position prediction unit 22 are non-integer values. These non-integer values can be considered to be predicted approximations of the start and end positions, respectively.

FIG. 2 is a diagram illustrating in more detail the configuration of a model for predicting phrase break positions. As shown in the figure, the model for predicting phrase break positions includes a fully connected recurrent layer 220. The fully connected recurrent layer 220 is a model included in the phrase break position prediction unit 22 shown in FIG. 1. The fully connected recurrent layer 220 includes two nodes ( _y1 and _y2 in the figure) connected to all outputs from the word embedding layer 110 (BERT embedding layer). The word embedding layer 110 is a model included in the word embedding unit 11 shown in FIG. 1. The nodes included in the word embedding layer 110 ( _x1 , _x2 , ..., _xD in the figure) correspond to embedded representations of words included in the input text. _y1 and _y2 are numerical values representing the start position (start_position) and end position (end_position) of a phrase break in a section, respectively. These numerical values represent the ordinal number of each word in the text. In other words, the model for predicting phrase break positions predicts the start and end positions of one or more sections. In other words, the fully connected recurrent layer 220 is a two-dimensional regression model for predicting these two quantities.

That is, the word embedding layer 110 outputs D-dimensional continuous values _x1 , _x2 , ..., _xD . The fully connected recurrent layer 220 outputs y1 and _y2 calculated based on the above _x1 , _x2 , ..., _xD . _y1 is the _start position and _y2 is the end position, and _y1 and _y2 are as shown in the following equation (1).

In equation (1), f is an activation function. Furthermore, _aij and _bj are internal parameters of the model of the fully connected recurrent layer 220. The values of these internal parameters are adjusted by learning. Note that j is either 1 or 2.

That is, the start position _y1 and the end position _y2 are calculated by applying an activation function to the weighted sum of the output values _x1 , _x2 , ..., _xD from the word embedding layer 110 (to which a predetermined parameter value _bj is added).

The outputs _y1 and _y2 from the fully connected regression layer 220 are compared with the intervals (correct answers) included in the training data. Specifically, the opinion analysis device 1 learns the internal parameters of the model using a criterion that minimizes the cross-entropy between the output from the fully connected regression layer 220 and the label intervals that appear in the training data. When calculating the criterion for the label intervals, all label intervals that appear in the training data are included in the criterion, without narrowing it down to one, except when pruning is performed for computational efficiency.

When learning the model, a loss function L _span (S) expressed by the following equation (2) is used.

In formula (2), SpanSet(S) is a set of annotation intervals that appear in sentence S, which is input text to the opinion analysis device 1. The start position of one interval l belonging to the set SpanSet(S) is Begin(l) in formula (2), and the end position is End(l). In other words, one interval (span) is defined by the value of the start position and the value of the end position. CrossEntropy(y, x) in formula (2) is the cross entropy between y (i.e., _y1 or _y2 ), which is one of the outputs from the fully connected recurrent layer 220, and the correct answer for the start position or end position given by the training data.

That is, L _span (S) is based on the sum, for all annotation sections, of the cross entropy between y ₁ and the correct answer for the start position for each annotation section and the cross entropy between y ₂ and the correct answer for the end position.

CrossEntropy(y,x) is calculated using equation (3) below.

In equation (3), k is an index representing the position within sentence S. Also, max_seq_length is a value corresponding to the sequence length of sentence S. Also, exp is an exponential function.

The opinion analysis device 1 uses a loss function _Lseq (S) for the model held by the sequence labeling unit 12 (FIG. 1) in addition to the loss function _Lspan (S) for the above-mentioned phrase break position prediction model. The loss function _Lseq (S) is a loss function for the sequence labeling problem according to conventional technology. The overall loss function _Ltotal (S) of the model when one sentence (S) is input to the opinion analysis device 1 is calculated as shown in the following formula (4):

That is, the overall loss function L _total (S) is calculated as the sum of the loss function L _span (S) for the phrase break position prediction model and the loss function L _seq (S) according to the prior art for the model held by the sequence labeling unit 12 .

Of the model held by the sequence labeling unit 12 and the model held by the phrase break position prediction unit 22, the model held by the phrase break position prediction unit 22 is used only during training, not during inference. In other words, as training progresses so that the value of the overall loss function (equation (4) above) decreases, the learning accuracy of the distributed representations in the lower layer (the word embedding layer 110 of the word embedding unit 11) improves. This improves the accuracy of labeling (the task of identifying opinion target sections and opinion sections) using the model held by the sequence labeling unit 12.

Figure 3 is a schematic diagram showing an example of text input to the opinion analysis device 1 and an example of labeling performed on the input text. In other words, Figure 3 is a schematic diagram showing an example of the process of extracting opinion target sections and opinion sections from the input text.

The example input text shown in the figure is a single sentence posted to a social networking site. This text is "Violence is bad, it's bad #anime." Note that the "#anime" in this text is an expression that follows the notation of tags (called hashtags because they use the "#"), which make it easier to search for specific topics. This input text is divided into eight words: "Violence/is/bad/,/bad/yo/#/anime" (word separators are used). Here, punctuation marks and symbols such as "#" are treated as words for convenience. The opinion target sections extracted from this input text are "Violence" (start position 1, end position 1) and "#anime" (start position 7, end position 8). The opinion sections extracted are "Bad" (start position 3, end position 3) and "Bad" (start position 5, end position 5).

Figure 4 is a schematic diagram showing an example of a word sequence corresponding to the example input text shown in Figure 3 and a correct tag sequence (label sequence) corresponding to that sequence. The example word sequence is "violence/is/not/,/not/yo/#/anime." In this diagram, numbers (position numbers) are assigned according to the order of the words. The correct tag sequence corresponding to such a word sequence is "TARGET/O/OPINION/O/OPINION/O/TARGET/TARGET." The tag "TARGET" indicates that the corresponding word belongs to the opinion target section. The tag "OPINION" indicates that the corresponding word belongs to the opinion section. The tag "O" indicates that the corresponding word belongs to neither the opinion target section nor the opinion section.

In other words, this correct tag sequence indicates that the expression "violence" (the section starts at 1 and ends at 1) and the expression "#anime" (the section starts at 7 and ends at 8) in the word sequence are each the opinion target section. Furthermore, the expressions "Ikenai" (the section starts at 3 and ends at 3) and "Ikenai" (the section starts at 5 and ends at 5) in the word sequence are each opinion sections.

Figure 5 is a schematic diagram showing another example of a word sequence corresponding to the example input text shown in Figure 3 and the corresponding correct tag sequence. Unlike the tags in Figure 4, the example in Figure 5 adds a prefix "B-" or "I-" to the "TARGET" and "OPINION" tags, respectively. Such tags are called BIO-format tags. "B-" indicates the beginning of a section, while "I-" indicates the middle of a section. The tag assigned to the word at the end of a section also begins with "I-". In the example shown in Figure 5, the correct tag sequence corresponding to the word sequence "violence/wa/gainai/,/gainai/yo/#/anime" is "B-TARGET/O/B-OPINION/O/B-OPINION/O/B-TARGET/I-TARGET."

In other words, this correct tag sequence indicates that the expression "violence" (the section starts at 1 and ends at 1) and the expression "#anime" (the section starts at 7 and ends at 8) in the word sequence are each the opinion target section. Also, the expressions "kenai" (the section starts at 3 and ends at 3) and "kenai" (the section starts at 5 and ends at 5) in the word sequence are each opinion sections. In BIO format tags, adding "B-" or "I-" explicitly indicates that, for example, the seventh and eighth words in Figure 5 belong to the same section.

Figure 6 is a schematic diagram showing an example of correct answer data for training a phrase break position prediction model. As shown, this correct answer data is provided as a set of pairs of section start and end positions. In the example shown, the correct answer data includes data for four sections. These sections are: start position 1 and end position 1, start position 3 and end position 3, start position 5 and end position 5, and start position 7 and end position 8. This correct answer data corresponds to the examples shown in Figures 4 and 5. The data shown in Figure 6 represents the correct answers for the start and end positions also shown in Figure 2.

Figure 7 is a flowchart showing the processing steps performed by the opinion analysis device 1 when learning a model. The model learning steps will be explained below in accordance with this flowchart.

In step S11, the training data supply unit 30 supplies training data. The training data is a pair of input text (word sequence) and correct answer data corresponding to the input data. The correct answer data includes the correct answer for the tag sequence and the position of the section (start position and end position). The word embedding unit 11 reads the input text (word sequence). The sequence labeling loss function calculation unit 17 also reads the correct answer tag sequence. The phrase break position loss function calculation unit 27 then reads the correct answer data for the start position and end position of the section (e.g., Figure 6).

In step S12, the opinion analysis device 1 adjusts the internal parameters of each model based on the calculated loss function value. Specifically, this is as follows:

The word embedding unit 11 passes a sequence of embedded representations corresponding to the input word sequence to the sequence labeling unit 12 and the phrase break position prediction unit 22. The sequence labeling unit 12 calculates sequence labels using a sequence labeling model with internal parameter values at that time.
The phrase break position prediction unit 22 determines the phrase break position using a phrase break position prediction model based on the internal parameter values at that time. The sequence labeling loss function calculation unit 17 calculates a loss function value based on the sequence label output by the sequence labeling unit 12 and the correct sequence label. The phrase break position loss function calculation unit 27 calculates a loss function value based on the phrase break position output by the phrase break position prediction unit 22 and the correct phrase break position (start position and end position). Based on these calculated loss function values, the internal parameters of each model are adjusted using backpropagation. Repeated adjustment of the internal parameters of such models acts to reduce the loss function value. In other words, repeated parameter adjustment enables each model to calculate an output value closer to the correct answer. Note that the method of adjusting model parameters using backpropagation itself belongs to existing technology.

In step S13, the opinion analysis device 1 saves the state of each model after the parameters have been adjusted. Specifically, the opinion analysis device 1 writes the adjusted parameter values to non-volatile memory or the like.

Figure 8 is a flowchart showing the steps taken by the opinion analysis device 1 when performing sequence labeling processing using a trained model. Because the model state was saved in step S13 of the processing procedure shown in Figure 7, the opinion analysis device 1 can read the state of each model. Below, the processing steps for estimating sequence labels will be explained using this flowchart.

In step S21, the word embedding unit 11 reads the input text (word sequence).

In step S22, the opinion analysis device 1 uses the model to infer a tag sequence corresponding to the input text. Specifically, the word embedding unit 11, which has a word embedding model, outputs a sequence of vectors calculated using the word embedding model. The sequence labeling unit 12, which has a sequence labeling model, determines a tag sequence based on the sequence of vectors passed from the word embedding unit 11.

In step S23, the sequence labeling unit 12 outputs the tag sequence determined in step S22. Examples of tag sequences corresponding to the input text (word string) are shown in Figures 4 and 5.

As described above, by using the opinion analysis device of this embodiment, the accuracy of tags assigned can be improved in the process of automatically assigning a tag sequence corresponding to an input word sequence. As a specific example, it becomes possible to accurately assign a tag indicating that the word represents an opinion subject or a tag indicating that the word represents an opinion to each word in the input word sequence. In other words, it becomes possible to accurately extract parts of the input word sequence that have specific properties (for example, an opinion subject or an opinion).

Such a device according to this embodiment can be used to extract portions of a particular nature (e.g., portions that are the subject of opinion or portions that contain opinions) from text posted on SNS. For example, when automatically analyzing reactions to viewing specific content (such as a broadcast program) based on the content posted on SNS, these portions can be automatically and accurately extracted even if the text of the posted content does not explicitly state which portions are the subject of opinion and which portions are opinions. Alternatively, even if the text of the posted content does not contain hashtags or the like that indicate which content the reaction is to, it is possible to automatically extract portions that are the subject of opinion or portions that contain opinions. In other words, the device according to this embodiment enables more powerful and accurate analysis than methods such as keyword search. In other words, this embodiment makes it possible to easily and accurately conduct large-scale market research, etc.

If it were possible to automatically extract opinion subjects and opinions from large amounts of text (for example, several thousand to several hundred thousand items) posted on social media, etc., it would be possible to automatically tally what kind of opinions were expressed and to what extent for each opinion subject. When tallying, it would be possible to tally the same opinion subject or perform clustering processing to group together similar opinion subjects. Conventional technology methods can be used for tallying and clustering. For example, it would be possible to automatically tally viewer opinions on a broadcast program.

Figure 9 is a block diagram showing an example of the internal configuration of the opinion analysis device 1 of the above embodiment. The opinion analysis device 1 can be realized using a computer. As shown in the figure, the computer includes a central processing unit 901, RAM 902, input/output port 903, input/output devices 904 and 905, etc., and bus 906. The computer itself can be realized using existing technology. The central processing unit 901 executes instructions contained in a program read from RAM 902, etc. In accordance with each instruction, the central processing unit 901 writes data to RAM 902, reads data from RAM 902, and performs arithmetic and logical operations. RAM 902 stores data and programs. Each element contained in RAM 902 has an address and can be accessed using the address. Note that RAM is an abbreviation for "random access memory." The input/output port 903 is a port through which the central processing unit 901 exchanges data with external input/output devices, etc. The input/output devices 904 and 905 are input/output devices. Input/output devices 904 and 905 exchange data with the central processing unit 901 via the input/output port 903. The bus 906 is a common communication path used within the computer. For example, the central processing unit 901 reads and writes data from RAM 902 via the bus 906. Also, for example, the central processing unit 901 accesses the input/output port via the bus 906.

At least some of the functions of the opinion analysis device 1 described above can be implemented by a computer. In this case, a program for implementing these functions may be recorded on a computer-readable recording medium, and the program may be loaded into a computer system and executed. Note that the term "computer system" here includes hardware such as an OS and peripheral devices. Furthermore, "computer-readable recording medium" refers to portable media such as flexible disks, optical magnetic disks, ROMs, CD-ROMs, DVD-ROMs, and USB memory, as well as storage devices such as hard disks built into computer systems. In other words, a "computer-readable recording medium" may be a non-transitory computer-readable recording medium. Furthermore, "computer-readable recording medium" may also include media that temporarily and dynamically store programs, such as communication lines used when transmitting programs over networks like the Internet or telephone lines, or media that store programs for a certain period of time, such as volatile memory within the computer systems that serve as servers or clients in such cases. The program may also be a program that implements some of the functions described above, or it may be a program that can be implemented in combination with a program already stored in the computer system.

Although several embodiments have been described above, the present invention can also be implemented in the following modified forms.

[First Modification]
In the above explanation, the text to be processed is written in Japanese, but this embodiment may also be applied to data other than Japanese. In the demonstration experiment described later, the text to be processed is in English as well as Japanese. Furthermore, text in other languages may also be processed.

[Second Modification]
In the above embodiment, one opinion analysis device 1 both trains a model and performs a process of assigning (estimating) a label sequence using the trained model. As a variation, the opinion analysis device 1 may perform only one of model training and label sequence estimation using the trained model. When the opinion analysis device 1 functions as a machine learning device, the parameters of the trained model can be transplanted to another device, and the destination device can perform a process of assigning (estimating) a label sequence. Alternatively, the opinion analysis device 1 may not perform machine learning itself, but may instead acquire parameters of a trained model and perform a process of assigning (estimating) a label sequence.

In the latter case, the opinion analysis device 1 functions as a natural language processing device that includes a word embedding unit that inputs a word string and outputs a word embedding sequence corresponding to the word string, and a sequence labeling unit that inputs the word embedding sequence output from the word embedding unit and outputs a label sequence corresponding to the word embedding sequence. In this case, at least the model that the word embedding unit 11 internally has is one that has been trained using the machine learning mechanism (machine learning device) described in the above embodiment.

Embodiments of the present invention and their variations have been described in detail above with reference to the drawings, but the specific configurations are not limited to these embodiments and include designs that do not deviate from the gist of the present invention.

[Implementation and Evaluation]
The opinion analysis device 1 was implemented and evaluated through an experiment using actual posts on SNS (Twitter) and movie reviews. The evaluation results of the experiment will be described below.

The data used for the evaluation consisted of Japanese tweets about television programs and English reviews of movies. Specifically, the Japanese text used was Japanese tweets from Twitter about the television program (NHK morning drama) "Natsuzora" over a nine-day period (24,610 tweets, 984,861 words). The English text used was reviews from the review site IMDb (986 reviews, 10,360 sentences, 321,807 words). Correct answers were manually annotated to the training data.

The evaluation measures used were precision, recall, and f1. These are defined by equations (5), (6), and (7), respectively. Here, TP stands for true positive, FP for false positive, and FN for false negative.

Table 1 shows the experimental results when limiting labels to the top level (first hierarchical level) for the task of analyzing Japanese text (tweets) related to broadcast programs.

In this experiment, labels were organized into three layers, with two main types at the first layer. These two types are REFERENCE labels, which are assigned to parts that refer to the content of the broadcast program, and OPINION labels, which are assigned to parts that express the sender's subjective opinions. REFERENCE labels have the following labels at the second layer: TITLE, MUSIC, PERSON, PROGRAM, SCENE, STORY, and QUOTE. On the other hand, OPINION labels have the following labels at the second layer: EVALUATION, ACTION, and INDEX. Of these, ECALUATION at the second layer has the following labels at the third layer: POSITIVE, NEGATIVE, NEUTRAL, and REQUEST.

In the experiment, the first-layer REFERENCE and OPINION labels were evaluated individually, and also as a whole. The prior art used for comparison does not use the phrase break position prediction unit 22 (see Figure 1) even during training.

Except for the REFERENCE label in the evaluation metric recall, where the results of this embodiment were slightly lower than those of the prior art, it was confirmed that in all other cases the results of this embodiment outperformed those of the prior art.

The effect of using this embodiment varies depending on the number of epochs used when training the model. In the early stages of training (less than about 10 epochs), there is no significant difference in performance evaluated by the f1 value between the conventional technology and this embodiment. However, once the number of epochs exceeds 10, the effect of this embodiment begins to be seen, and once the number of epochs reaches about 20, the superiority of this embodiment (the difference in performance in the f1 value compared to the conventional technology) becomes apparent.

Table 2 shows the experimental results when the labels were limited to the top first level for the task of analyzing movie review text written in English. As with Table 1, in Table 2 we evaluated the REFERENCE and OPINION labels in the first level individually, as well as the overall evaluation.

This analysis of English text also confirmed that this embodiment outperformed the results of the prior art in most cases. The only time the evaluation results of this embodiment fell short of those of the prior art was for the REFERENCE label in the recall evaluation scale. In all other cases, the evaluation results of this embodiment outperformed those of the prior art.

Table 3 below shows the significance level (p-value) indicating whether the improvement in this embodiment is significant compared to the conventional technology. The significance level shown here is the result of a McNemar test.

The experimental results for the task of analyzing Japanese text (tweets) related to broadcast programs, without restricting the label hierarchy, are shown in Table 4 below. The values shown in Table 4 are the overall evaluation results (micro-average) for all labels.

As shown in Tables 1, 2, and 4 above, experiments have confirmed that the accuracy of extracting expressions with specific positions within text (extracting opinion targets) using this embodiment is better than that of conventional technology. Furthermore, as shown in Table 3, testing has confirmed that the improvement in accuracy is significant.

The present invention can be used, for example, to extract elements of a specific position from sentences written in natural language. As an example, it can be used to extract elements of a specific position from text posted on social media. However, the scope of use of the present invention is not limited to the examples given here.

1. Opinion analysis device (machine learning device, natural language processing device)
11 Word embedding unit 12 Sequence labeling unit 17 Sequence labeling loss function calculation unit 22 Phrase break position prediction unit 27 Phrase break position loss function calculation unit 30 Learning data supply unit 220 Fully connected regression layer 110 Word embedding layer (BERT embedding layer)
901 Central processing unit 902 RAM
903 Input/output port 904, 905 Input/output device 906 Bus

Claims (6)

a word embedding unit that receives a word string and outputs a word embedding string corresponding to the word string;
a sequence labeling unit that receives the word embedding sequence output from the word embedding unit and outputs a label sequence corresponding to the word embedding sequence;
a phrase break position prediction unit that receives the word embedding sequence output from the word embedding unit and outputs phrase break position information corresponding to the word embedding sequence;
a learning data supply unit that supplies a word sequence to be input to the word embedding unit, supplies a correct label sequence corresponding to the label sequence output by the sequence labeling unit, and further supplies correct phrase break position information corresponding to the phrase break position information output by the phrase break position prediction unit;
Equipped with
the sequence labeling unit adjusts parameters of an internal model by performing backpropagation based on an error between the label sequence and the correct label sequence;
the phrase break position prediction unit adjusts parameters of an internal model by performing back propagation based on an error between the phrase break position information and the correct phrase break position information;
the word embedding unit adjusts parameters of an internal model by backpropagating errors from the sequence labeling unit and backpropagating errors from the phrase break position prediction unit;
Machine learning device. the phrase break position prediction unit outputs phrase break position information corresponding to the word embedding sequence by using a fully connected regression model based on all word embeddings included in the input word embedding sequence.
The machine learning device according to claim 1 . the sequence labeling unit outputs at least a label indicating that a predetermined subsequence included in the original input word sequence is an opinion target, and a label indicating that the predetermined subsequence is an opinion;
The machine learning device according to claim 1 or 2. the phrase break position prediction unit outputs, as the phrase break position information, numerical values representing the start position and end position of the substring;
the correct phrase-break position information provided by the learning data providing unit is numerical information indicating the correct start and end positions of the subsequence;
The machine learning device according to claim 3 . a word embedding unit that receives a word string and outputs a word embedding string corresponding to the word string;
a sequence labeling unit that receives the word embedding sequence output from the word embedding unit and outputs a label sequence corresponding to the word embedding sequence;
Equipped with
At least the model internal to the word embedding unit has been trained by the machine learning device according to any one of claims 1 to 4.
Natural language processing device. a word embedding unit that receives a word string and outputs a word embedding string corresponding to the word string;
a sequence labeling unit that receives the word embedding sequence output from the word embedding unit and outputs a label sequence corresponding to the word embedding sequence;
a phrase break position prediction unit that receives the word embedding sequence output from the word embedding unit and outputs phrase break position information corresponding to the word embedding sequence;
a learning data supply unit that supplies a word sequence to be input to the word embedding unit, supplies a correct label sequence corresponding to the label sequence output by the sequence labeling unit, and further supplies correct phrase break position information corresponding to the phrase break position information output by the phrase break position prediction unit;
Equipped with
the sequence labeling unit adjusts parameters of an internal model by performing backpropagation based on an error between the label sequence and the correct label sequence;
the phrase break position prediction unit adjusts parameters of an internal model by performing back propagation based on an error between the phrase break position information and the correct phrase break position information;
the word embedding unit adjusts parameters of an internal model by backpropagating errors from the sequence labeling unit and backpropagating errors from the phrase break position prediction unit;
A program that enables a computer to function as a machine learning device.