JP6312467B2 - Information processing apparatus, information processing method, and program - Google Patents

Description

  The present invention relates to an information processing apparatus that performs natural language processing, and more particularly, to an information processing apparatus that divides a given character string into words and obtains a semantic similarity of the divided words.

  Conventionally, as a method of dividing a given character string into words, a method is known in which a word dictionary is prepared in advance and the given character string is divided into words based on the dictionary (Non-Patent Document 1). ). There is also known a method of dividing a character string into words by unsupervised learning without preparing a word dictionary. In this method, character N-grams and word N-grams are probabilistically modeled by a nonparametric Bayes method to estimate words (Non-Patent Document 2).

  As a method for estimating the semantic similarity between words, a method is known in which a concept base is prepared in advance and the similarity between words is estimated by collation with the concept base. In this method, for a word that is not registered in the concept base, a similarity is estimated in a sentence including the word based on a relationship with a word that is in the sentence and registered in the concept base (Patent Literature). 1). Also, a method is known in which a multi-layer neural network is used without using a concept base, and a similarity between words is spontaneously configured in an intermediate layer of the neural network (Non-patent Document 3).

JP 2010-224887 A

"Introduction to machine learning for language processing", supervised by Manabu Okumura, Daiya Takamura (Corona) "Unsupervised morphological analysis using Bayesian hierarchical language model", Daichi Mochihashi Takeshi Yamada, Nobuo Ueda, Natural Language Processing Society (NL190) “Linguistic Regularities in Continuous Space Word Representations”, T. Mikolov, W-T Yih, G. Zweig, (INTERSPEECH 2013)

  However, the method of performing word division based on a word dictionary has a problem in that word division cannot be performed correctly for a character string including an unknown word that is not registered in the dictionary. In order to reduce the number of unknown words, a large word dictionary must be created. In the word division by unsupervised learning, the above-mentioned problems are cleared, but the semantic similarity of the divided words cannot be estimated.

  In order to estimate the semantic similarity, it is necessary in the prior art to prepare a concept base in advance as described in Patent Document 1, but for the construction of the concept base, a very large number of human resources are required. Cost. In addition, for words that are not registered in the concept base, words that are registered in other concept bases appearing around the word are used, but those words that are not related to the target word are included. Therefore, there is a problem that the meaning cannot be estimated correctly. Further, in the method of Patent Document 1, it is assumed that the words are correctly divided in advance, and the above-described problems relating to word division similarly apply.

  According to the method described in Non-Patent Document 3, a word is embedded in a continuous space representing a semantic relationship without preparing a concept base, and a semantic similarity between words is obtained by a distance in the space. Can do. However, even in the method of Non-Patent Document 3, it is assumed that the words are correctly divided in advance, and the above-described problem regarding word division is similarly applied.

  As described above, in the prior art, it is not possible to simultaneously estimate the word division and the semantic similarity of the divided words without preparing teacher data in advance.

  There are the above-mentioned problems to automatically acquire word division and semantic similarity of words without a teacher, but no framework for solving these problems has existed until now.

  Accordingly, an object of the present invention is to provide an information processing apparatus that performs word division of given learning data and automatically acquires the semantic similarity of the divided words.

  An information processing apparatus according to the present invention includes an input unit that inputs sentence data as learning data, a word division unit that divides the learning data using a character N-gram or a word division model, and divided word data. A character N-gram learning unit that learns a character N-gram based on the character N-gram and stores the learned character N-gram in a character N-gram storage unit, and a word by sequence labeling that recognizes a word boundary based on the divided word data A word boundary learning unit that learns a division model and stores a word division model obtained by learning in a word division model storage unit, an input layer, an intermediate layer, and an output layer, and outputs from the intermediate layer to the input layer The word N-gram represented by the recursive neural network that is also input is learned using the divided word data as teacher data and stored in the word N-gram storage unit A word N-gram learning unit, a word data input to a recursive neural network stored in the word N-gram storage unit, and a concept data calculation unit that obtains data obtained in an intermediate layer as concept data; An output unit for outputting concept data, wherein the word division unit uses the character N-gram learned by the character N-gram learning unit and the word division model learned by the word boundary learning unit. A process of alternately performing word division using a word, and a process of learning the word N-gram using the word data divided by the word division unit by the word N-gram learning unit, with a predetermined convergence condition Repeat until it is met.

  Concept data representing the word concept in the intermediate layer by repeatedly learning the word N-gram represented by the recursive neural network using the word data divided by the word dividing unit as teacher data. Can be requested. Therefore, according to the present invention, it is possible to automatically acquire data representing the concept of a word from given learning data without preparing a word dictionary or a concept dictionary. Utilization of natural language processing such as blogs and spoken language, which has been difficult to apply, can be promoted.

  Further, the recursive neural network may receive data of the first to Nth words constituting the sentence and output the N + 1th word. As a result, conceptual data reflecting contextual factors appears in the intermediate layer.

  The information processing apparatus of the present invention includes a clustering unit that clusters words having a similarity between words larger than a predetermined threshold into the same group based on the concept data, and the output unit outputs a clustering result. May be. At this time, as a word representing the cluster, the most frequently used word among the words existing in the cluster may be output. The clustering unit may perform clustering hierarchically.

  In the information processing apparatus of the present invention, the word division unit may perform initial division of the learning data based on a character code when the learning data is given. Thereby, even when there is no character N-gram or word division model, initial division from learning data into words can be performed appropriately.

  The information processing method of the present invention is a method of dividing input learning data into words by an information processing device and obtaining a concept of the divided words, wherein the information processing device inputs sentence data as learning data. And the information processing apparatus alternately performs word division using a character N-gram and word division using a word division model on the learning data, and the divided word data is used as teacher data. A process of learning a word N-gram represented by a recurrent neural network that has an input layer, an intermediate layer, and an output layer, and that also inputs the output from the intermediate layer to the input layer, and satisfies a predetermined convergence condition And the information processing device inputs the word data to the recursive neural network stored in the word N-gram storage unit and obtains it in the intermediate layer. Obtaining the data to be processed as conceptual data, and the information processing apparatus outputting the conceptual data, wherein the information learning apparatus uses the character N gram to learn the word N-gram. A step of dividing the learning data into words, and the information processing apparatus learns a word N-gram represented by the recursive neural network using the divided word data as teacher data, and stores the word N-gram A step of storing in a section, and the information processing apparatus learns a word division model by sequence labeling that recognizes word boundaries based on divided word data, and the word division model obtained by learning is divided into words Storing in the model storage unit, and the information processing apparatus uses the word division model to store the learning data. A step of dividing a word; and the information processing apparatus learns a word N-gram represented by the recursive neural network using the divided word data as teacher data and stores it in a word N-gram storage unit And a step of learning the character N-gram based on the divided word data and storing the learned character N-gram in the character N-gram storage unit.

  The program of the present invention is a program that causes a computer to execute the information processing method described above.

  According to the present invention, it is possible to automatically acquire the semantic similarity of words from given text data without preparing a word dictionary and a concept dictionary. It is possible to promote natural language processing for blogs and spoken words that were difficult to apply in the past.

It is a figure which shows the functional block of the information processing apparatus of embodiment. It is a figure which shows the example of a recursive neural network. It is a figure which shows the hardware constitutions of the information processing apparatus of embodiment. It is a figure which shows the outline | summary of operation | movement of the information processing apparatus of embodiment. It is a figure which shows an example of the process of word division | segmentation and the learning of a word N-gram. It is a figure which shows an example of the process of word division | segmentation and the learning of a word N-gram.

Hereinafter, an information processing apparatus according to an embodiment of the present invention will be described with reference to the drawings.
FIG. 1 is a diagram illustrating functional blocks of the information processing apparatus according to the embodiment. The information processing apparatus includes an input unit 10 that inputs learning data 21, a word dividing unit 11 that performs word division of the learning data 21, a character N-gram learning unit 12 that performs learning based on divided word data, It has a word N-gram learning unit 14 and a word boundary learning unit 16. The word N-gram learning unit 14 learns a word N-gram using the recursive neural network shown in FIG. The information processing apparatus also includes a concept vector calculation unit 18 that calculates a word concept vector 22 using a learned recursive neural network, a clustering unit 19 that performs word clustering based on the concept vector 22, and a concept vector. Output unit 20 for outputting 22.

The information processing apparatus treats word data as a vector. Specifically, the maximum length of one word K, defines the L ^K-number of possible words the character type as L, giving a word vectors that uniquely identifies the words that are defined. When defining a word, the maximum number of character type switching may be limited to M. Specifically, the word vector is a vector in which only the component representing the word among the L ^K components is “1” and the rest is “0”. Such expression of word vectors is called 1 of N coding.

  The input unit 10 of the information processing apparatus accepts input of sentence text data as learning data 21. The learning data 21 may be acquired from, for example, a blog on the web.

  The word dividing unit 11 has a function of dividing the learning data 21 into words. The word division unit 11 can perform word division by the following three methods. That is, (1) word division is performed at the point where the character code is switched. (2) Perform word division using character N-grams. (3) Perform word division using a word division model.

  As in (1), the word division is performed at the point where the character code is switched at the time of the first word division given the learning data 21. When learning data 21 is given for the first time, since character N-gram and word division model for learning data 21 are not obtained, initial division is performed by switching character codes. In the present embodiment, a method using character code switching is used for the initial division, but the division can also be performed by an existing morpheme analyzer (as a weak classifier).

The word division using the character N-gram of (2) is performed as follows.
The division candidate w of sentence x is written as follows.

Word segmentation is sought to maximize the following equation:

  A known technique can be used for word division using the word division model of (3).

  The character N-gram learning unit 12 learns a character N-gram after inserting virtual characters indicating the word start and word end before and after the divided word. For learning the character N-gram, a statistical N-gram model can be used. For example, a method called Kneser-Ney smoothing may be used. The character N-gram learning unit 12 stores the character N-gram data obtained by learning in the character N-gram storage unit 13.

  As described above, the word N-gram learning unit 14 learns the word N-gram using the recursive neural network shown in FIG. The recursive neural network includes an input layer 30 to which a word vector w (t) of divided words is input, an intermediate layer 32, and an input layer that receives the output of the intermediate layer 32 and outputs it to the intermediate layer 32 again. 31 and an output layer 33. The intermediate layer 32 is recursively updated with the input data in the network. The output layer 33 outputs the Nth word vector following the input (N−1) word vectors in the form of a probability distribution. In FIG. 2, the alphabet described in the vicinity of the arrow is a combined load vector of each layer. Moreover, the vector which appears in each layer is described on each layer.

  Here, when the word vector w (t) is input, the vector s (t) appearing in the intermediate layer 32 corresponds to the concept vector of the word. In the recursive neural network, the output of the intermediate layer 32 is input to the input layer 31 and input to the intermediate layer 32 again. That is, the previous word vector w (t−1) affects the concept vector s (t) of the word vector w (t), and the word vector w (t) is obtained in consideration of the context. Become.

  In the recursive neural network, learning is performed by the back propagation method based on the Nth word vector when (N-1) word vectors are input, using the divided word data as teacher data. The word N-gram learning unit 14 stores the recursive neural network data updated by learning in the word N-gram storage unit 15.

  The word boundary learning unit 16 learns a word division model by sequence labeling that recognizes word boundaries. As the word model, a statistical model such as CRF (Conditional Random Field) may be used, or a neural network such as structured perceptron may be used. The word boundary learning unit 16 stores the learned word division model in the word division model storage unit 17.

  The information processing apparatus updates the recursive neural network indicating the word N-gram using the word data divided by the word dividing unit 11 described above as teacher data, and uses the word N-gram based on the divided word data. And word boundaries are learned, and the process of dividing words using the learned word N-gram and word division model is repeated. As the convergence determination for completing the iterative processing, for example, when the word division is performed a predetermined number of times I, or when the perplexity is calculated and the value does not change more than the predetermined value, the iterative processing is performed. You can think of how to complete it.

  The concept vector calculation unit 18 inputs a word vector to the recursive neural network learned by the word N-gram learning unit 14 and obtains an intermediate layer vector corresponding to the input as the concept vector 22. The concept vector 22 gives the position of the word in the concept space, and the distance and direction of the concept vector 22 represent the semantic similarity.

  The clustering unit 19 has a function of clustering similar words based on the concept vector 22 calculated by the concept vector calculation unit 18. The clustering unit 19 may perform clustering hierarchically.

  The output unit 20 outputs the data of the concept vector 22 calculated by the concept vector calculation unit 18 and the clustering result. The output unit 20 may output a word having the highest frequency among words existing in the cluster as a word representing the cluster. In addition to the above, the output unit 20 may output data of a word N-gram.

  FIG. 3 is a diagram illustrating a hardware configuration for realizing the functions of the information processing apparatus described above. The hardware of the information processing apparatus is a normal computer including a CPU 40, a RAM 41, a ROM 42, a communication interface 44, a hard disk 45, a keyboard 46, and a monitor 47. By reading and executing the program 43 stored in the ROM 42, the information processing apparatus described above is realized. Such a program 43 is also included in the scope of the present invention.

  Next, the operation of the information processing apparatus according to the embodiment will be described. FIG. 4 is a diagram showing an outline of the operation of the information processing apparatus, and FIGS. 5 and 6 are diagrams showing the operation of word division and word N-gram learning. First, an outline of the operation of the information processing apparatus will be described with reference to FIG.

First, the information processing apparatus defines L ^K possible words, where the maximum length of one word is K and the character type is L, and gives a word vector that uniquely identifies the defined word (S10). Subsequently, the information processing apparatus inputs learning data 21 (S11).

  The information processing apparatus learns the word N-gram based on the word division of the input learning data 21 and the divided word data (S12). This process will be described later with reference to FIGS. 5 and 6. When the word segmentation and the learning of the word N-gram converge, the information processing device obtains a concept vector 22 of each word using a recursive neural network of the word N-gram (S13), and word clustering based on the concept vector 22 (S14). Then, the information processing apparatus outputs the word concept vector 22 and the word clustering result (S15).

  With reference to FIG. 5, word division and word N-gram processing will be described. The information processing apparatus first performs initial word division (S20). In the present embodiment, the information processing apparatus divides words by switching character codes. Next, the information processing apparatus learns a word N-gram using the divided word data (S21). More specifically, the recursive neural network learns which N + 1th word appears after 1 to N words included in the sentence.

  The information processing apparatus learns the character N-gram based on the divided word data (S22), and performs word division of the learning data 21 using the learned character N-gram (S23).

  Next, the information processing apparatus learns a word N-gram using the divided word data (S24), and learns a word division model by series labeling (S25). Then, the learning data 21 is divided into words using the learned word division model (S26).

  The information processing apparatus determines whether or not the convergence condition of the repetition process of word division and word N-gram learning is satisfied (S27). If the convergence condition is satisfied (YES in S27), the information processing apparatus ends word division and word N-gram learning. If the convergence condition is not satisfied (NO in S27), the information processing apparatus starts the process of learning the word N-gram again (S21).

  As described above, the information processing apparatus according to the present embodiment alternately performs word division using the character N-gram and word division using the word division model, and learns the word N-gram after each word division process. Thus, learning of the recurrent neural network constituting the word N-gram is performed, and the concept vector 22 of the word can be obtained by the intermediate layer 32 of the recursive neural network.

  FIG. 6 is a diagram illustrating another example of word division and word N-gram learning by the information processing apparatus. The process shown in FIG. 6 is basically the same as the process shown in FIG. 5, but in the example shown in FIG. 5, after the initial division (S20), learning of character N-gram (S22), and by character N-gram While the word division (S23) is performed, in the example shown in FIG. 6, after the initial division (S30), the word division model is learned (S32) and the word division using the word division model (S33) is performed. Is different. The example shown in FIG. 6 is also the same as the example shown in FIG. 5 in that word division by the character N-gram and word division by the word division model are alternately performed and the word N-gram is learned after each word division process. It is. In this way, it is arbitrary whether learning after word division is performed first, the character N-gram or the word division model.

  The information processing apparatus according to the embodiment of the present invention has been described with reference to the embodiment. However, the present invention is not limited to the above-described embodiment. For example, the learning of the word N-gram performed after the word division is not necessarily performed every time the word division is performed, for example, based on the word division based on the character N-gram and the word division model. When the word division is performed, the word N-gram may be learned.

  The information processing apparatus according to the present invention has an effect that a semantic similarity of words can be automatically acquired from given text data without preparing a word dictionary and a concept dictionary. It is useful as a device for processing.

DESCRIPTION OF SYMBOLS 10 Input part 11 Word division part 12 Character N-gram learning part 13 Character N-gram storage part 14 Word N-gram learning part 15 Word N-gram storage part 16 Word boundary learning part 17 Word division model storage part 18 Concept vector calculation part 19 Output part 20 learning data 21 concept vector 30 input layer 31 input layer 32 intermediate layer 33 output layer 40 CPU
41 RAM
42 ROM
43 Program 44 Communication interface 45 Hard disk 46 Keyboard 47 Monitor

Claims (7)

An input unit for inputting sentence data as learning data;
A word division unit for dividing the learning data into words using a character N-gram or a word division model;
A character N-gram learning unit that learns a character N-gram based on the data of the divided words and stores the learned character N-gram in a character N-gram storage unit;
A word boundary learning unit that learns a word division model by sequence labeling that recognizes word boundaries based on divided word data, and stores the word division model obtained by learning in a word division model storage unit;
A word N-gram represented by a recursive neural network that has an input layer, an intermediate layer, and an output layer, and outputs the output from the intermediate layer to the input layer as well, using the divided word data as teacher data A word N-gram learning unit stored in the word N-gram storage unit;
A concept data calculation unit for inputting word data to a recursive neural network stored in the word N-gram storage unit and obtaining data obtained in the intermediate layer as concept data;
An output unit for outputting the conceptual data;
With
Processing in which the word division unit alternately performs word division using the character N-gram learned by the character N-gram learning unit and word division using the word division model learned by the word boundary learning unit An information processing apparatus that repeatedly performs the process of learning the word N-gram using the word data divided by the word dividing unit until the word N-gram learning unit satisfies a predetermined convergence condition.   The information processing apparatus according to claim 1, wherein the recursive neural network receives data of first to Nth words constituting a sentence and outputs N + 1th word. Based on the concept data, a clustering unit that clusters words in which the similarity between words is greater than a predetermined threshold into the same group,
The information processing apparatus according to claim 1, wherein the output unit outputs a result of clustering.   The information processing apparatus according to claim 3, wherein the clustering unit performs clustering hierarchically.   The information processing apparatus according to claim 1, wherein the word dividing unit performs initial division of the learning data based on a character code when the learning data is given. An information processing device divides input learning data into words, and obtains the concept of the divided words,
The information processing apparatus inputs sentence data as learning data;
The information processing apparatus alternately performs word division using a character N-gram and word division using a word division model on the learning data, and inputs the divided word data as teacher data. Repeating a process of learning a word N-gram represented by a recursive neural network having a layer, an intermediate layer, and an output layer, and inputting the output from the intermediate layer to the input layer until a predetermined convergence condition is satisfied When,
The information processing apparatus inputs word data to the recursive neural network, and obtains data obtained in an intermediate layer as concept data;
The information processing apparatus outputting the conceptual data;
With
The step of learning the word N-gram includes:
The information processing apparatus performing word division of the learning data using a character N-gram;
The information processing apparatus learning the word N-gram represented by the recursive neural network using the divided word data as teacher data, and storing it in the word N-gram storage unit;
The information processing apparatus learns a word division model by sequence labeling for recognizing a word boundary based on divided word data, and stores the word division model obtained by learning in a word division model storage unit When,
The information processing apparatus performing word division of the learning data using the word division model;
The information processing apparatus learning the word N-gram represented by the recursive neural network using the divided word data as teacher data, and storing it in the word N-gram storage unit;
The information processing apparatus learns a character N-gram based on the divided word data, and stores the learned character N-gram in a character N-gram storage unit;
An information processing method comprising: A program for dividing the input learning data into words and obtaining a concept of the divided words.
Inputting sentence data as learning data;
For the learning data, word division using a character N-gram and word division using a word division model are alternately performed, and the divided word data is used as teacher data, and an input layer, an intermediate layer, and an output Repeating a process of learning a word N-gram represented by a recursive neural network having a layer and inputting an output from an intermediate layer to an input layer until a predetermined convergence condition is satisfied;
Inputting word data to the recursive neural network and obtaining data obtained in the intermediate layer as concept data;
Outputting the conceptual data;
And execute
In the step of learning the word N-gram,
Performing word division of the learning data using a character N-gram;
Learning the word N-gram represented by the recursive neural network using the divided word data as teacher data, and storing it in the word N-gram storage unit;
Learning a word division model by sequence labeling that recognizes word boundaries based on the data of the divided words, and storing the word division model obtained by learning in a word division model storage unit;
Performing word division of the learning data using the word division model;
Learning the word N-gram represented by the recursive neural network using the divided word data as teacher data, and storing it in the word N-gram storage unit;
Learning a character N-gram based on the divided word data, and storing the learned character N-gram in a character N-gram storage unit;
A program that repeatedly executes