JP3606159B2 - Text processing device - Google Patents

Description

[0001]
BACKGROUND OF THE INVENTION
The present invention relates to a sentence processing apparatus, and more particularly to a sentence processing apparatus capable of accurately reflecting and extracting the topic of a sentence from a natural language sentence such as English or Japanese.
[0002]
[Prior art]
In the Internet and electronic libraries, how to effectively perform information retrieval, information extraction, and information summarization from sentences (text, documents) written in natural languages such as English and Japanese is becoming an increasingly important issue. Yes.
[0003]
Conventionally, a technique for expressing a topic (topic) of a sentence in the form of a keyword and automatically extracting the keyword from the sentence has been developed. By presenting the keyword to the user, it is possible to roughly understand the topic of the sentence. For example, a method of calculating a weight called tf-idf of a word appearing in a sentence and using a word having a large value of tf-idf as a keyword is widely used (G. Salton, A. Wong and CS. Yang, A Vector Space Model for Automatic Indexing, Communications of the ACM, Volume 18, Number 11, pp 613-620, 1975). The value of tf-idf is proportional to the frequency of the word in the attention sentence. That is, if a word appears frequently in a note, the value of tf-idf is large.
[0004]
Another prior art is the development of an automatic sentence segmentation technique. For example, the method proposed by Hurst (Marti Heart, TextTiling: Segmenting Text into Multi-paragraph Subtopic Passages, Computational Linguistics, Volume 23, Number 33, Number 19, Ppp. On the basis of it, the sentence can be divided into several parts. Further, an extension of the Hurst method such as “JP-A-11-272699: sentence summarizing apparatus and method” has been proposed.
[0005]
[Problems to be solved by the invention]
In the conventional keyword method, when a topic of a sentence is extracted based on a keyword, that is, a word, it may not be extracted well. For example, when the frequency of words representing topics is low, it is difficult to extract the topics represented by those words. For example, assume that the words “trade”, “tariff”, “import”, and “export” appear once in a newspaper article. Since “trade”, “tariff”, “import” and “export” are all related to trade, the topic “trade” should be extracted. However, since the appearance frequency of individual words is low, those words are difficult to be extracted as keywords. Therefore, it was difficult to extract the topic “trade” represented by these words.
[0006]
Further, in the conventional automatic sentence dividing method, since the sentence is divided based on the word appearance frequency vector, the sentence cannot be divided well according to the change in the topic structure. Furthermore, there is no method other than extracting the topic of each divided part using the keyword extraction method. Here too, there is a problem of topic extraction by the keyword.
[0007]
Accordingly, an object of the present invention is to provide a sentence processing apparatus capable of accurately reflecting and extracting from a sentence in a natural language such as English or Japanese, what the sentence is a topic.
[0008]
[Means for Solving the Problems]
In order to solve the above problems, the present invention provides a co-occurrence information storage means for storing co-occurrence information between words in a sentence made of a natural language, a sentence input means for inputting a sentence made of a natural language, and the sentence input means. The sentence inputted from the morpheme analyzing means for dividing the sentence in units of words, and the co-occurrence information stored in the co-occurrence information storage means based on the words divided by the morpheme analyzing means, the sentence Topic extraction means for extracting the topic , the topic extraction means inputs the words divided by the morpheme analysis means, and stores words that are likely to co-occur with the words in the co-occurrence information storage means The co-occurrence information is referred to, and the co-occurrence information reference means for only the words appearing in the input sentence among the easy-to-occur co-occurrence words and the divided words are shared. Easy to start A word cluster grouping unit that collects words appearing in the input sentence among words into a word cluster (group), and a first calculation unit that calculates an appearance ratio of individual words in the word cluster collected by the word cluster grouping unit. It is characterized by comprising: an appearance ratio calculating means; and a second appearance ratio calculating means for calculating an appearance ratio of the topic composed of a plurality of word clusters collected by the word cluster collecting means in the entire inputted sentence.
[0009]
Further, the present invention is characterized by further comprising a sentence dividing means for dividing the sentence input from the sentence input means into paragraphs.
In this way, the topic extraction unit extracts the topic of the sentence based on the co-occurrence information of the words constituting the sentence, so that the topic of the sentence can be accurately extracted.
[0010]
Furthermore, in this way, as shown in FIG. 12B, for example, the occurrence ratio of individual words (for example, trade ) in a cluster of words that are likely to co-occur (for example, clusters of trade-related topics) (in this case, 0) . 4 (= 40%)) can be determined at a glance.
Moreover, in this way, for example, as shown in FIG. 12 (B), the extent of the topic composed of a cluster of easily occurring words (trading related topic, time related topic, and OTHERS cluster) in the whole sentence is increased. The rate of appearance can be determined at a glance. For example, OTHERS clusters appear at a rate of 0.4 (= 40%) in the entire sentence.
[0011]
In addition, a display means for displaying the topic of the sentence extracted by the topic extraction means is provided.
If it does in this way, it can display as FIG. 12 (B), for example.
[0012]
In addition, the display means for displaying the topic of the sentence extracted by the topic extraction means, the display means, the sentence divided by the sentence dividing means, the word cluster collected by the word cluster collecting means, and the second The appearance ratio calculated by the appearance ratio calculation means is displayed in three dimensions .
In this way, for example, as shown in FIG. 16, the word cluster is expressed in the X-axis direction, the paragraph is expressed in the Y-axis direction, and the probability (appearance ratio) is expressed in the Z-axis direction. Can be determined.
[0013]
DETAILED DESCRIPTION OF THE INVENTION
Hereinafter, the present invention will be described based on first to fourth embodiments shown in the drawings. The basic configuration of the text processing apparatus B of the first to fourth embodiments is shown in FIG. 1, and the difference between the respective embodiments is only the configuration of the processing unit 40.
[0014]
(1) First Embodiment As shown in FIG. 1, the text processing apparatus B of this embodiment includes a storage unit 10 that is a “co-occurrence information storage unit”, an input unit 20 that is a “text input unit”, A morpheme analysis unit 30 that is a “morpheme analysis unit”, a processing unit 40 that is a “topic extraction unit”, and a display unit 50 are provided.
[0015]
The storage unit 10 stores word co-occurrence information, and stores word co-occurrence information indicating which words are likely to co-occur. Here, co-occurrence means that two or more linguistic phenomena can occur at the same time in the same utterance or sentence in linguistics, for example, “Toyachi” co-occurs with “walking” but “runs” (Sanseido, Shinmeikan Japanese Dictionary, 5th edition). FIG. 2 shows a specific example of co-occurrence information of words (English). For example, the word “trade” is information that is likely to co-occur with the words “import”, “export”, and the like, that is, information that is easy to co-occur between words.
[0016]
The word co-occurrence information in the storage unit 10 may be created by a person or automatically collected by a computer. When a computer automatically collects, the simplest method is to collect a large amount of sentences, count the co-occurrence frequency of any two words appearing in the same sentence, and easily co-occur the word pairs with high co-occurrence frequency It is to judge. However, in the present invention, the method of collecting the co-occurrence information is not limited to the method described here.
[0017]
The input unit 20 inputs natural language sentences such as English and Japanese. The sentence may be long or short.
[0018]
The morphological analysis unit 30 inputs a sentence from the input unit 20 and divides the sentence into word strings. FIG. 3A shows an example of a Japanese sentence. FIG. 3B shows an example of a result of performing morphological analysis on the sentence of FIG. That is, in Japanese, sentences are divided into words by morphological analysis. FIG. 4A shows an example of an English sentence. FIG. 4B shows an example of a result of performing morphological analysis on the sentence of FIG. That is, in English, a word is converted into a prototype by morphological analysis.
[0019]
FIG. 5 is a configuration diagram of the processing unit 40 of the first embodiment.
The processing unit 40 inputs a word string from the morphological analysis unit 30 and refers to the word co-occurrence information stored in the storage unit 10, a co-occurrence information reference unit 41, A word cluster summarizing unit 42 that is a “word cluster summarizing unit” for summarizing words that are likely to co-occur in a column into a word cluster is provided.
[0020]
Further, the processing unit 40 includes a linear combination model creating unit 43 that creates a probability model composed of word clusters. The probability model linearly combines the probability distributions in the word clusters with the probability distributions between the word clusters (referred to as a linear combination model) (see FIG. 12B, which will be described later). In addition, a probability model calculation unit 44 that calculates a probability distribution within a word cluster and a probability distribution between word clusters in the linear combination model is provided.
[0021]
For example, even if the appearance frequency of a certain word is low in a probability distribution within a certain word cluster, the certain word may have a high appearance frequency between word clusters. Therefore, by creating a linear combination model, it is possible to accurately reflect not only the appearance frequency within a certain word cluster but also the topic of the entire sentence including the appearance frequency between word clusters. The linear combination model creation unit 43 and the probability model calculation unit 44 constitute “first appearance ratio calculation means” and “second appearance ratio calculation means”.
[0022]
The display unit 50 receives a probability distribution between word clusters from the processing unit 40, and displays a probability distribution between word clusters or a word cluster having a high probability in the probability distribution between word clusters to the user of the sentence processing apparatus. .
[0023]
FIG. 6 shows a flowchart of processing of the text processing apparatus of this embodiment.
If the text processing apparatus of the present embodiment is used, the topic expressed by the word cluster shown in FIG. 7B is presented to the user via the display unit 50 from the input text shown in FIG. The word cluster “month-January-February” in FIG. 7B represents one topic, and the word cluster “trade-export-import” in FIG. 7C represents another topic.
[0024]
FIG. 7C shows keywords (topics) extracted from the text in FIG. 7A using tf-idf. Although both consist of 7 words, it can be seen that the topic extracted in FIG. 7B is more appropriate than the topic extracted in FIG. It is also clear that the user can easily understand FIG. 7B than FIG. 7C.
[0025]
Hereinafter, how the word cluster compiling unit 42 collects words into word clusters, the linear combination model creation unit 43 creates a linear combination model, and the probability model calculation unit 44 calculates the probability distribution in the linear combination model. Will be described in detail.
[0026]
For example, the word cluster collecting unit 42 collects words in the input sentence into word clusters as follows. However, in the present invention, the method of collecting word clusters is not limited to the method described here.
First, the value of tf-idf of each word in the input sentence is calculated as shown in FIG. 8A, and those having a high value of tf-idf are collected. Hereinafter, a word having a large tf-idf is referred to as a seed word. FIG. 8B shows a seed word extracted from the sentence of FIG.
[0027]
The co-occurrence information reference unit 41 refers to a word that is likely to co-occur with respect to the seed word from the storage unit 10. It should be noted that only words that appear in the input sentence among words that tend to co-occur are targeted. What is shown in FIG. 9A is a word that appears easily in the input sentence and is likely to co-occur with the seed word.
[0028]
The word cluster consolidating unit 42 consolidates the seed words and the words that appear easily in the input sentence into one word cluster. That is, one word set of synonyms and related words is created around the seed word. There is always a seed word in a word cluster. Further, other words appearing in the input sentence are collected into one special cluster “OTHERS”. FIG. 9B shows an example of a clustered word cluster.
[0029]
Next, the word cluster collecting unit 42 merges the created word clusters into several larger word clusters. However, this merging process may be omitted. The merging method is not limited to the method described here.
The word cluster collecting unit 42 merges two clusters in which seed words are likely to co-occur with each other. For example, “surplus” is a co-occurrence word of the seed word “trade”. At the same time, there is “trade” in the co-occurrence word of the seed word “surplus”. The word cluster collecting unit 42 combines two word clusters having “trade” and “surplus” as seed words into one. The word cluster merge algorithm of the processing unit 40 is shown in FIG. Furthermore, the flowchart is shown in FIG.
[0030]
The linear combination model creation unit 43 of the processing unit 40 creates a linear combination model composed of word clusters. As described above, the linear combination model accurately reflects changes in topics within a sentence. Specifically, a word cluster corresponds to a topic, and a word belonging to the word cluster can be interpreted as a word related to the topic. In general, sentences are concentrated on one topic, but may temporarily move to another topic away from the topic. Also, when sentences are concentrated on one topic, there is a tendency to use many words that are closely related to that topic. For example, a topic “trade” is often discussed in a certain sentence, and at that time, a word closely related to “trade” is often used, but sometimes it is shifted to a topic “politics”. That is, words that are deeply related to “politics” are used.
[0031]
FIG. 12A shows a mathematical expression of a linear combination model. W represents a word and k represents a word class. The occurrence probability of the word w from the word cluster k is expressed as P (w / k). Pk (w / k) is a probability in the intra-cluster distribution. The occurrence probability of the word cluster k is expressed as P (k). P (k) is the probability in the distribution between clusters.
[0032]
FIG. 12B shows an example of a linear combination model. A word cluster represents a topic. In FIG. 12B, for example, the left word cluster represents a trade-related topic, and the middle word cluster represents a time-related topic. The right word cluster corresponds to OTHERS. The same word may belong to multiple word clusters.
[0033]
The probability distribution between the word clusters represents the probability (relative frequency) that each topic is mentioned in the input sentence. The probability distribution within a word cluster represents how often each word is referred to in the related topic. For example, in FIG. 12B, the values “0.3 (30%), 0.3 (30%), 0.4 (40%)” indicate the distribution between word clusters. This is mentioned in a sentence with a probability of 0.3 (30%), 0.3 (30%), and 0.4 (40%) for trade-related topics, time-related topics, and OTHERS, respectively. It means being (appearing). Further, for example, in a trade-related word cluster, the values “0.4, 0.1, 0.25, 0.25” form a probability distribution within one word cluster. It means that the word “trade” is mentioned in a trade-related topic with a probability of 0.4.
[0034]
The probability model calculation unit 44 of the processing unit 40 calculates the probability of the linear combination model based on the appearance frequency of words in the input sentence. The calculation algorithm is shown in FIG. In this calculation, the initial value of the probability value in the linear combination model is first set as appropriate. Next, each probability value of the linear combination model is calculated by iterative calculation based on the appearance frequency of words in the sentence. By this calculation, for example, if there are many trade-related words in a sentence, the probability of trade-related topics (word clusters) increases. In addition, if the occurrence frequency of “import” is high in a sentence, the probability of “import” is increased in a trade-related topic (word cluster). However, in the present invention, the probability calculation method is not limited to the method described here.
[0035]
The display unit 50 finally displays a high-probability word cluster as a topic to the user instead of a high-probability keyword. Alternatively, the display unit 50 displays the probability distribution itself between the word clusters to the user. However, in the present invention, the method of displaying the topic to the user is not limited to the method described here.
[0036]
(2) Second Embodiment The configuration of this embodiment differs from the first embodiment only in the processing unit 40A as described above, and the other storage unit 10, input unit 20, morpheme analysis unit 30, and display unit 50 are the first. The same as the embodiment.
FIG. 14 shows a processing unit 40A of the present embodiment, and the co-occurrence information reference unit 41, the word cluster summarizing unit 42, and the linear combination model creating unit 43 are the same as those in the first embodiment, and thus the duplicate description is omitted.
[0037]
The sentence division unit 45 of the processing unit 40A divides the input word string into several parts, and the linear combination model creation unit 43 generates a linear combination model composed of the word clusters collected in each divided part. create. Further, the probability distribution calculation unit 46 calculates the probability distribution in the word clusters and the probability distribution between the word clusters in the linear combination model in each part.
The display unit 50 inputs a probability distribution between word clusters in each portion or a word cluster having a high probability in the probability distribution between word clusters from the probability distribution calculation unit 46 and displays it to the user.
[0038]
FIG. 15 shows a flowchart of processing of the text processing apparatus of this embodiment.
The sentence processing apparatus of this embodiment divides a sentence into several parts (first paragraph, second paragraph, third paragraph, etc.), and extracts the word cluster as an important topic of each part. Is different.
It is conceivable to use a conventional method, for example, the Hurst method, for dividing the sentence. However, it is not limited to the Hurst method.
[0039]
For example, FIG. 16 shows an example of the topic of each part in the text displayed by the text processing apparatus of this embodiment. A sentence is divided according to its contents, and the topic of each divided part is expressed by a word cluster and its probability. That is, the word cluster is shown in the X-axis direction, the block divided in the Y-axis direction is shown, and the probability of the word cluster is shown in the Z-axis direction. This display allows the user to quickly and accurately grasp the topic structure of the sentence.
[0040]
Also, for example, FIG. 17 shows an example of each divided part of the text displayed by the text processing apparatus of this embodiment and the topic of each part. This display allows the user to accurately grasp the topic structure of the sentence. Here, the topic of each part may be displayed in bold or red so that the user can easily understand what the topic is at a glance.
In the present invention, the method of displaying the topic to the user is not limited to the method described here.
[0041]
(3) Third Embodiment As described above, the configuration of the third embodiment is different from the first embodiment only in the processing unit 40B, and the other storage unit 10, input unit 20, morphological analysis unit 30, and display unit 50 are the first. The same as the embodiment.
FIG. 18 shows the processing unit 40B of the present embodiment, and the co-occurrence information reference unit 41, the word cluster summarizing unit 42, and the linear combination model creation unit 43 are the same as those in the first embodiment, and thus redundant description is omitted.
[0042]
The sentence division unit 47 of the processing unit 40B divides the input word sequence into several parts using the linear combination model, and the probability distribution calculation unit 48 includes a word cluster in the linear combination model in each part. Compute probability distribution and probability distribution between word clusters.
The display unit 50 inputs a probability distribution between word clusters in each portion or a word cluster having a high probability value in the probability distribution between word clusters from the probability distribution calculation unit 48 and displays it to the user.
[0043]
FIG. 19 shows a flowchart of processing of the text processing apparatus of this embodiment.
The sentence processing apparatus according to the present embodiment first creates a linear combination model, and then divides the sentence using the created linear combination model. The second embodiment is different from the second embodiment in that division is performed using a linear combination model.
The linear combination model created by the linear combination model creation unit 43 of the processing unit 40B is the same as that shown in FIG.
The sentence dividing unit 47 divides sentences using a linear combination model. The method is shown below. However, the sentence division method using the linear combination model is not limited to the following description.
[0044]
First, a point that can be a division point in an input sentence is specified. Usually, the end of each sentence in a sentence is such a point. For example, in the sentence of FIG. 7A, the point immediately after “said.”, “Respectively.”, “Year.” Can be a division point.
Next, the words in the k words at the left and right of the dividing point are collected, and each is made into one pseudo sentence. Alternatively, the words in the left and right k sentences may be collected and each may be made into one pseudo sentence. k is a predetermined value. The probability distribution in the linear combination model created based on the left and right pseudo sentences is calculated. The probability calculation method may be as shown in FIG. 13, but may be as shown in FIG. 20, for example.
[0045]
Next, the calculated distance between the two left and right linear combination models is calculated. FIG. 21 shows a formula for calculating the distance between the probability models. If the distance between the left and right linear combination models is greater than or equal to a predetermined threshold, it is determined that this point is a division point.
The above processing is performed for all candidates for division points, points to be division points are determined, and sentences are divided.
The display unit 50 displays, for example, a word cluster having a high probability of each divided part as a topic and displays it to the user.
[0046]
(4) Fourth Embodiment In this embodiment, a large amount of text is input to the input unit 20, and the configuration of the text processing apparatus is the same as that of the third embodiment.
The sentence dividing unit 47 of the processing unit 40 divides the input word sequence into several parts using the linear combination model, and the probability distribution in the word clusters in the linear combination model in each part and the word clusters are divided. Calculate the probability distribution.
[0047]
The display unit 50 inputs from the processing unit 40 a probability distribution between word clusters in each portion or a word cluster having a high probability in the probability distribution between word clusters and displays it to the user.
The feature of the sentence processing apparatus of the present embodiment is that it can display topic changes in a time series of a large amount of sentences. For example, it is possible to input a newspaper article for one year and see how topics change in the newspaper article for that year.
[0048]
【The invention's effect】
As described above, according to the present invention, a cluster of meaning words is extracted from a sentence and used as the topic of the sentence. Therefore, the topic can be extracted accurately and easily understood by the user using keywords. This can be clearly seen from the examples of FIGS. 7B and 7C.
Also, by dividing the sentence and showing the topic of each divided part, the contents of the sentence can be displayed in a form that is more easily understood by the user. Using the present invention, the user can see not only important topics but also changes in topics within the text.
[Brief description of the drawings]
FIG. 1 is a configuration diagram of each embodiment of the present invention.
FIG. 2 is an example of word co-occurrence information applied to each embodiment.
FIG. 3 is an example of an input sentence.
FIG. 4 is an example of another input sentence.
FIG. 5 is a configuration diagram of a processing unit according to the first embodiment.
FIG. 6 is a flowchart of the first embodiment.
7 is a processing example in the first embodiment, where (A) is an example of a sentence, (B) is a word cluster extracted from the sentence of (A), and (C) is a keyword extracted from the sentence of (A). It is.
8A is a formula for calculating the tf-idf value of a word, and FIG. 8B is a word having a large tf-idf value in FIG. 7A.
9A is a seed word and co-occurrence information of the sentence in FIG. 7A, and FIG. 9B is a seed word and cluster of the sentence in FIG. 7A.
FIG. 10 is an algorithm for merging word clusters.
FIG. 11 is a flowchart of merging word clusters.
12A is a calculation formula of a linear combination model, and FIG. 12B is an example of a linear combination model.
FIG. 13 is a calculation method of a probability distribution in a linear combination model.
FIG. 14 is a configuration diagram of a processing unit according to the second embodiment.
FIG. 15 is a flowchart of processing in the second embodiment.
FIG. 16 is an output example of the second embodiment.
FIG. 17 is an output example of the second embodiment.
FIG. 18 is a configuration diagram of a processing unit according to a third embodiment.
FIG. 19 is a flowchart of the third embodiment.
20 is a probability distribution calculation method in a linear combination model. FIG. 21 is a formula for calculating a distance between linear combination models.
[Explanation of symbols]
B Text processing device 10 Storage unit 20 Input unit 30 Morphological analysis unit 40, 40A, 40B Processing unit 41 Co-occurrence information reference unit 42 Word cluster summary unit 43 Linear combination model creation unit 44 Probability model calculation unit 45 Word rearrangement division unit 46 Probability Distribution calculation unit 47 Text division unit 48 Probability distribution calculation unit 50 Display unit

Claims (4)

Co-occurrence information storage means for storing co-occurrence information between words in sentences composed of natural language;
A sentence input means for inputting sentences composed of natural language;
A morpheme analyzing means for dividing the sentence input from the sentence input means in units of words;
Topic extraction means for referring to the co-occurrence information stored in the co-occurrence information storage means based on the words divided by the morpheme analysis means and extracting the topic of the sentence;
With
The topic extraction means includes
A word divided by the morphological analysis unit is input, and a word that is likely to co-occur with the word is referred to from the co-occurrence information stored in the co-occurrence information storage unit, and this reference is likely to occur with the co-occurrence. Co-occurrence information reference means for only words appearing in the input sentence among words;
Word cluster collecting means for collecting words that appear in the input sentence among words that are likely to co-occur with the divided words into a word cluster (group);
First appearance ratio calculating means for calculating the appearance ratio of individual words in the word cluster collected by the word cluster collecting means;
Second appearance ratio calculating means for calculating an appearance ratio of the topic composed of a plurality of word clusters collected by the word cluster collecting means in the entire inputted sentence;
Text processing apparatus according to claim <br/> further comprising a. 2. The sentence processing apparatus according to claim 1, further comprising sentence dividing means for dividing the sentence input from the sentence input means into paragraphs. The sentence processing apparatus according to claim 1 or 2, further comprising display means for displaying a topic of the sentence extracted by the topic extracting means . Display means for displaying the topic of the sentence extracted by the topic extraction means;
The display means displays in three dimensions the sentences divided by the sentence dividing means, the word clusters collected by the word cluster collecting means, and the topic appearance ratio calculated by the second appearance ratio calculating means < The sentence processing apparatus according to claim 2, wherein

Images