JP5740228B2 - Representative comment extraction method and program - Google Patents

Description

  The present invention relates to a method and program for extracting representative comments from a text set for a specific topic such as a microblog.

  Conventionally, as an automatic summarization technique for realizing natural summarization using a computer, there has been a technique for extracting only an important sentence (summary) from a sentence set constituting a document (Non-patent Document 1). In addition, there has been a technique for classifying opinions on a product or topic into pros and cons by a computer (Patent Document 1, Non-Patent Documents 2 and 3).

JP 2011-65380 A

"Important sentence extraction using Support Vector Machine" Transactions of Information Processing Society of Japan 44 (8), 2230-2243, 2003 "Classification of opinions based on text and behavior in online debate" The 24th National Conference of Artificial Intelligence (JSAI2010), No. 3A4-3, 2010. "Retrieving Opinion Sentences from Blogs" Information Processing Society of Japan Research Report. Natural Language Processing Study Group Report 2006 (124), 121-128, 2006

  Patent Document 1 and Non-Patent Documents 1 to 3 determine whether a sentence belongs to a predetermined sentence category based on not only the statistical characteristics of the surface layer of the expression but also the meaning. Non-Patent Document 1 is to reduce the number of sentences to be extracted while maintaining the word coverage. Patent Document 1 and Non-Patent Documents 2 and 3 classify opinions on a certain object into approval / disapproval and positive / negative.

  Currently, posting and browsing microblogs while watching TV programs has begun to spread. However, it is difficult to browse all opinions comprehensively when browsing opinions on the topic of a television program. Therefore, there is a need for a device that narrows down representative comments according to the type of opinion from posted comments on television programs. For example, with the spread of Twitter, people have enjoyed watching TV while watching viewers' tweets about TV programs. When displaying program-related tweets on a tablet terminal, the display area of the tablet is limited, so it is necessary to narrow down the tweets to be displayed from a large number of tweets.

  However, when the prior art is applied to a text set for a specific topic, it is only possible to extract a summary or classify the text as an approval or disagreement. There was a problem.

  Therefore, an object of the present invention is to provide a comment extracting method and program for extracting a representative opinion from a text set for a specific topic.

In order to achieve the above object, a representative comment extracting method according to the present invention calculates the importance of words included in the text set in the representative comment extracting method for extracting a representative comment from a text set including a plurality of comments . A keyword extraction step of extracting, from the text set , words having the highest N (N: integer greater than or equal to N) words in the text set as key words indicating the characteristics of the text set; and the plurality of comments and a clustering step of clustering based on the key words extracted by the important word extracting step, from each cluster divided by the clustering step, a representative comments extracting a representative comments one by one, at least each Have

  In the important word extracting step, it is preferable that a comment of a contributor set in advance by a user is extracted from the text set, and an important word indicating a characteristic of the extracted text set is further extracted.

In the important word extracting step, it is preferable that a text set including a comment posted by a user is extracted from the text set, and an important word indicating the characteristics of the extracted text set is further extracted.

  The important word extracting step extracts a word from the text set by morphological analysis, obtains a TF-IDF value of the word, and extracts top N (N: integer of 1 or more) words as the important word. It is also preferable to do.

The clustering step generates a vector by vectorizing the comment based on the number of times the important word is included, calculates a similarity between the comments at an angle between the vectors, and an approximate similarity is It is also preferable to combine comments within a certain range into one cluster.

  It is also preferable to calculate the similarity after weighting the vector with an approximate topic word with the top M (1 ≦ M ≦ N) of the important words as topic words.

  In the representative comment extracting step, the comment is scored based on the number of times the important word is included, and the top L (an integer equal to or greater than L: 1) comments of the score are extracted as representative comments. Is also preferable.

  The representative comment extracting step preferably extracts a comment including a feature word set in advance or a sentence set in advance as a representative comment.

  The representative comment extracting step preferably extracts representative comments from the clusters in the order of the number of comments in each cluster, and outputs the representative comments in the extracted order.

  In addition, the representative comment extraction step extracts a representative comment from a cluster having the largest number of comments in each cluster, and thereafter extracts a representative comment from a cluster including a comment having the highest similarity to the representative comment, It is also preferable to output representative comments in the extracted order.

  The representative comment extraction step preferably extracts representative comments from the clusters in the order of the number of comments posted by users in each cluster, and outputs the representative comments in the extracted order.

The program according to the present invention for achieving the above object, from the text set including a plurality of comments, a computer to extract a representative comments, calculates the importance of words the text set comprises, from the text set, the Important word extracting means for extracting words having the highest importance (N: integer greater than or equal to N) in the text set as important words indicating the characteristics of the text set, and extracting the plurality of comments from the important word Clustering means for clustering based on the important words extracted by the means, and representative comment extracting means for extracting at least one representative comment from each cluster divided by the clustering means .

  According to the present invention, since a representative opinion can be preferentially browsed from a text set for a specific topic such as a microblog, browsing efficiency can be improved. For this reason, a representative opinion of a microblog such as Twitter can be displayed in a limited display area on the tablet terminal. It is also possible to extract opinions from different viewpoints by setting feature words in advance.

The processing flowchart of this invention is shown. Here is an example of a microblog. Indicates important words extracted from microblogs. Indicates the weight of the topic word. An example of microblog vector display is shown. Indicates the cosine similarity between microblogs. The clustering result based on the similarity of the microblog is shown. Extraction of representative microblogs by microblog scoring is shown. The representative opinion of the extracted microblog is shown.

  The best mode for carrying out the present invention will be described in detail below with reference to the drawings. FIG. 1 shows a processing flowchart of the present invention. Hereinafter, the processing of the present invention will be described in detail according to the flowchart.

と表せる。
ステップ４：重みを考慮したマイクロブログベクトルを生成。各マイクロブログに対して、重要語（Ｉ_１，Ｉ_２，・・・，Ｉ_Ｎ）を要素としたマイクロブログベクトルを生成する。各マイクロブログ中に重要語が含まれる回数をカウントし、マイクロブログをベクトル形式で表す。例えば、マイクロブログ１をベクトル形式Ｃ_１で表すと、
Ｃ_１＝｛Ｃ_１に含まれるＩ_１の個数，Ｃ_１に含まれるＩ_２の個数，・・・，Ｃ_１に含まれるＩ_Ｎの個数｝
＝｛Ｃ_１１，Ｃ_１２，・・・，Ｃ_１Ｎ｝
となる。例えば、マイクロブログ１に、Ｉ_１が１つ、Ｉ_４が１つ含まれる場合、
Ｃ_１＝｛１，０，０，１，０，０，０，０｝
と表せる。さらに，ステップ３で求めた重み行列Ｗをかけて重みつきマイクロブログベクトルＣ’_１を生成する。
Ｃ’_１＝Ｃ_１Ｗ
例えば、Ｃ_１が上記のような例である場合、
Ｃ’_１＝Ｃ_１Ｗ

＝｛Ｗ（Ｔ_１），０，０，Ｗ（Ｔ_４），０，０，０，０｝
となる。
ステップ５：マイクロブログのクラスタリング。マイクロブログを重みつきベクトルを用いて、複数のクラスタにまとめる。本実施形態では以下のように行う。ステップ４で作成した重みつきベクトルを用いて、各マイクロブログ間の類似度を計算する。例えば、コサイン類似度を用いて、マイクロブログ間の類似度を算出する。
マイクロブログ１とマイクロブログ２の類似度は、
ｃｏｓ（Ｃ’_１，Ｃ’_２）＝（Ｃ’_１１＊Ｃ’_２１＋Ｃ’_１２＊Ｃ’_２２＋・・・＋Ｃ’_１Ｎ＊Ｃ’_２Ｎ）／｛ｓｑｒｔ（Ｃ’_１１ ^２＋Ｃ’_１２ ^２＋・・・＋Ｃ’_１Ｎ ^２）＊ｓｑｒｔ（Ｃ’_２１ ^２＋Ｃ’_２２ ^２＋・・・＋Ｃ’_２Ｎ ^２）｝
と計算できる。これにより、ある閾値以上の類似度をもつマイクロブログは、同一クラスとみなし、クラスタを作成する。もしくは、クラスタ数がαになるまで、類似度が高いマイクロブログを同じクラスタに分類していく。
ステップ６：マイクロブログのスコアリング。マイクロブログを重要語が含まれる回数に基づいてスコアリングする。本実施形態では、ステップ４で求められたマイクロブログの重みつきベクトルの各成分の和を求めることで、マイクロブログにスコアを付ける。
例えば、
マイクロブログ１のスコア＝Ｃ’_１１＋Ｃ’_１２＋・・・＋Ｃ’_１ｎ
となる。
ステップ７：代表マイクロブログを抽出。クラスタ毎に、ステップ６で求めたスコアの上位Ｌ個のマイクロブログを代表意見として抽出する（Ｌは、あらかじめ定めた１以上整数）。また、スコアが最も高いマイクロブログを代表意見として抽出することのできる。これにより、近い意見は自動的に省かれ、様々な観点に基づいた意見が抽出できる。 Step 1: Collect target microblogs. Collect microblogs for certain things using keywords. For example, when collecting Twitter opinions on a TV program, search for TV station names, program names, and program-specific hash tags as keywords, and extract the opinions of the target program.
Step 2: Extract important words. From the microblog collected in step 1, important words indicating the characteristics of the microblog are extracted. In the present embodiment, important words are extracted as follows. The microblog collected in step 1 is subjected to morphological analysis, and a TF-IDF (Term Frequency-Inverse Document Frequency) value is obtained for the extracted word. The top N words having the TF-IDF value are defined as important words (I ₁ , I ₂ ,..., I _N ) (N is a predetermined integer of 1 or more). Each TF-IDF value of the important word is represented as S (I ₁ ), S (I ₂ ),..., S (I _N ).
Step 3: Topic word weight calculation. Among important words, an M word having a higher TF-IDF value is referred to as a topic word (T ₁ , T ₂ ,..., T _M ) (M is a predetermined integer of 1 or more and N or less). Topic words are weighted in the following way.
W (T ₁ ) = 1 + {S (I ₁ ) / (S (I ₁ ) + S (I ₂ ) +... + S (I _M ))}
W (T ₂ ) = 1 + {S (I ₂ ) / (S (I ₁ ) + S (I ₂ ) +... + S (I _M ))}
...
W (T _M ) = 1 + {S (I _M ) / (S (I ₁ ) + S (I ₂ ) +... + S (I _M ))}
The weight matrix is an N × N matrix,

It can be expressed.
Step 4: Generate a microblog vector considering the weight. For each microblog, a microblog vector having key words (I ₁ , I ₂ ,..., I _N ) as elements is generated. Count the number of times that each microblog contains an important word and represent the microblog in vector format. For example, if microblog ₁ is represented in vector format C1,
C ₁ = {number of _{I 1} included in _{C 1,} the number of _{I 2} contained in the _{C 1,} · · ·, the number of _{I N} included in _{C 1}}
= {C ₁₁ , C ₁₂ ,..., C _1N }
It becomes. For example, if microblog 1 contains 1 I _{1 and} 1 I ₄ ,
C ₁ = { _1, 0, 0, _{1, 0,} 0, 0, 0}
It can be expressed. Further, a weighted microblog vector C ′ ₁ is generated by multiplying the weight matrix W obtained in step 3.
C ′ ₁ = C ₁ W
For example, if C ₁ is an example as above,
C ′ ₁ = C ₁ W

= {W (T ₁ ), 0, 0, W (T ₄ ), 0, 0, 0, 0}
It becomes.
Step 5: Microblog clustering. Group microblogs into clusters using weighted vectors. In this embodiment, the process is performed as follows. Using the weighted vector created in step 4, the similarity between each microblog is calculated. For example, the similarity between microblogs is calculated using the cosine similarity.
The similarity between microblog 1 and microblog 2 is
cos (C ′ ₁ , C ′ ₂ ) = (C ′ ₁₁ * C ′ ₂₁ + C ′ ₁₂ * C ′ ₂₂ +... + C ′ _1N * C ′ _2N ) / {sqrt (C ′ ₁₁ ² + C ′ ₁₂ ² + ... + C ′ _1N ² ) * sqrt (C ′ ₂₁ ² + C ′ ₂₂ ² + ... + C ′ _2N ² )}
Can be calculated. As a result, microblogs having a similarity equal to or greater than a certain threshold are regarded as the same class, and a cluster is created. Or, until the number of clusters reaches α, microblogs with high similarity are classified into the same cluster.
Step 6: Microblog scoring. Scoring microblogs based on the number of times important words are included. In this embodiment, the microblog is scored by obtaining the sum of the components of the weighted vector of the microblog obtained in step 4.
For example,
Microblog 1 score = C ′ ₁₁ + C ′ ₁₂ +... + C ′ _1n
It becomes.
Step 7: Extract representative microblogs. For each cluster, the top L microblogs of the score obtained in step 6 are extracted as representative opinions (L is a predetermined integer of 1 or more). In addition, the microblog with the highest score can be extracted as a representative opinion. Thereby, close opinions are automatically omitted, and opinions based on various viewpoints can be extracted.

  Also, before step 2, it is possible to classify the microblog as positive / negative and extract important words from the classification result. In this case, it is possible to easily extract a representative opinion as a positive or negative opinion.

  In step 6, weighting is performed in consideration of feature words (for example, “w” and “(laugh)”) set in advance, such as a user's speech history and reference URL, or sentences set in advance. It is also possible to increase the score of the microblog that includes it and extract it with priority. In this case, it is possible to extract a microblog according to the user's preference.

  Further, in step 7, it is possible to omit those having a microblog score of 0 from the representative opinion. A microblog having a score of 0 does not include any important words, and therefore, it is possible to exclude microblogs that are not closely related to a specific topic.

Examples of the present invention are shown below. FIG. 2 shows an example of a microblog. The microblog described here is a part of the remarks about television programs. In fact, there are more than 10 microblogs, but only a few are listed. FIG. 3 shows important words extracted from the microblog. The important word in the case of N = 8 and the TF-IDF value are shown in parentheses. Since FIG. 2 is a part of the microblog, important words not shown in FIG. 2 are also included in FIG. FIG. 4 shows the weights of topic words. The weight when M = 3 is shown. FIG. 5 shows an example of a weighted vector display of a microblog. Each example shows a vector display of the microblog of FIG. FIG. 6 shows the cosine similarity between microblogs. Note that C ₁ C ₂ (0.81) indicates cos (C ₁ , C ₂ ) = 0.81.

FIG. 7 shows a clustering result based on the similarity of the microblog. Here, the result of dividing into six clusters is shown. A dotted line shows a cluster. C ₁ , C ₂ , C ₃ , C ₆ , and C ₇ are grouped into one cluster because of their high similarity. A microblog surrounded by a square line excluding C ₄ and C ₁₀ indicates that the microblog includes the important word in FIG. FIG. 8 shows representative microblog extraction by microblog scoring. _Here, from among the _{C 1, C 2, C 3} , C 6, C 7, a large _{C 7} most score is selected. FIG. 9 shows a representative opinion of the extracted microblog.

As in this example, a representative opinion C ₇ including the most important word can be displayed from C ₁ , C ₂ , C ₃ , C ₆ , and C ₇ having the same contents. Moreover, it is also possible to display except C ₄ and C ₁₀ which do not contain any important words.

  In the above embodiment, representative opinions are extracted based only on the contents of the microblog. If a microblogging contributor name can be used, the contributor name can also be used for representative opinion extraction.

  In the key word extraction in step 2 of the above embodiment, all microblogs are treated equally. When the poster name can be used, an important word is extracted from the microblog of the poster set in advance by the user, and this can also be used as the important word. Specifically, the user's preset microblog of the user is extracted, the extracted microblog is subjected to morphological analysis, a TF-IDF value is obtained for the extracted word, and the TF-IDF value is The top N ′ words are important words (N ′ is a predetermined integer of 1 or more). The union of the N important words extracted from the entire microblog and the N ′ important words obtained here can be used as the important word. In addition, the poster set in advance by the user can be, for example, a poster who has an influence on the remarks (Twitter has many followers, celebrities, etc.).

  It is also possible to extract an important word from a microblog posted by a user of the apparatus and make it an important word. Specifically, the microblog posted by the user of the present apparatus is extracted, the extracted microblog is subjected to morphological analysis, and the TF-IDF value is obtained for the extracted word. “A case word is an important word (N” is a predetermined integer of 1 or more). The union of the N important words extracted from the entire microblog and the N ″ important words obtained here can be used as an important word. Also, in combination with the above, N important words And the union of N ′ important words and N ″ important words can be used as important words.

  Further, the embodiment so far has not considered anything about the output order of the representative opinions output in step 7. However, the convenience for the user is improved when the microblog useful for the user is displayed first. In the following, an embodiment considering the output order of representative opinions will be described.

  In the first embodiment considering the output order, representative opinions are extracted from the clusters in the order of the number of comments of each cluster. For example, it is assumed that microblogs are grouped into four clusters by clustering, and the number of comments in each cluster is 5, 2, 1, 3. In this case, one representative opinion is extracted from the cluster having 5 comments, then one representative opinion is extracted from the cluster having 3 comments, and then 1 representative opinion is extracted from the cluster having 2 comments. Finally, one representative opinion is extracted from a cluster having one comment. Representative opinions are output in this extracted order.

  In the second embodiment in consideration of the output order, the representative opinion is first extracted from the cluster having the largest number of comments, and then the representative opinion is extracted from the cluster including the representative opinion having the highest similarity to the representative opinion. For example, it is assumed that microblogs are grouped into four clusters by clustering, and the number of comments in each cluster is 5, 2, 1, 3. In this case, first, one representative opinion is extracted from a cluster having five comments. If the representative opinion extracted from the cluster with five comments and the cluster including the comment with the highest similarity are clusters with two comments, then the representative opinion is one from the cluster with two comments. Are extracted. Assuming that the representative opinion extracted from the cluster having two comments and the cluster including the comment having the highest similarity are clusters having one comment, then the representative opinion is 1 from the cluster having one comment. Are extracted. Finally, one representative opinion is extracted from the cluster having three comments, and the representative opinions are displayed in this extracted order.

  In the third embodiment in consideration of the output order, the representative opinion is extracted based on the number of comments posted by the user. For example, if the microblog is grouped into four clusters by clustering, and the number of comments in each cluster is 5, 2, 1, 3, the user has 2 comments in the cluster with 3 comments. Assume that the user has posted one comment to a cluster with two comments. In this case, one representative opinion is first extracted from the cluster having three comments, and then one representative opinion is extracted from the cluster having two comments. Thereafter, one representative opinion is extracted from the cluster based on the first form or the second form, and the representative opinions are displayed in the extracted order.

  In the first to third embodiments in consideration of the output order, the representative opinion is extracted by the method described above. For example, as in step 6, scoring is performed based on the number of times the important word is included.

  A typical comment extraction method according to the present invention can be realized by a program that causes a computer to function as the above-described steps. These computer programs can be stored in a computer-readable storage medium or distributed via a network. Furthermore, the present invention can be realized by a combination of hardware and software.

  Moreover, all the embodiments described above are illustrative of the present invention and are not intended to limit the present invention, and the present invention can be implemented in other various modifications and changes. Therefore, the scope of the present invention is defined only by the claims and their equivalents.

Claims (12)

In a representative comment extraction method for extracting a representative comment from a text set including a plurality of comments,
The importance of the words included in the text set is calculated, and from the text set, the words having the highest N (N: integer of 1 or more) importance in the text set are the important words indicating the characteristics of the text set. A keyword extraction step to extract as
A clustering step of clustering the plurality of comments based on the important words extracted in the important word extraction step ;
A representative comment extracting step of extracting at least one representative comment from each cluster divided in the clustering step ;
A representative comment extraction method characterized by comprising: The important word extracting step extracts from the text set a text set including a user's preset comment set by a user, and further extracts important words indicating the characteristics of the extracted text set. The representative comment extraction method according to claim 1. 2. The important word extracting step extracts a text set including a comment posted by a user from the text set, and further extracts a key word indicating characteristics of the extracted text set. Alternatively, the representative comment extraction method described in 2.   In the important word extraction step, a word is extracted from the text set by morphological analysis, a TF-IDF value of the word is obtained, and top N (N: integer of 1 or more) words are extracted as the important word. The representative comment extracting method according to any one of claims 1 to 3. The clustering step, based on the number of times that includes said key word, the comment vectorization generates vector, an angle between the vectors, calculates the similarity between the comments, the degree of similarity is predetermined range 5. The representative comment extraction method according to claim 1, wherein the following comments are grouped into one cluster. Examples important word-level M (1 ≦ M ≦ N) pieces topic words, the vector after weighting by the topic words, representative comments extract according to claim 5, characterized in that to calculate the degree of similarity Method.   In the representative comment extracting step, the comment is scored based on the number of times the important word is included, and the top L (an integer of L: 1 or more) comments of the score are extracted as representative comments. The representative comment extracting method according to any one of claims 1 to 6.   The representative comment extraction according to any one of claims 1 to 7, wherein the representative comment extraction step extracts a comment including a feature word set in advance or a sentence set in advance as a representative comment. Method.   9. The representative comment extraction step according to claim 1, wherein the representative comments are extracted from the clusters in the order of the number of comments in each cluster, and the representative comments are output in the extracted order. The representative comment extraction method described.   In the representative comment extraction step, a representative comment is extracted from a cluster having the largest number of comments in each cluster, and thereafter, a representative comment is extracted and extracted from a cluster including a comment having the highest similarity to the representative comment. 7. The representative comment extracting method according to claim 5, wherein the representative comments are output in order.   The representative comment extracting step extracts representative comments from the clusters in the order of the number of comments posted by users in each cluster, and outputs the representative comments in the extracted order. The representative comment extraction method according to any one of the above items. A computer that extracts representative comments from a text set containing multiple comments.
The importance of the words included in the text set is calculated, and from the text set, the words having the highest N (N: integer of 1 or more) importance in the text set are the important words indicating the characteristics of the text set. A keyword extraction means for extracting as
Clustering means for clustering the plurality of comments based on the important words extracted by the important word extracting means ;
Representative comment extracting means for extracting at least one representative comment from each cluster divided by the clustering means ;
A program characterized by making it function.

Images