JP5269481B2 - Related keyword extraction device, program, and computer-readable recording medium - Google Patents

Description

The present invention relates to a related keyword extracting DeSo location and program and a computer-readable recording medium, in particular, the relevant keyword in the appropriate relationship with the keyword entered by the user, by analyzing the documents given on related keywords extracted DeSo location and program and a computer-readable recording medium for automatically extract.

  With the development of the Internet and other information media, anyone can easily transmit information, and blog articles and other documents written by individuals with various interests are distributed and stored on the Internet. It has come to be. Since these documents contain matters related to individual interests and interests, information that can be used for marketing such as consumer interests can be obtained by reading the contents. For example, by looking through a collection of documents including a specific product name, it is possible to find related matters that are of interest to people who are interested in the product.

  Since the above work is time consuming and labor intensive, there is a need to automatically extract words and phrases related to preset analysis target words by mechanically analyzing a given large collection of documents. To do.

  Many methods have been proposed for extracting related keywords of analysis target phrases input by users.

As a conventional technique, there is a technique of extracting words that co-occur more frequently as related keywords by analyzing the number of co-occurrence of words in a large number of documents (for example, see Patent Document 1).
JP-A-5-282367

  However, in the above conventional technique, since the related keyword is extracted from the document including the analysis target phrase set by the user, if the related keyword is not described in the same document as the analysis target phrase, it is not extracted as the related keyword. . For this reason, keywords that are highly relevant in content, such as keywords contained in different news articles in the same field, but cannot be extracted as related keywords, and only related keywords in a narrow range can be extracted. There was a problem.

  Moreover, in the above-described conventional technology, phrases that are frequently used in the same field such as “strike” and “strike” are also extracted as related phrases, and the fields in which the phrases are used are the same. However, there is a problem that phrases that are used differently as phrases are also extracted as related keywords.

The present invention has been made in view of the above points, and by analyzing a pattern in which a phrase is used in an outgoing document from each sender, the related keyword to be analyzed is used in a similar manner to the analysis target phrase. and to provide a related keyword extracting DeSo location and program capable of extracting focuses only words that a person.

  FIG. 2 is a principle configuration diagram of the present invention.

（但し、分析対象キーワードを含む文書を発信したことのある発信者数をＵｉ、Ｌ番目の発信者によって発信されたｗｋを含む文書の数をｄｆＬ（ｗｋ），情報源に含まれるＬ番目の発信者が発信した文書数をＮＬ，情報源に含まれるｗｋを含む文書の数をｄｆ（ｗｋ）とし、情報源に含まれる全文書数をＮａｌｌとし、ｗｋがある文書に現れる確率をｐ（ｗｋ）＝ｄｆ（ｗｋ）／Ｎａｌｌとする）により求める。 The present invention (Claim 1 ) is a related keyword extraction device that extracts and outputs a related keyword of an analysis target keyword input from a large amount of document information ,
Receiving an analyte keywords for which to extract the relevant topic from outside, information of the caller accesses the information source 270 for storing documents granted, you may have created a document containing an analyte keyword caller and related caller extraction means 220 to extract a set of,
The document set that was created by the caller obtained by the associated caller extracting unit 220 extracts access the information source 270 as documents related to the analysis target keyword, as the related word candidate phrases contained in the document Related phrase candidate extracting means 230 to extract ,
For each word / phrase wk of the related word / phrase candidate, when a document including the word / phrase wk has been transmitted to a sender who has transmitted the keyword to be analyzed, a positive score is obtained, and a document including the word / phrase wk is selected. If no transmission has been made, a negative score is calculated, and a positive score and a value obtained by adding up the negative scores for all the senders who have transmitted the analysis target keyword are calculated as the relevance of the phrase wk. Relevance calculating means 240 ,
For each phrase wk of the related phrase candidates, an average of the ratio of documents including the phrase wk in the total number of outgoing documents of the sender is obtained as a repeatability score for the sender who has transmitted the analysis target keyword. Similarly, the repeatability score of the analysis target keyword is obtained, the relevance of the phrase wk is equal to or higher than the first threshold, and the difference between the repeatability score of the phrase wk and the repeatability score of the analysis target keyword is equal to or less than the second threshold. If it has a repetition rate calculating means 250 for extracting the phrase wk as related keywords, the.
In the present invention (Claim 2), in the relevance calculation means, the relevance of the word wk is calculated.

(However, Ui is the number of senders who have transmitted a document including the keyword to be analyzed, dfL (wk) is the number of documents including wk transmitted by the Lth sender, and the Lth element included in the information source. The number of documents transmitted by the sender is NL, the number of documents including wk included in the information source is df (wk), the total number of documents included in the information source is Nall, and the probability that wk appears in a document is p ( wk) = df (wk) / Nall).

The present invention (claim 3), a computer, a related keyword extraction program for causing functions as a respective means associated keyword extracting device according to claim 1 or 2.

The present invention (Claim 4 ) is a computer-readable recording medium in which the related keyword extraction program according to Claim 3 is stored.

  As described above, according to the present invention, by performing an analysis focusing on the sender of each document, the user is interested in the analysis target phrase regardless of whether it appears in the same document as the analysis target phrase. Related keywords that people are similarly interested in can be extracted. Extract only related terms that are used in a manner similar to the keyword to be analyzed by extracting the words whose degree of repeated use in the group of outgoing documents for each sender is close to the degree to which the word to be analyzed is used repeatedly. It becomes possible.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

  FIG. 3 shows a configuration of a related keyword extracting device according to an embodiment of the present invention.

  The related keyword extraction device 200 shown in FIG. 1 includes a document database 270 in which a large amount of document information is accumulated, a keyword input device 210 for inputting a processing target keyword to be analyzed by the device, and related keyword information output by the device. A related keyword display device 260 to be displayed is connected.

  The related keyword extraction device 200 includes a related sender extraction unit 220, a related phrase candidate extraction unit 230, a relevance level calculation unit 240, and a repetition rate calculation unit 250.

  The document database 270 stores a large amount of documents with sender information. For example, the document database 270 can be constructed by recording documents on a site operated by an individual on the web as input along with the sender's information. Further, in the case of an information source in which new documents are sequentially updated, such as a diary site, even when the documents in the site are updated, collection may be performed assuming that a new document has been created. An existing web page search engine may be used in place of the document database 270 by regarding each page's domain URL as sender information. In the following, it is assumed that the document database 270 is used as an information source. FIG. 4 shows an example of document database information in the first exemplary embodiment of the present invention.

  When a processing target keyword is input from the keyword input device 210, the related caller extraction unit 220 starts operation, and accesses the document database 270 to acquire a list of sender information of documents including the processing target keyword. By removing duplicate contents from the acquired sender information, a list of senders who have sent documents including the keyword to be processed even once is created, and the information is related word / phrase candidates as shown in FIG. It accumulates in the sender buffer 231 of the extraction unit 230.

  The related phrase candidate extraction unit 230 starts operation when information is accumulated in the caller buffer 231 by the related caller extraction unit 220, and documents sent by each caller stored in the caller buffer 231 in the past. Information is obtained by accessing the document database 270, morphological analysis is performed on the body information of all the obtained document information, and the obtained part-of-speech information is aggregated to be accumulated in the sender buffer 231. A list of all the words included in the document sent by each sender is created and stored in the word list buffer 241 of the relevance calculation unit 240 as shown in FIG. At this time, only the nouns may be stored in the phrase list buffer 241 in the part-of-speech group obtained by the morphological analysis process in order to reduce the processing.

  The degree-of-association calculation unit 240 starts operation when information is accumulated in the phrase list buffer 241 by the related phrase candidate extraction unit 230, and has transmitted the analysis target keyword for all the phrases in the phrase list buffer 241. A word frequently used by a person is used as a related keyword, and a degree of relation representing the degree of the relation is calculated and output to the related keyword buffer 251 of the repetition degree calculation unit 250 as shown in FIG.

  Next, the flow of processing of the relevance calculation unit 240 will be described.

  FIG. 8 is a flowchart of processing of the relevance calculation unit according to the first embodiment of this invention.

Step 110) When information is accumulated in the phrase list buffer 241 by the related phrase candidate extraction unit 230, the related degree calculation unit 240 starts operation, extracts one word from the phrase list buffer 241, and calculates the degree of relevance calculation target. It is a phrase. Hereinafter, this relevance calculation target phrase is denoted as w _k . At this time, if the number of words included in the word list buffer 241 is N, _k in w _k takes a value from 1 to N.

Step 120) accesses the document database 270, acquires the total number of documents contained in the number and document database 270 of a document containing the _{w k,} determining the probability p _{(w k)} appearing in the documents where _{w k.} w _k the number of documents containing df _{(w k)} and then, when the total number of documents contained in the document database 270 and _{N all,} the probability appear in documents where _{w k} p _{(w k)} is the following formula (1) It is obtained by.

ステップ１３０） 文書データベース２７０にアクセスすることによって、発信者バッファ２３１に含まれる各発信者について、ｗ_ｋを含む文書を発信しているか否かを抽出し、ｗ_ｋを含む文書を発信している場合には、当該発信者の発信した文書数に応じた正のスコアを算出し、ｗ_ｋを含む文書を発信していない場合には発信した文書数に応じた負のスコアを算出し、得られた各発信者に対応するスコアを集計することにより、ｗ_ｋの処理対象語句に対する関連度スコア（Ｒ（ｗ_ｋ）と表記することとする）を算出する。

By accessing step 130) the document database 270, for each caller included in caller buffer 231, it extracts whether originated the document containing the w _k, is transmitting a document containing w _k In this case, a positive score corresponding to the number of documents transmitted by the sender is calculated, and when a document including w _k is not transmitted, a negative score is calculated according to the number of documents transmitted. The relevance score (denoted as R (w _k ) with respect to the processing target phrase of w _k is calculated by totaling the scores corresponding to the respective senders.

The number of callers included in the caller buffer is U _i , the number of documents including w _k sent by the Lth caller is df _L (w _k ), and the Lth caller in the document database 270 sends When expressed as document number _{N L,} relevance score R _{(w k)} for the processing target phrase _{w k} can be calculated by the following equation (2). The calculated relevance score R (w _k ) is stored in the relevance score buffer 242.

ステップ１４０） ステップ１３０で得られたｗ_ｋとＲ（ｗ_ｋ）との組を反復度算出部２５０の関連キーワードバッファ２５１に蓄積する。関連キーワードバッファ２５１に含まれる情報の例を図７に示す。

Step 140) The set of w _k and R (w _k ) obtained in step 130 is stored in the related keyword buffer 251 of the repetition rate calculation unit 250. An example of information included in the related keyword buffer 251 is shown in FIG.

  Step 150) It is confirmed whether or not the processing from Step 110 to Step 140 has been performed for all the words / phrases in the word / phrase list buffer 241. If any unprocessed words / phrases remain, the relevance score buffer 242 is emptied and the process goes to Step 110. Return and continue processing. If there is no unprocessed word remaining, the processing of the relevance calculation unit 240 ends.

  When the process of the relevance calculation unit 240 ends, the repetition degree calculation unit 250 starts its operation, and accesses the document database 270 for each word / phrase information included in the related keyword buffer 251 for each transmission source. The ratio of the number of documents included in the number of documents included in the set of documents sent from is obtained, and the set of documents sent from each source obtained in the same way for the obtained ratio and analysis target phrases The phrase within the predetermined range of the difference from the ratio of how much it is included in is regarded as a related keyword of the analysis target keyword and is output to the related keyword output device 260.

  FIG. 9 is a flowchart of the process of the repetition rate calculation unit in one embodiment of the present invention.

Step 201) When the process is started, the repetition degree calculation unit 250 extracts one phrase from the related keyword buffer 251 and sets it as a repetition degree calculation target phrase. Hereinafter, the repetition rate calculation target phrase is denoted as w _j .

Step 202) Next, a document group including w _j is obtained by accessing the document database 270, and the sender information included in the document group is tabulated without duplication, thereby obtaining a document including w _j. Get a list of callers who have sent

Step 203) By accessing the document database 270 for each sender included in the obtained sender list, the number of documents transmitted in the past and the number of documents including the phrase w _j are obtained, and the phrase w _j is included. A value obtained by dividing the number of documents by the number of documents transmitted in the past is obtained. The average of the obtained values for all the callers included in the caller list is calculated as the repetition rate of the phrase w _j .

Specifically, the number of callers included in the caller list is V _i , the number of documents sent by the _Lth caller in the past is N _L , and the phrase w _j sent by the Lth caller is included. When notation the number of documents and _df L _{(w j),} repeating scores T _{(w j)} of _{w j} can be calculated by the following equation (2).

ステップ２０４） 関連キーワードバッファ中の全ての語句についてステップ２０１〜ステップ２０３の処理を行ったかを確認し、未処理の語句が残っている場合には、ステップ２０１に移行し、処理を継続する。未処理の語句が残っていない場合には、ステップ２０５に移行する。

Step 204) It is confirmed whether or not the processing of Step 201 to Step 203 has been performed for all the words / phrases in the related keyword buffer. If any unprocessed words / phrases remain, the process proceeds to Step 201 and the processing is continued. If no unprocessed words remain, the process proceeds to step 205.

  Step 205) By applying the repeatability score calculation method of step 203 to the analysis target keyword, the repeatability score of the analysis target keyword is calculated.

Step 206) For each phrase for which the repeatability score is calculated, the difference between the repeatability score of the keyword to be analyzed and the repeatability score of the phrase is equal to or less than a predetermined threshold, and the relevance score (R (w) of the phrase _k )) is output to the related keyword display device 260 as related keywords that are related to the analysis target keyword and similar in usage.

  It is possible to construct the operation of each component of the configuration of FIG. 3 as a program and install it on a computer used as a relevance topic extraction device, or to distribute it via a network. .

  Further, the constructed program can be stored in a portable storage medium such as a hard disk, a flexible disk, or a CD-ROM, and can be installed or distributed in a computer.

  The present invention is not limited to the above-described embodiment, and various modifications and applications can be made within the scope of the claims.

  The present invention is applicable to a search technique, particularly a technique for extracting a keyword from a document on the Internet.

It is a figure for demonstrating the principle of this invention. Principle configuration diagram of the present invention It is a block diagram of the related topic extraction apparatus in one embodiment of this invention. It is an example of the information accumulate | stored in the document database in one embodiment of this invention. It is an example of the information accumulate | stored in the sender | caller buffer in one embodiment of this invention. It is an example of the information accumulate | stored in the phrase list buffer in one embodiment of this invention. It is an example of the information accumulate | stored in the related keyword buffer in one embodiment of this invention. It is a flowchart of the process of the relevance calculation part in one embodiment of this invention. It is a flowchart of the process of the repetition degree calculation part in one embodiment of this invention.

Explanation of symbols

200 Related Topic Level Extraction Device 210 Keyword Input Device 220 Related Caller Extraction Means, Related Caller Extraction Unit 230 Related Word Phrase Candidate Extraction Means, Related Word Phrase Candidate Extraction Unit 231 Caller Buffer 240 Relevance Level Calculation Means, Relevance Level Calculation Unit 241 List buffer 250 Repeatability calculation means, repeatability calculation unit 251 Related keyword buffer 270 Information source, document database

Claims (4)

A related keyword extraction device that extracts and outputs related keywords of analysis target keywords input from a large amount of document information ,
Receiving an analyte keywords for which to extract the relevant topic from outside, information of the caller accesses the information source for storing documents granted, you may have created a document containing the analyte keyword caller and related caller extracting means to extract a set of,
The document set that was created by the previous SL callers obtained in the previous SL associated caller extracting means extracts to access the information source as documents related to the analyte keywords, related phrases included in the document Related phrase candidate extracting means for extracting as a phrase candidate ;
For each word / phrase wk of the related word / phrase candidate, if a document containing the word / phrase wk has been transmitted to a sender who has transmitted the keyword to be analyzed, a positive score is included, and the word / phrase wk is included. If the document has never been transmitted, a negative score is calculated, and the positive score and the total value of all the senders who have transmitted the keyword to be analyzed are added to the positive score and the phrase wk. and degree-of-association calculation means for calculating a degree of association,
For each word / phrase wk of the related word / phrase candidate, an average of the ratio of documents including the word / phrase wk in the total number of outgoing documents of the caller is obtained as a repeatability score for the caller who has transmitted the keyword to be analyzed. Similarly, the repeatability score of the analysis target keyword is obtained, the relevance of the phrase wk is equal to or higher than the first threshold value, and the difference between the repeatability score of the phrase wk and the repeatability score of the analysis target keyword is the second An iterative degree calculating means for extracting the phrase wk as a related keyword when it is equal to or less than a threshold ;
The related keyword extracting device characterized by having . The relevance calculation means calculates the relevance of the word wk.

(However, Ui is the number of senders who have transmitted a document including the keyword to be analyzed, dfL (wk) is the number of documents including wk transmitted by the Lth sender, and the Lth element included in the information source. The number of documents transmitted by the sender is NL, the number of documents including wk included in the information source is df (wk), the total number of documents included in the information source is Nall, and the probability that wk appears in a document is p ( wk) = df (wk) / Nall)
Related keyword extracting device according to claim 1, wherein the determining by. Computer
Related keyword extraction program for causing a function as a respective means associated keyword extracting device according to claim 1 or 2. A computer-readable recording medium storing the related keyword extraction program according to claim 3 .

Images