JP4128209B2 - Keyword extraction system - Google Patents

Description

  The present invention relates to a keyword extraction system, and more particularly to a system for automatically extracting important keywords worthy of attention from a large amount of digitized document data.

With the spread of personal computers and the Internet, and the development of electronic filing technology, an environment where a large amount of digitized document data can be used is being prepared. On the other hand, important keywords are automatically selected from a vast amount of information. There is a need for a system to extract.
For example, in the case of a robot-type search engine, it is possible to improve search accuracy by extracting important keywords from a large number of Web pages published on the Internet and indexing them. Further, in a company, it is necessary to list keywords in each data in order to effectively use knowledge data accumulated in the company. Even on an individual basis, it is possible to arrive at the necessary literature by extracting key words from a large amount of research paper data and patent data.

For this reason, various keyword extraction systems have been proposed and put into practical use. For example, Non-Patent Document 1 discloses a service for listing keywords in a document using an automatic terminology extraction algorithm called “TermExtract”. Non-Patent Document 2 describes a keyword extraction system called “tea bowl”.
Technical term (keyword) automatic extraction service / Welcome to “Kotosen Web” [Search November 2, 2006] Internet URL: http://www.nttdata.co.jp/release/2005/081500-01.html Morphological analysis system tea bowl [searched November 2, 2006] Internet URL: http://chasen.naist.jp/hiki/ChaSen/

By using such an existing system, it has been possible to automatically extract keywords from document data.
However, many of the conventional keyword extraction systems are based on a single extraction algorithm, and each extraction algorithm has a unique habit (advantageous / unprofessional), so extraction depends on the description content and description format of the document data. There was a problem that leakage was likely to occur.
In addition, the conventional keyword extraction system is a mechanism for determining the importance of a word in the target document data, and since it is not compared with other multiple document data, the importance of the extracted keyword is not objective. There was a problem that it was not secured.

  The present invention has been devised to solve the above-mentioned problem. When automatically extracting important keywords from a large amount of digitized document data, a combination of a plurality of extraction algorithms is combined to prevent extraction. The first object is to realize a system that can reduce the occurrence of the above. A second object is to realize a system that can objectively secure the importance of a keyword by considering the appearance of each keyword in other document data when extracting the keyword.

In order to achieve the above object, the keyword extraction system according to claim 1 is a keyword extraction system for extracting a keyword from a document storage unit storing a plurality of document data and storing the keyword in a predetermined storage unit. A plurality of filters that extract keyword candidates based on unique extraction criteria, and a majority filter that matches keyword candidates extracted by each filter and certifies keyword candidates extracted by at least half of the filters as keywords, One of the above-mentioned filters for extracting keyword candidates is (1) extracting nouns contained in each document as attention words, (2) calculating the appearance frequency of each attention word in all documents, and (3) each attention word. Expand the range to the morpheme before and / or after the word and include it in all documents of the word of interest including this expanded range. (4) If the appearance frequency calculated by the process in (3) above is a predetermined number or more, the range is further expanded to the previous or subsequent morpheme, and this expanded range is included. The process of calculating the appearance frequency of all the attention words in all the documents is repeated until the appearance frequency becomes less than the predetermined number. (5) Among the attention words including the first attention word and the extended range, Is selected as a keyword candidate .
Here, “morpheme” refers to the smallest linguistic unit having meaning. For example, when “my name is Suzuki” is broken down into morphemes, “I (pronoun)” “no (particle)” “name (general noun)” “ha (counsel)” “Suzuki (proprietary noun)” “ Is (auxiliary verb) ".

The keyword extraction system according to claim 2 is the system according to claim 1, wherein one of the filters for extracting the keyword candidates searches for a predetermined dependency expression included in each document, and the dependency extraction is performed. It is characterized in that at least a part of the expression is selected as a keyword candidate.

The keyword extraction system according to claim 3 is the system according to claim 1 or 2, wherein one of the filters for extracting the keyword candidates searches for a predetermined delimiter included in each document, and the delimiter A character string surrounded by characters is selected as a keyword candidate.

The keyword extraction system according to any one of claims 1 to 3 , wherein a plurality of filters are used to individually extract keyword candidates from document data, and at least half of these are extracted by a filter. Since it is provided with a mechanism for identifying a keyword as important, it is possible to prevent an important keyword from being missed and to prevent an unimportant noise from being mixed into the keyword.
In addition, when extracting keyword candidates, it has a mechanism to determine whether or not to select an attention word as a keyword candidate based on the appearance frequency in a plurality of document data. On the other hand, it becomes possible to have objectivity.

  FIG. 1 is a block diagram showing a keyword extraction system 10 and a search system 11 using the same according to the present invention. The keyword extraction system 10 includes a document DB 12, a keyword extraction unit 14, and a keyword DB 16. Yes. In addition to the above, the search system 11 includes a relevance calculation unit 18, a keyword co-occurrence frequency table 20, a keyword combination frequency sum table 22, a keyword frequency sum table 24, a keyword relevance table 26, a proper noun. A DB 28 and a search processing unit 30 are provided.

  The keyword extraction unit 14, the relevance calculation unit 18, and the search processing unit 30 are realized by the CPU of the computer executing necessary processing according to the OS and a dedicated application program.

The document DB 12, the keyword DB 16, the keyword co-occurrence frequency table 20, the keyword combination frequency sum table 22, the keyword frequency sum table 24, the keyword relevance table 26, and the proper noun DB 28 are stored in the hard disk of the computer.
A large number of electronic data (text data) such as newspaper articles, academic journals, and papers is stored in the document DB 12 in advance. In the proper noun DB 28, a number of proper nouns such as company names, product names, service names, and person names are registered for each category.

  As shown in FIG. 2, the keyword extraction unit 14 includes a dependency expression extraction filter 32, a delimiter extraction filter 34, a character string frequency statistical filter 36, a TermExtract filter 38, and a majority decision filter 40.

Next, the keyword extraction process by the keyword extraction unit 14 will be described with reference to the flowchart of FIG.
First, the keyword extraction unit 14 applies a dependency expression extraction filter 32 to each document data stored in the document DB 12, and extracts a character string having a predetermined dependency expression from each document data (S10).
That is, the dependency expression extraction filter 32 is provided with a large number of dependency expression patterns “XX manufacturer”, “XX is the main force”, and “XX is produced” in advance. After the expression pattern that applies to is detected, a character string corresponding to “XX” is extracted as a keyword candidate.

  Next, the keyword extraction unit 14 applies a delimiter extraction filter 34 to each document data, and commas such as “XX”, “XX”, (XX), [XX],. The XX part surrounded by delimiters such as parentheses, spaces, tabs, etc. is extracted as a keyword candidate (S12).

Next, the keyword extraction unit 14 applies a character string frequency statistical filter 36 to each document data, and counts how many times each character string included in each document data appears, including other documents. A character string having an appearance frequency is extracted as a keyword candidate (S14).
First, as shown in FIG. 4, the character string frequency statistical filter 36 pays attention to a noun (here, “DVD”) in the document, and the attention word “DVD” appears in each document data stored in the document DB 12. Add up the number you want. Next, the character string frequency statistical filter 36 expands the range to the morpheme before and after this attention word, totals the frequency that appears in all the documents, and the appearance frequency becomes less than a certain value (for example, 20 or less). Stop character range expansion at this point.

  For example, since the appearance frequency of “done DVD” including the previous morpheme of the DVD is as low as “2”, the range is not expanded to the previous morpheme. On the other hand, since the appearance frequency of “DVD recorder” including the next morpheme of DVD is as many as “862”, the appearance frequencies of “DVD recorder” including the next morpheme are tabulated. Since the appearance frequency is as low as “5”, the expansion of the range to subsequent morphemes is stopped.

Next, the character string frequency statistical filter 36 extracts “DVD” and “DVD recorder” as keyword candidates because they have an appearance frequency within a predetermined range (for example, 20 to 5,000). On the other hand, “done DVD” and “in the DVD recorder” are out of the above range, and are excluded from keyword candidates.
This is because, if the frequency of occurrence is less than 20 in all documents, it is not an important word in the first place, and if it exceeds 5,000, it is considered a general word or general word with no features. The amount is adjusted as appropriate according to the amount of use and the purpose of use of the search system.

  By the way, since it takes an enormous amount of time to count the appearance frequency of each character string included in a large amount of document data stored in the document DB 12, as shown in FIG. An index (so-called transposed index) is generated that lists whether or not each morpheme appearing in all document data is present in each document data. For this reason, the keyword extracting unit 14 can acquire the appearance frequency in a relatively short time by referring to the index.

Next, the keyword extracting unit 14 applies the TermExtract filter 38 to the document data stored in the document DB 12, and extracts a character string having a score equal to or higher than a predetermined value from each document data as a keyword candidate (S16).
This TermExtract is a string extraction algorithm devised to automatically extract technical terms from a specialized corpus (a huge text data consisting mainly of natural language sentences collected mainly for research purposes). A function is provided for extracting single nouns and compound nouns from the document data as candidate words, and calculating the respective importance based on the appearance frequency and the connection frequency of each candidate word. Since this TermExtract itself is a known technique, further explanation is omitted.

Next, the keyword extraction unit 14 inputs each keyword candidate extracted by the dependency expression extraction filter 32, the delimiter extraction filter 34, the character string frequency statistical filter 36, and the TermExtract filter 38 to the majority filter 40, and narrows down the keywords.
The majority filter 40 matches the keyword candidates listed by each filter, recognizes those listed as keyword candidates by two or more filters as final keywords, and stores them in the keyword DB 16 (S18).

  In this way, by using the four filters of the dependency expression extraction filter 32, the delimiter extraction filter 34, the character string frequency statistical filter 36, and the TermExtract filter 38, important words are leaked when keywords are extracted from document data. In addition to this, it is possible to prevent unnecessary keywords (noise) from being mixed by narrowing down using the majority filter 40.

As described above, the keyword candidate selected by two or more of the four filters is recognized as an official keyword, and selection by three or more filters may be a requirement for keyword recognition. it can.
Further, the number of filters is not limited to the above, and other effective keyword candidate extraction filters may be provided in the keyword extraction unit 14.

Next, according to the flowchart of FIG. 6, the relevance calculation process between the keywords by the relevance calculation unit 18 will be described.
First, the relevance calculation unit 18 aggregates the co-occurrence frequencies of each keyword in each document data, and generates a keyword co-occurrence frequency table 20 (S20).
FIG. 7 shows a specific example of the keyword co-occurrence frequency table 20. The appearance frequency of each keyword KW-1 to n is described for each document D1 to Dn stored in the document DB 12.

ただし、文書データの分量及びキーワードの総数が多い場合には膨大な計算量が発生し、多くの処理時間を要することとなる。
そこで、この実施の形態では、キーワード共起頻度表20に基づいてキーワード組合せ頻度総和表22及びキーワード頻度総和表24を生成することにより、計算工程の簡素化を図っている。 Here, the degree of association between a keyword X and Y can be theoretically calculated by substituting the appearance frequency of X and Y described in the keyword co-occurrence frequency table 20 into i of Equation 1. is there.

However, when the amount of document data and the total number of keywords are large, an enormous amount of calculation occurs, and a lot of processing time is required.
Therefore, in this embodiment, the calculation process is simplified by generating the keyword combination frequency summation table 22 and the keyword frequency summation table 24 based on the keyword co-occurrence frequency table 20.

FIG. 8 illustrates the procedure. In this case, the keyword co-occurrence frequency table 20 describes the appearance frequencies of the keywords KW-1 to KW-5 in the document D1, and among them, the appearance frequencies of KW-3 and KW-4 are 0. The combination of keywords for which the relevance is to be actually calculated is the following three patterns.
(KW-1, KW-2), (KW-1, KW-5), (KW-2, KW-5)
Next, the degree-of-relevance calculation unit 18 generates a keyword combination frequency sum table 22 describing values multiplied by the appearance frequency for each combination, and a keyword frequency sum table 24 describing values obtained by squaring the appearance frequency of each keyword. (S22, S24).

In the keyword combination frequency summation table of FIG. 8, only the value for the document D1 is described. However, the same processing is executed for each document, and the values are added based on the result. Is equivalent to the numerator of Equation 1.
Similarly, in the keyword frequency total table of FIG. 8, only the value for the document D1 is described, but the value obtained by squaring the appearance frequency of each keyword in each document is added, and the final value of each keyword is calculated. By obtaining the square root, a value corresponding to the denominator of Equation 1 is obtained.

As a result, as shown in FIG. 9, the relevance between the keywords can be calculated relatively easily, and the value is described in the keyword relevance table 26 (S26).
As described above, a combination pattern between keywords is extracted for each document, the product and the square value of each keyword are obtained, and the value of each document is added to obtain a keyword having a value of 0. It is possible to omit the calculation processing related to.
For this reason, it is realistic to calculate the degree of association between all keywords without being limited to the company name as in the search system of Patent Document 1.

Further, when new document data is added to the document DB 12, data related to each keyword in the new document data is added to the keyword combination frequency sum table 22 and the keyword frequency sum table 24, and the existing total value is added. By adding the additional values, the degree of association between keywords can be easily recalculated.
Even when the influence of obsolete document data is excluded, the data related to each keyword in the document data is deleted from the keyword combination frequency summation table 22 and the keyword frequency summation table 24, and the deleted value from the existing total value By subtracting, it is possible to easily maintain the degree of association between keywords in the latest state.

Next, a search processing procedure in the system 10 will be described with reference to the flowchart of FIG.
First, when the user inputs a search word from the terminal device α, the search processing unit 30 that has received the search word (S40) refers to the keyword relevance table 26 as shown in FIG. A keyword having a similarity is specified, and a list of keywords having a predetermined degree of relevance to the keyword is extracted (S42).
Next, the search processing unit 30 refers to, for example, the company name DB in the proper noun DB 28 and extracts the company name included in the list (S44).
The extracted list of company names is transmitted to the terminal device α as a company list closely related to the search term (S46).

As a result, the user can recognize a company closely related to the input search word (for example, current affair term), and can use it for the judgment of investment behavior.
If a person name DB is designated as the proper noun DB 28, a person closely related to the input search word can be picked up.

However, a list of keywords closely related to the search term may be returned to the terminal device α as it is without matching with the company name DB or the person name DB.
Thereafter, when the user designates a specific keyword in the keyword list as a search word, the search processing unit 30 further extracts a list of keywords having a predetermined relationship with the keyword and transmits it to the terminal device α. .
As a result, the user can expand the search range in a chain from related words to related words, and can be expected to arrive at an unexpected keyword.

When the user designates a specific keyword in the search result list and requests the presentation of a document as the basis thereof (S48), the search processing unit that accepts the request (S48) assigns the search word and the keyword as shown in FIG. Based on the keyword co-occurrence frequency table 20, a list of document numbers in which co-occurrence has occurred is generated (S50).
Next, the search processing unit 30 searches the document DB 12 based on the document number list, generates a list of document texts, and transmits it to the terminal device α (S52, S54).
As a result, the number, title, abstract, date, etc. of the document in which the search word and the keyword appear simultaneously are displayed in a list on the display of the terminal device α.

When the user selects one of them, the search processing unit 30 extracts the corresponding document data from the document DB 12 and transmits it to the terminal device α.
As a result, the user can browse the contents of the document data and individually confirm the relevance between the search term and the keyword.

It is a block diagram which shows the function structure of the keyword extraction system which concerns on this invention, and the search system using the same. It is a block diagram which shows the function structure of a keyword extraction part. It is a flowchart which shows a keyword extraction process. It is explanatory drawing which shows operation | movement of a character string frequency statistical filter. It is explanatory drawing which shows a mode that the morpheme index is formed in document DB. It is a flowchart which shows the related degree calculation process between keywords. It is explanatory drawing which shows an example of a keyword co-occurrence frequency table. It is explanatory drawing which shows the method of simplifying a relevance calculation process. It is explanatory drawing which shows a mode that a keyword relevance table is produced | generated based on a keyword combination frequency total table and a keyword frequency total table. It is a flowchart which shows the procedure of a search process. It is explanatory drawing which shows a mode that a company name list | wrist is extracted based on a search term. It is explanatory drawing which shows a mode that the basis of the relevance degree between a search word and a specific keyword is shown.

Explanation of symbols

10 Keyword extraction system
11 Search system
12 Document DB
14 Keyword extractor
16 Keyword DB
18 Relevance calculator
20 Keyword co-occurrence frequency table
22 Keyword combination frequency summation table
24 Keyword Frequency Summation Table
26 Keyword Relevance Table
28 proper noun DB
30 Search processing section
32 Dependency Expression Extraction Filter
34 Delimiter extraction filter
36 String frequency statistics filter
38 TermExtract filter
40 Majority filter

Claims (3)

A keyword extraction system for extracting a keyword from a document storage unit storing a plurality of document data and storing the keyword in a predetermined storage unit,
Multiple filters that each extract keyword candidates based on unique criteria,
A majority voting filter that matches keyword candidates extracted by each filter and certifies keyword candidates extracted by at least half of the filters as keywords;
One of the filters that extract the keyword candidates is
(1) Extract nouns contained in each document as attention words,
(2) Calculate the appearance frequency of each attention word in all documents,
(3) Expand the range to the morpheme one and the next before each attention word, and calculate the appearance frequency of the attention word including this expansion range in all documents,
(4) If the appearance frequency calculated by the processing in (3) above is a predetermined number or more, the range is further expanded to the previous or subsequent morpheme, and all documents of the attention word including this expanded range The process of calculating the appearance frequency in the inside is repeated until the appearance frequency is less than a predetermined number,
(5) A keyword extraction system characterized by selecting, as a keyword candidate, a word having an appearance frequency within a predetermined range among the first attention word and the attention word including the extended range . One of the filters that extract the keyword candidates is
The keyword extraction system according to claim 1, wherein a predetermined dependency expression included in each document is searched, and at least a part of the dependency expression is selected as a keyword candidate. One of the filters that extract the keyword candidates is
The keyword extraction system according to claim 1 or 2, wherein a predetermined delimiter character included in each document is searched, and a character string surrounded by the delimiter character is selected as a keyword candidate.

Images