JP5483716B2 - Keyword extraction apparatus and method, and program - Google Patents

Description

  The present invention relates to a keyword extraction apparatus and method, and a program.

  In order to classify or search a plurality of documents, there are techniques for extracting keywords from these documents. For example, in Patent Document 1, a word vector of each document is extracted, a set of documents having the maximum inner product is grouped as the same cluster, a word having a high appearance frequency extracted for each cluster is set as a characteristic keyword of the cluster, The keywords extracted from the subclusters included in each cluster are hierarchized as lower keywords of the cluster characteristic keywords.

JP 2009-3888 A

  In the above-described technique, since keywords are extracted based on the appearance frequency in the document, each word commonly used in documents belonging to any cluster is not a word characteristic of each cluster. There was a problem that it could be extracted as a keyword in the upper layer of the cluster. In addition, a large number of keywords are extracted, and there is a problem that it is difficult to narrow down keywords that should be upper and lower levels.

  The present invention has been made in order to solve the above-described problem, and an object of the present invention is to select characteristic words in a plurality of documents as upper-level keywords, extract hierarchical keywords, It is an object to provide a keyword extraction apparatus and method, and a program, which can appropriately narrow down the program.

In order to solve the above problems, the present invention extracts keywords from a plurality of documents, counts the number of documents in which each of the extracted keywords appears to obtain a cumulative count number, and the same combination of keywords appears. For a plurality of documents, assuming that the number of documents that appear is 1, a keyword extraction unit that obtains a unique count by counting the number of documents in which the extracted keyword appears, and the cumulative count obtained by the keyword extraction unit Among the keywords whose number is equal to or greater than a predetermined condition and whose unique count is equal to or greater than a predetermined condition, a keyword that is a single word is selected as a keyword in the highest hierarchy and obtained by the keyword extraction unit Cumulative count number is greater than or equal to a predetermined condition, or the unique count number is a predetermined condition Of the above keywords, a keyword analysis unit that selects a keyword that is a compound word as a keyword in a lower hierarchy, and a keyword in the lower hierarchy that appears in the same document as the keyword in the highest hierarchy, And a keyword layer generation unit that extracts the keyword as a second layer keyword under the keyword.
In the keyword extracting device described above, the keyword extracting unit sets the keyword to 1 for a plurality of documents in which the keyword in the same combination appears, with respect to the keyword having a cumulative count number equal to or greater than a predetermined number. The unique count number may be obtained by counting the number of the documents in which.

  Further, the present invention is the keyword extracting apparatus described above, wherein the keyword extracting unit deletes a predetermined unnecessary word from a document and extracts a keyword.

Further, the present invention is the above-described keyword extraction device, wherein the keyword hierarchy generation unit is configured to add the lower hierarchy of the lower hierarchy that is subordinate to any keyword in order from the second hierarchy for the lower hierarchy after the second hierarchy. Another keyword of a compound word that appears in the same document as the keyword and that is not under any of the keywords is acquired, and the acquired other keyword is further subordinate to the keyword of the lower hierarchy. It is characterized by repeating the extraction process.
In the keyword extraction device described above, when the accumulated count of the acquired other keyword is equal to or greater than a predetermined value, the acquired another keyword is further subordinate to the keyword of the lower hierarchy. May be extracted as

  The present invention is also a keyword extraction method executed by the keyword extraction device, wherein the keyword extraction unit extracts keywords from a plurality of documents, counts the number of the documents in which the extracted keywords appear, and accumulates them. A keyword extraction process for obtaining a unique count number by obtaining the count number and counting the number of documents in which the extracted keyword appears with respect to a plurality of documents in which the same combination of keywords appears. And a keyword analysis unit selects a keyword that is a single word among the keywords in which the cumulative count obtained in the keyword extraction process is equal to or greater than a predetermined condition and the unique count is equal to or greater than a predetermined condition. Select as the keyword of the highest hierarchy and A keyword analysis process for selecting a keyword that is a compound word as a lower-level keyword from keywords whose cumulative count obtained in the process is equal to or greater than a predetermined condition or whose unique count is equal to or greater than a predetermined condition, and keyword hierarchy generation A keyword hierarchy generation step of extracting a keyword of the lower hierarchy appearing in the same document as the keyword of the highest hierarchy as a keyword of the second hierarchy under the keyword of the highest hierarchy. This is a keyword extraction method.

  The present invention also provides a computer that extracts keywords from a plurality of documents, counts the number of the documents in which each of the extracted keywords appears to obtain a cumulative count, and a plurality of documents in which the same combination of keywords appears. The number of documents that appear is 1, a keyword extraction unit that counts the number of documents in which the extracted keyword appears to obtain a unique count number, and the cumulative count obtained by the keyword extraction unit is a predetermined condition Of the keywords whose unique count is equal to or greater than a predetermined condition, a keyword that is a single word is selected as a keyword of the highest hierarchy, and the cumulative count obtained by the keyword extraction unit is More than a predetermined condition, or the unique count number is more than a predetermined condition A keyword analysis unit that selects a keyword that is a compound word as a lower-level keyword from among a certain keyword, and the lower-level keyword that appears in the same document as the highest-level keyword is subordinate to the upper-level keyword. It is a program characterized by functioning as a keyword layer generation unit that extracts as a second layer keyword.

  According to the present invention, from a plurality of input documents, a keyword hierarchical structure in which each hierarchical level is appropriately narrowed down, with the broadest keyword characteristic in the document being the highest hierarchical level and the narrower keyword being the second hierarchical level or lower, is extracted. be able to. In particular, it is difficult to extract by keyword extraction based on appearance frequency as in the prior art, and it is possible to extract characteristic keywords in a document although the number of appearances is small.

It is a figure which shows the process outline | summary of the keyword extraction apparatus according to one Embodiment of this invention. It is a functional block diagram which shows the structure of the keyword extracting device by the same embodiment. It is a figure which shows the data structural example of the frequent keyword list by the same embodiment. The data structural example of the unique count keyword list by the embodiment is shown. It is a figure which shows the data structural example of the table for keyword extraction by the embodiment. It is a figure which shows the example of a data structure of the table for hierarchy by the same embodiment. It is a figure which shows the operation | movement sequence of the keyword extracting device by the same embodiment. It is a figure for demonstrating the counting method of the accumulation count number by the same embodiment, and a unique count number. It is a figure which shows the processing flow of the keyword extraction process to the 2nd hierarchy by the embodiment. It is a figure which shows the processing flow of the keyword correlation process of the upper hierarchy by the same embodiment, and a lower hierarchy. It is a figure which shows the processing flow of the lower hierarchy keyword extraction process after the 2nd hierarchy by the embodiment. FIG. 10 is a diagram illustrating a generation example of a frequent keyword list by the process illustrated in FIG. 9. It is a figure which shows the example of a production | generation of the upper hierarchy keyword list and the lower hierarchy keyword list by the process shown in FIG. It is a figure which shows the example of keyword hierarchy structure generation | occurrence | production to two layers by the process shown in FIG. It is a figure which shows the example of keyword hierarchy structure generation | occurrence | production after 2 hierarchy by the process shown in FIG.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.
FIG. 1 is a diagram showing a processing outline of a keyword extracting apparatus according to an embodiment of the present invention.
First, for example, when data of a plurality of documents to be subjected to keyword extraction, such as e-mail titles, are input, keywords are extracted from each document by removing unnecessary words, and the keywords appear for each extracted keyword. The cumulative count that is the number of documents to be counted is counted. Then, after generating a frequent keyword list that is a list of keywords having a cumulative count number of a predetermined value or more, a cumulative count keyword list that is a list of keywords having a higher cumulative count number is generated. Furthermore, for each keyword registered in the frequent keyword list, the unique count number that is the number of documents in which the keyword appears is counted, and the unique count number is a list of the higher-order keywords. Generate a list. The unique count is a method in which when the same keyword combination appears in a plurality of documents, the plurality of documents are virtually regarded as one document, and the number of appearing documents is counted once.

Subsequently, a single word appearing in both the cumulative count keyword list and the unique count keyword list is extracted as a top level keyword to generate an upper level keyword list, and at least one of the cumulative count keyword list and the unique count keyword list is generated. A compound word appearing in either one is extracted as a keyword in the second hierarchy to generate a lower hierarchy keyword list. This utilizes the fact that a single word consisting of one word is a broad sense word, while a compound word consisting of two or more words is a narrow sense word. After the extraction, the highest-level keyword and the second-level keyword included in the same document are associated with each other to be hierarchized.
Thereafter, for each extracted lower-level keyword, the process of extracting the keyword of the compound word that appears in the same document and registered in the frequent keyword list is further repeated as one lower-level keyword.

  Thereby, the keyword of each hierarchy including the keyword of the highest hierarchy can be extracted appropriately, and the comprehensiveness of a lower keyword is also improved. For example, in a regular report or the like, the same keyword is often included, so the unique count number is lower than the cumulative count number. On the other hand, keywords that often co-occur with other keywords and characteristic keywords that do not appear in many sentences have less decrease from the cumulative count and rank in the unique count keyword list than the cumulative count keyword list. Becomes higher. In the present embodiment, noise is removed in a manner that does not impair semantic information by deleting unnecessary words, and a plurality of documents in which keywords of the same combination appear are considered as a unique count that is virtually considered as one document. A broad-sense keyword that cannot be extracted simply by looking at the frequency can be set as the highest hierarchy, and a narrow-sense word composed of compound words can be set as a keyword in the second hierarchy or lower. In the third and lower layers, compound words having a relatively large cumulative count number registered in the frequent keyword list can be extracted as keywords.

  Hereinafter, when the highest hierarchy is the first hierarchy and the nth hierarchy (n is an integer of 2 or more) counted from the highest hierarchy is the nth hierarchy, the keyword of the nth hierarchy is described as the nth hierarchy keyword. To do. In particular, the keyword of the highest hierarchy (first hierarchy) is described as “the highest keyword”.

  FIG. 2 is a block diagram showing the configuration of the keyword extraction device 10 according to an embodiment of the present invention. The keyword extraction device 10 can be realized by, for example, a computer device such as one or a plurality of servers. As shown in the figure, an interface unit 11, a hierarchical structure generation unit 12, and a keyword hierarchy result storage unit 18 are provided. It is prepared for.

  The interface unit 11 includes a document input unit 13 and a keyword hierarchy presentation unit 19. The document input unit 13 receives input of document data describing a keyword extraction target document using text data or the like, and outputs the document data to the keyword extraction unit 14. One document data including a plurality of documents may be input, and a plurality of document data including one or a plurality of documents may be input. The document input unit 13 may receive document data from a client terminal connected via a network, and may read document data from a database device or a computer-readable recording medium. The keyword hierarchy presentation unit 19 reads the keyword hierarchy generated by the hierarchy structure generation unit 12 from the keyword hierarchy result storage unit 18 and outputs it. The keyword hierarchy presentation unit 19 may transmit and display the keyword hierarchical structure to a client terminal connected via a network, or may display the keyword hierarchy on a display included in the keyword extraction device 10.

The hierarchical structure generation unit 12 includes a keyword extraction unit 14, a keyword analysis unit 15, a hierarchical structure rule definition unit 16, and a keyword hierarchy generation unit 17.
The keyword extraction unit 14 extracts keywords from each keyword extraction target document included in the document data input from the document input unit 13 to create a frequent keyword list, and writes the keyword list in the keyword hierarchy result storage unit 18. The keyword analysis unit 15 generates a cumulative count keyword list and a unique count keyword list using the keywords in the frequent keyword list that matches the conditions stored in the hierarchical structure rule definition unit 16, and the keyword hierarchy result storage unit 18. Write to. Furthermore, the keyword analysis unit 15 generates a keyword extraction table based on the generated cumulative count keyword list and the unique count keyword list, and uses the generated keyword extraction table to generate an upper hierarchy keyword list, A hierarchical keyword list is generated and written in the keyword hierarchical result storage unit 18. The keyword extraction table indicates whether each keyword is a single word or a compound word, whether it appears in the cumulative count keyword list, the unique count keyword list, or the like. The hierarchical structure rule definition unit 16 stores keyword registration conditions for each of the cumulative count keyword list and the unique count keyword list. The registration condition can be arbitrarily specified by, for example, the lower count number, the appearance order of the lower limit when arranged in the order of the count number, or the upper percentage when registering in the order of the count number. . The keyword hierarchy generation unit 17 extracts a keyword hierarchical structure based on the upper hierarchy keyword list and the lower hierarchy keyword list, and writes a hierarchization table indicating the result in the keyword hierarchy result storage unit 18.

  Next, the data structure of various data stored in the keyword hierarchy result storage unit 18 will be described.

  FIG. 3 is a diagram illustrating a data configuration example of the frequent keyword list. As shown in the figure, in the frequent keyword list KL, a keyword extracted from a document and an accumulated count that is the number of documents in which the keyword appears are associated and registered. Since a keyword having a cumulative count number of a predetermined value or more is registered in the frequent keyword list KL, all keywords in the document can be included by registering a keyword having a cumulative count number of 1 or more.

  Note that the cumulative count keyword list NL has the same data structure as the frequent keyword list KL. However, the cumulative count keyword list NL includes the keywords that satisfy the registration condition in the cumulative count keyword list defined in the hierarchical structure rule definition unit 16 among the keywords registered in the frequent keyword list KL, and the cumulative count thereof. It is the data which registered the number. That is, the cumulative count keyword list NL is extracted from a part of the frequent keyword list KL having a large cumulative count.

  FIG. 4 shows a data configuration example of the unique count keyword list. As shown in the figure, the unique count keyword list UL includes a keyword that satisfies the registration condition for the unique count keyword list defined in the hierarchical structure rule definition unit 16 among the keywords registered in the frequent keyword list KL. This is the data registered with the unique count number.

  FIG. 5 is a diagram illustrating a data configuration example of the keyword extraction table. In the figure, the keyword extraction table KT includes a keyword that appears in at least one of the cumulative count keyword list NL and the unique count keyword list UL, a configuration number indicating the number of words constituting the keyword, a cumulative count number, The unique count number is described in association with it. When a cumulative count number of 1 or more is set, it indicates that it is registered in the cumulative count keyword list N. When a unique count number of 1 or more is set, it is registered in the unique count keyword list UL. It shows that. A single word is indicated when the number of components is set to 1, and a compound word is indicated when 2 or more is set as the number of components.

  FIG. 6 is a diagram illustrating a data configuration example of the hierarchization table. In the figure, the hierarchization table HT includes an upper layer keyword (upper layer keyword), a lower layer keyword (lower layer keyword) subordinate to the upper layer keyword, and a lower layer keyword. The hierarchy number to which it belongs is described in association with it.

FIG. 7 is a sequence diagram between functional units in the hierarchical structure generating unit 12 of the keyword extracting device 10 shown in FIG.
The document input unit 13 of the interface unit 11 outputs the document data input from the client terminal to the keyword extraction unit 14 (step S1). The keyword extraction unit 14 deletes unnecessary words from each keyword extraction target document indicated by the input document data, and extracts keywords. The keyword extraction unit 14 writes document-specific keyword information indicating correspondence between a document or document identification information and a keyword extracted from the document in the keyword hierarchy result storage unit 18. When the keyword extraction unit 14 obtains the cumulative count number of each extracted keyword, the keyword extraction unit 14 creates the frequent keyword list KL and writes it in the keyword hierarchy result storage unit 18 (step S2).

The keyword analysis unit 15 uses the keywords that satisfy the conditions stored in the hierarchical structure rule definition unit 16 based on the document-specific keyword information stored in the keyword hierarchy result storage unit 18 and the frequent keyword list KL, and is unique. The count keyword list UL and the cumulative count keyword list NL are generated and written in the keyword hierarchy result storage unit 18 (step S3). The keyword analysis unit 15 performs morphological analysis of keywords included in the generated unique count keyword list UL and cumulative count keyword list NL, and classifies each keyword into a single word and a compound word (step S4), and based on the result. Then, the upper hierarchy keyword list LL ₁ which is a list of the highest keywords and the lower hierarchy keyword list LL ₂ of the second hierarchy which is a list of second hierarchy keyword candidates are created and written in the keyword hierarchy result storage unit 18 (step S5). The keyword hierarchy generation unit 17 refers to the upper hierarchy keyword list LL ₁ and the lower hierarchy keyword list LL ₂ to generate the hierarchy table HT by associating the highest keyword and the second hierarchy keyword with the keyword hierarchy result. The data is written in the storage unit 18 (step S6). The keyword hierarchy generation unit 17 further obtains the keyword hierarchy for the third hierarchy and below and updates the hierarchy table HT. The keyword hierarchy presentation unit 19 generates screen data for displaying the keyword hierarchical structure based on the hierarchy table HT and displays it on the client terminal (step S7).

  FIG. 8 is a diagram for explaining a counting method of the cumulative count number and the unique count number of keywords. For the sake of simplicity, an example will be described in which the keyword extraction target is three documents Da, Db, and Dc. Also, the keyword extraction target document is the title of an e-mail that is a classification or search target document.

  First, the cumulative count will be described. As shown in the figure, it is assumed that keywords such as “employee” and “general contract” are extracted from documents Da and Dc and keywords such as “employee” are extracted from document Db as a result of removing unnecessary words. Since the keyword “employee” appears in the three documents Da, Db, and Dc, the cumulative count number is “3”, and the keyword “general contract” appears in the two documents Da and Dc. The number is “2”. In the frequent keyword list KL, keywords having a cumulative count number equal to or greater than a predetermined value and the cumulative count number thereof are registered. In the cumulative count keyword list NL, keywords having a higher cumulative count in the frequent keyword list KL are registered.

  Next, the unique count will be described. Since the frequently used keyword combinations “employee” and “general contract” included in the documents Da and Dc are the same, for each of these keywords, the document Da and Dc are also counted as one occurrence. On the other hand, since the combination of frequent keywords is different from that of other documents, the document Db counts the number of appearances of the keyword “employee” as only one document Db. Therefore, the unique count for the keyword “employee” is “2”, and the unique count for the keyword “general contract” is “1”. In the unique count keyword list UL, a keyword having a higher unique count number and the unique count number are registered.

  When performing a unique count, a count is performed assuming that only documents with the same combination of all extracted keywords appear in the same document. For example, the processing objects are documents Da to Dd, the frequently used keywords “A”, “B”, and “C” are included in the documents Da and Db, and the frequently used keywords “A” and “B” are included in the documents Dc and Dd. It is assumed that the frequently used keywords “A” and “C” are included in the document De. At this time, for the keyword “A”, the documents Da and Db are combined once, the documents Dc and Dd are combined once, the document De is once, and the unique count is “3”. Similarly, the unique count number of the keyword “B” is “2”, and the unique count number of the keyword “C” is “2”.

9 to 11 show the process flow of each process shown in the sequence shown in FIG.
FIG. 9 is a diagram showing a processing flow of keyword extraction processing up to the second hierarchy, and corresponds to the processing from steps S1 to S5 shown in FIG.
When the document input unit 13 receives input of document data, the document input unit 13 outputs the document data input to the keyword extraction unit 14. The keyword extraction unit 14 deletes unnecessary words from each keyword extraction target document included in each input document data, and extracts keywords. For example, the keyword extraction unit 14 deletes hiragana (particles), numerical values, parentheses and words in parentheses, punctuation marks, symbols, year / month / date / time, period, number of times, and the like. If the document is an e-mail title, the mail title prefixes “Fw:”, “Re:”, etc. are also deleted. Alternatively, if the document is a table of contents such as a book or the title of each chapter / section, the chapter number, section number, number of pages, etc. are also deleted. Then, among the phrases delimited by the deleted words, all or one-character phrases specified in advance are further deleted, and the remaining phrases are extracted as keywords. In morphological analysis, compound words composed of two or more words are also separated, and keywords that retain their original meaning cannot be extracted in many cases. Therefore, by deleting unnecessary words as described above, the compound words can be used as keywords as they are. Any method can be used as long as it extracts keywords so that compound words are not separated.

  The keyword extraction unit 14 writes, for each document, document-specific keyword information indicating correspondence between the document or document identification information and a keyword included in the document in the keyword hierarchy result storage unit 18. For example, the keyword information for each document can be each document in which unnecessary words are invalidated. When the document input unit 13 obtains a cumulative count number by counting the number of documents in which each extracted keyword appears, the correspondence between the cumulative count number and the cumulative count number is large. The frequent keyword list KL arranged in order is created and written in the keyword hierarchy result storage unit 18 (step S105).

  The keyword analysis unit 15 refers to the document-specific keyword information stored in the keyword hierarchy result storage unit 18 and the frequent keyword list KL, and performs a unique count of each keyword registered in the frequent keyword list KL. That is, the keyword analysis unit 15 extracts keywords registered in the frequent keyword list KL from the keywords included in each document in the document-specific keyword information. The keyword analysis unit 15 virtually considers documents having the same combination of extracted keywords as one document, and performs a unique count for each keyword registered in the frequent keyword list KL. The keyword analysis unit 15 selects a keyword whose unique count number matches the registration condition in the unique count keyword list stored in the hierarchical structure rule definition unit 16, and associates the selected keyword with the unique count number. The attached unique count keyword list UL is created and written in the keyword hierarchy result storage unit 18. Furthermore, the keyword analysis unit 15 selects a keyword whose cumulative count number matches the registration condition in the cumulative count keyword list stored in the hierarchical structure rule definition unit 16 from the frequent keyword list KL, A cumulative count keyword list NL that associates the cumulative count number is created and written in the keyword hierarchy result storage unit 18 (step S110).

  The keyword analysis unit 15 acquires keywords from each of the cumulative count keyword list NL and the unique count keyword list UL generated in step S110, and leaves only one when there is an overlap in the acquired keywords. The keyword analysis unit 15 performs morphological analysis of each acquired keyword, obtains the number of words constituting the keyword, and classifies it into a single word and a compound word, and generates a keyword extraction table KT based on the result (step) S115). That is, in the keyword extraction table KT, the keyword registered in at least one of the cumulative count keyword list NL or the unique count keyword list UL, the number of components obtained from the morphological analysis result of the keyword, the cumulative count keyword list The cumulative count number of the keyword obtained from NL and the unique count number of the keyword obtained from the unique count keyword list UL are registered in association with each other. A keyword with a large cumulative count or unique count may be selected so that a predetermined number of keywords are registered in the keyword extraction table KT.

When the keyword analysis unit 15 selects one keyword from the keyword extraction table KT (step S120), the selected keyword is a single word registered in both the cumulative count keyword list NL and the unique count keyword list UL. It is determined whether it is a keyword (step S125). If YES is determined in step S125, the keyword analyzing unit 15, a keyword selected in step S120 determines that the uppermost keyword is registered in the upper layer keyword list LL ₁ (step S130).

On the other hand, if NO is determined in step S125, the keyword analysis unit 15 determines whether the keyword selected in step S120 is a compound word (step S135). If it is determined in step S135 that the word is a compound word (step S135: YES), the keyword analysis unit 15 determines that the keyword selected in step S120 is the second hierarchy keyword, and enters the lower hierarchy keyword list LL2 in the _second hierarchy. Register (step S140).

  If there is an unselected keyword in the keyword extraction table KT after step S130 or S140 or after determining NO in step S135 (step S145: NO), the keyword analysis unit 15 performs step S120. , And a new keyword is selected in the keyword extraction table KT and the same processing is performed. If it is determined in step S145 that selection has been completed for all keywords (step S145: YES), the process ends.

FIG. 10 is a diagram showing a processing flow of the keyword linking process of the upper hierarchy and the lower hierarchy, and corresponds to the process of step S6 shown in FIG.
Keyword hierarchy generation unit 17 reads one top-level keywords from the upper layer keyword list LL ₁ in the keyword hierarchy result storage unit 18 (step S205). The top-level keyword thus read out to the k _1.

Subsequently, the keyword hierarchy generator 17 refers to the document by keyword information in the keyword hierarchy result storage unit 18, reads out the keyword that appears in the top keyword k ₁ identical to the document. Among the read keywords, the keywords k ₂₁ , k ₂₂ ,... (Hereinafter collectively referred to as “keyword k ₂ ”) registered in the lower hierarchy keyword list LL ₂ in the keyword hierarchy result storage unit 18. Is extracted (step S210).
The keyword hierarchy generator 17, the keyword k ₁ read in step S205 to an upper layer keyword, the keyword k ₂ read in step S210 to the lower layer keyword, a hierarchical table HT set to level number "2" It writes in the keyword hierarchy result storage part 18 (step S215).

When the keyword hierarchy generation unit 17 determines that there is a keyword that has not yet been extracted in the upper hierarchy keyword list LL ₁ (step S220: NO), the keyword hierarchy generation unit 17 performs the processing from step S205, and newly starts from the upper hierarchy keyword list LL _1. extracted uppermost keyword as k ₁ performs the processing. If it is determined that all keywords in the upper hierarchy keyword list LL ₁ have been extracted (step S220: YES), the process ends.

FIG. 11 is a diagram showing a processing flow of the lower hierarchy keyword extraction processing for the second and subsequent hierarchies, which corresponds to the processing in step S6 shown in FIG. 6, and for the keyword linking processing for the upper hierarchy and the lower hierarchy shown in FIG. Will be executed later.
First, the keyword hierarchy generation unit 17 sets the hierarchy i as a lower hierarchy extraction target to 2 (step S305).

The keyword hierarchy generation unit 17 refers to the document-specific keyword information and the frequent keyword list KL in the keyword hierarchy result storage unit 18, is registered in the lower-layer keyword list LL _i of the i-th layer, and is (I-1) For each i-th level keyword k _i under the hierarchy keyword, other keywords that are registered in the frequent keyword list KL and appear in the same document as the i-th level keyword k _i are extracted. (Step S310). Therefore, unlike the top-level keyword and the second hierarchy keyword, keywords that do not appear in the cumulative count keyword list NL or the unique count keyword list UL are also extracted.

The keyword hierarchy result storage unit 18 has already been selected as the keyword of the highest hierarchy to the i-th hierarchy among the keywords read out in step S310, specifically, the upper hierarchy keyword or the lower hierarchy of the hierarchy table HT. Excludes keywords registered as hierarchical keywords. Then, morphological analysis is performed on the remaining keywords that are excluded, keywords that are compound words are further extracted, and the extracted keyword is defined as the (i + 1) th hierarchical keyword k _{(i + 1)} . The keyword hierarchy generation unit 17 generates a lower hierarchy keyword list LL _{(i + 1)} , which is a list of the (i + 1) th hierarchy keyword k _{(i + 1)} , and writes it in the keyword hierarchy result storage unit 18 (step S315).

When the keyword layer generation unit 17 selects one _i- th layer keyword k _i registered in the lower-layer keyword list LL _i of the i-th layer, the keyword layer generation unit 17 appears in the same document as the _i- th layer keyword k _{i in} S310. Are associated with the (i + 1) th hierarchical keyword k _{(i + 1)} registered in the lower hierarchical keyword list LL _{(i + 1)} and written to the hierarchical table HT (step S320). That is, the i-th layer keyword k _i is set as the upper layer keyword, the (i + 1) th layer keyword k _{(i + 1)} is set as the lower layer keyword, and “i + 1” is set as the layer number.

If the keyword hierarchy generation unit 17 determines that there is an _i- th hierarchy keyword ki that has not yet been selected in the lower-layer keyword list LL _i of the i-th hierarchy (step S325: NO), the keyword hierarchy generation unit 17 performs the process of step S320 again. When it is determined that the selected all of the i-th layer keyword k _i of the lower hierarchy keyword list LL _i (step S325: YES), the keyword hierarchy generation unit 17 determines whether have been processed to a predetermined hierarchy . If it is determined that the processing has not been completed up to the predetermined hierarchy (step S330: NO), the process starts from step S310 again with the hierarchy i to be extracted as the lower hierarchy being set to the next lower hierarchy (i + 1). If it is determined that the process has been completed up to a predetermined level set in advance (step S330: YES), the process ends.

  In the above description, the process is terminated when a predetermined level is reached, but the process may be performed until no subordinate keyword can be detected.

After the above processing, the keyword hierarchy presentation unit 19 generates and outputs screen data for displaying the keyword hierarchical structure based on the hierarchization table HT stored in the keyword hierarchy result storage unit 18. Specifically, the keyword hierarchy presenting unit 19 reads a set of the upper hierarchy keyword and the lower hierarchy keyword stored in association with the hierarchy number “2” from the hierarchy table HT. The keyword hierarchy presentation unit 19 arranges the second hierarchy keyword read from the lower hierarchy keyword corresponding to the upper and lower hierarchy keyword under the highest keyword set as the upper hierarchy keyword, and connects these with a line Generate a hierarchical image. Further, the keyword hierarchy presentation unit 19 reads out from the hierarchy table HT the upper hierarchy keyword in which the second hierarchy keyword is set and the lower hierarchy keyword associated with the hierarchy number “3”. A third hierarchical keyword read from a lower hierarchical keyword having the second hierarchical keyword as an upper hierarchical keyword is arranged under the second hierarchical keyword, and an image of a hierarchical structure in which these are connected by lines is generated. Add to images up to 2 levels. Similarly, images of the keyword hierarchical structure are added to the third and lower layers, and screen data for displaying the generated images is output.
Note that the keyword extraction device 19 may generate an image indicating the keyword hierarchical structure for each top-level keyword. In addition, the user may specify the highest keyword in the hierarchical structure.

In the embodiment described above, the scale of the hierarchical structure can be adjusted by increasing or decreasing the number of keywords registered in the frequent keyword list KL, the cumulative count keyword list NL, the unique count keyword list UL, and the keyword extraction table KT.
Further, the number of keywords to be extracted in each hierarchy may be limited. In this case, as a keyword of each hierarchy obtained by the above processing, a keyword having a large cumulative count obtained from the frequent keyword list KL is selected by a predetermined number. In the case of the highest keyword, it may be selected from those having a large unique count.

Next, the processing of the keyword extraction device 10 described above will be described using specific data examples.
FIG. 12 is a diagram showing an example of generating the frequent keyword list KL by the processing shown in FIG. In step S105 of FIG. 9, when document data including a keyword extraction target document as shown in FIG. 12A is input, the keyword extraction unit 14 reads the documents D1, D2, D3, ... delete unnecessary words from each. For example, in the case of the document D1, “No. 008” indicating the number of times, symbols ““ ”and“ ””, “H21 year” indicating the year, “hira” “no”, “about”, parentheses and the phrase “ (Request) "is deleted. Therefore, “business plan” and “formulation” are extracted from the document D1. In the case of the document D2, the parentheses and the phrases “(request)” and “(July)”, the punctuation marks “,”, the hiragana “ni”, “do”, and the one-letter phrase “seki” are deleted. The Therefore, “employee”, “general contract”, and “implementation report” are extracted from the document D2. Here, as a result of the condition that it is equal to or greater than a predetermined cumulative count (here, 2 or more), the keywords “business plan”, “employee”, “general plan”, and “implementation report” surrounded by dotted lines ,... Are selected, and as shown in FIG. 12B, a frequent keyword list KL in which the selected keywords and their accumulated counts are set is generated.

FIG. 13 is a diagram showing an example of generation of the upper layer keyword list LL ₁ and the lower layer keyword list LL ₂ by the processing shown in FIG.
When the keyword extraction unit 14 generates the frequent keyword list KL shown in FIG. 12 in step S105 of FIG. 9, the keyword extraction unit 14 counts the unique count number for each keyword registered in the frequent keyword list KL in step S110 of FIG. As shown in FIG. 13 (a), a unique count keyword list UL in which seven keywords having the unique count number and the unique count number are set is generated. Further, as shown in FIG. 13 (a), among the keywords registered in the frequent keyword list KL, the cumulative count keyword list NL in which seven keywords with the highest cumulative count number and the cumulative count number are set is displayed. Generate.

  In step S115 of FIG. 9, the seven keywords “employee”, “confirmation”, “general contract”, “development project”, “business plan”, “contract cost”, “execution” registered from the unique count keyword list UL The “Report” is extracted from the 7 keywords “Employee”, “Update”, “Trend”, “Development”, “Modification”, “General contract”, “Main measures” registered from the cumulative count keyword list NL Is done. Then, in the keyword extraction table KT, these keywords are registered so as not to overlap. As shown in FIG. 13B, the number of components obtained from the morphological analysis result of the keywords in association with the registered keywords. The unique count number obtained from the unique count keyword list UL and the unique count number obtained from the cumulative count keyword list NL are registered.

The process of step S120~S140 of FIG. 9, as shown in FIG. 13 (c), the upper layer keyword list LL _1, values for both the cumulative count and a unique count is set, i.e., unique Among the keywords registered in both the count keyword list UL and the cumulative count keyword list NL, “employee”, which is a keyword having the number of components “1”, is registered (step S130 in FIG. 9). On the other hand, as shown in FIG. 13 (d), the lower layer keyword list LL _2, while the value of at least one of the accumulated count number or unique count is set, i.e., unique count keyword list UL or , Among the keywords registered in one or both of the cumulative count keyword list NL, the keywords “development projects”, “general contracts”, “main measures”, “business plans”, which have a composition number “2” or more, “Contract cost” and “Implementation report” are registered.

FIG. 14 is a diagram showing an example of generating a keyword hierarchical structure up to two levels by the processing shown in FIG.
As shown in FIG. 14 (a), the upper layer keyword list LL ₁ produced by the process of FIG. 10, the keyword "employee" is registered. Further, in the document shown in FIG. 12 (a), are included in the same document as the keyword "employee", and keywords registered in the lower layer keyword list LL ₂ shown in FIG. 13 (a), the document D2 , “General Contract” included in D7, “Implementation Report”, and “Contract Cost” included in Document D4.

  Therefore, as shown in FIG. 14B, the hierarchy table HT includes an upper hierarchy keyword “employee”, lower hierarchy keywords “general contract”, “contract cost”, and “execution report”, The hierarchy number “2” is registered in association with each other. FIG. 14C shows a hierarchical structure of keywords according to the setting contents of the hierarchization table HT shown in FIG. This figure shows the case where the keyword displayed above is a higher level, with the top keyword “employee” at the top, the second level keywords “general contract”, “contract cost”, and The “implementation report” is arranged below the top keyword, and the top keyword is connected with a line indicating that the keyword is subordinate to the three second-layer keywords. When the keyword hierarchical structure of up to two levels is presented to the user, the keyword level presenting unit 19 is based on the hierarchization table HT stored in the keyword level result storage unit 18 as shown in FIG. Screen data for displaying a hierarchical structure of various keywords is generated and displayed on the display of the client terminal or the display included in the keyword extraction device 10. Here, an example is shown in which the top is associated with the top, but any direction may be associated with the top.

  FIG. 15 is a diagram illustrating an example of generating a keyword hierarchy structure after the second hierarchy by the process shown in FIG. Here, keyword extraction under the second hierarchy, that is, processing when i = 2 is set in step S310 of FIG. 11 will be described.

By the processing from FIG. 9 to FIG. 10, the hierarchization table HT shown in FIG. In the lower hierarchy keyword list LL ₂ of the second hierarchy generated in the processes of FIGS. 9 to 10, six keywords are registered as shown in FIGS. 13D and 14A. As shown in FIG. 15A, the keyword used as the second hierarchy keyword is “general contract” which is a lower hierarchy keyword stored in association with the hierarchy number “2” in the hierarchy table HT. , “Contract costs” and “implementation report”. Therefore, as shown in FIG. 15 (b), among the lower hierarchical keyword list LL _2, a target for extracting a lower layer and only these second hierarchy keyword.

Keyword hierarchy generator 17 Figure 15 (a) second level keywords registered in Hierarchical keyword list LL ₂ shown in "General Terms", "contract costs", each "Implementation Report", the same document Other keywords that appear and are registered in the frequent keyword list KL are extracted. In the case of the documents D1 to D7 shown in FIG. 12, “employee” and “execution report” registered in the frequent keyword list KL are extracted from the documents D2 and D7 corresponding to the keyword “general contract”, and the keyword “contract” is extracted. “Contract costs” and “Monthly reports” registered in the frequent keyword list KL corresponding to the “cost” are extracted, and the frequent keyword lists from the documents D2 and D7 corresponding to the keyword “implementation report” are extracted. “Employee” and “general contract” registered in KL are extracted.

  Here, “employee” is registered as an upper hierarchy keyword of the hierarchization table HT, and “general contract” and “contract cost” are registered as lower hierarchy keywords of the hierarchization table HT. As shown in FIG. 15B, only the “monthly report” extracted corresponding to the keyword “contract cost” remains. Therefore, as shown in FIG. 15 (c), in the stratification table HT, the upper hierarchy keyword in which the second hierarchy keyword “contract cost” is set, and the lower hierarchy keyword in which the third hierarchy keyword “monthly report” is set. , And a record associated with the hierarchy number “3” is added.

  FIG. 15D shows a hierarchical structure of keywords according to the setting contents of the hierarchization table HT of FIG. This is because the third hierarchical keyword “monthly report” is added to the hierarchical structure of the keywords up to the second hierarchical level shown in FIG. The structure is shown. That is, the third-layer keyword “monthly report” is arranged under the second-layer keyword “contract cost”, and the two are connected by a line. The keyword hierarchy presentation unit 19 generates screen data for displaying the keyword hierarchy as shown in FIG. 15D based on the hierarchization table HT stored in the keyword hierarchy result storage unit 18. And displayed on the display of the client terminal or the display provided in the keyword extraction device 10.

  As described above, it can be understood that keywords used for e-mail search or classification can be extracted by a hierarchical structure in which a broadly-characteristic keyword is the highest hierarchy and a narrowly-defined keyword is a lower hierarchy.

  In the above example, the keyword extraction target document is not the main body of the electronic mail to be searched or classified, but the electronic mail title. By using an e-mail title as a keyword extraction target document, a characteristic keyword can be obtained efficiently and accurately. This is because e-mail titles often contain characteristic words and phrases related to the contents of e-mails, and words that are not suitable for keywords such as addresses and greetings. This is because it is difficult.

When the content to be searched or classified is a document such as a book or a document, the title of a chapter or section included in the table of contents or the text may be used as a document from which keywords are extracted. In addition, when the document is a paper or the like, the summary included in the paper may be a processing target document, and in the case of digitalized content, the metadata added to the content may be the processing target document.
Further, the content itself to be classified, such as the body of an email or the text of a book or document, may be used as a document for keyword extraction.

  The contents to be searched or classified are stored in the database device in association with the keywords constituting the hierarchical structure described above. The user selects the keyword used for the search by clicking on the keyword in the display screen of the keyword hierarchical structure based on the hierarchization table generated by the keyword extraction device 10. As a result, the database device returns a document corresponding to the selected keyword to the client terminal. If the selected keyword is in a lower hierarchy, all keywords from the selected keyword to the highest keyword may be used for the search.

A scene in which the keyword hierarchical structure generated by the keyword extraction apparatus 10 according to the present embodiment can be effectively applied will be described.
For example, it can be used for automatic classification of electronic mail stored in a mailer and contents in a file server. This makes it possible to automatically create a hierarchical structure of keywords for searching the enormous amount of accumulated data, thereby facilitating management of scattered data.

Also, if you want to classify a large number of books, it is almost impossible to check the contents of each book one by one. Therefore, by using the keyword extraction device 10 of the present embodiment, it is possible to automatically create a keyword hierarchical structure for searching a huge amount of accumulated books, and to easily classify books. For example, a book search using a keyword hierarchical structure generated by the keyword extraction device 10 is provided in a shopping server of a company that carries out mail order over the Internet.
It can also be used for meaning information in semantic technology. In other words, by creating a hierarchical structure of keywords for a large amount of content according to the present embodiment, it becomes easy to create a classification structure necessary for content meaning, and the construction of a taxonomy can be simplified.

  As described above, according to the present embodiment, from a plurality of input documents, the highest hierarchy is defined as a broad keyword characteristic of the document, and the second and lower hierarchy is defined as a narrow keyword, and each hierarchy is appropriately narrowed down. The keyword hierarchy can be extracted. In particular, it is difficult to extract by keyword extraction based on the appearance frequency, and it is possible to extract characteristic keywords in the document although the number of appearances is small.

  The keyword extracting device 10 has a computer system inside. The operation processes of the interface unit 11 and the hierarchical structure generation unit 12 of the keyword extraction device 10 described above are stored in a computer-readable recording medium in the form of a program, and the computer system reads and executes this program. Thus, the above process is performed. Here, the computer system includes a CPU, various memories, an OS, and hardware such as peripheral devices.

Further, the “computer system” includes a homepage providing environment (or display environment) if a WWW system is used.
The “computer-readable recording medium” refers to a storage device such as a flexible medium, a magneto-optical disk, a portable medium such as a ROM and a CD-ROM, and a hard disk incorporated in a computer system. Furthermore, the “computer-readable recording medium” dynamically holds a program for a short time like a communication line when transmitting a program via a network such as the Internet or a communication line such as a telephone line. In this case, a volatile memory in a computer system serving as a server or a client in that case, and a program that holds a program for a certain period of time are also included. The program may be a program for realizing a part of the functions described above, and may be a program capable of realizing the functions described above in combination with a program already recorded in a computer system.

DESCRIPTION OF SYMBOLS 10 ... Keyword extraction apparatus 11 ... Interface part 12 ... Hierarchical structure generation part 13 ... Document input part 14 ... Keyword extraction part 15 ... Keyword analysis part 16 ... Hierarchical structure rule definition part 17 ... Keyword hierarchy generation part 18 ... Keyword hierarchy result storage part 19 ... Keyword hierarchy presentation section

Claims (5)

A keyword is extracted from a plurality of documents, the number of the documents in which each of the extracted keywords appears is counted to obtain a cumulative count, and for a plurality of documents in which the same combination of keywords appears, the number of documents that appear 1, a keyword extraction unit that counts the number of documents in which the extracted keyword appears to obtain a unique count number;
Of the keywords whose cumulative count obtained by the keyword extraction unit is equal to or greater than a predetermined condition and whose unique count is equal to or greater than a predetermined condition, a keyword that is a single word is selected as a keyword in the highest hierarchy. In addition, a keyword analysis that selects a keyword that is a compound word as a lower-level keyword from among keywords whose cumulative count obtained by the keyword extraction unit is greater than or equal to a predetermined condition or whose unique count is greater than or equal to a predetermined condition. And
A keyword hierarchy generation unit that extracts the lower-level keyword appearing in the same document as the highest-level keyword as a second-level keyword under the highest-level keyword;
A keyword extracting device comprising:   The keyword extracting apparatus according to claim 1, wherein the keyword extracting unit extracts a keyword by deleting a predetermined unnecessary word from the document. The keyword hierarchy generation unit appears in the same document as the keyword of the lower hierarchy that is subordinate to any keyword for the lower hierarchy after the second hierarchy in order from the second hierarchy, and Acquiring another keyword of a compound word that is not subordinate, and repeating the process of extracting the acquired other keyword as a keyword of one lower hierarchy that is subordinate to the keyword of the lower hierarchy,
The keyword extraction device according to claim 1 or 2, wherein A keyword extraction method executed by a keyword extraction device,
The keyword extraction unit extracts keywords from a plurality of documents, counts the number of the documents in which each of the extracted keywords appears to obtain a cumulative count number, and appears for a plurality of documents in which the same combination of keywords appears. A keyword extraction process of obtaining a unique count number by counting the number of documents in which the extracted keyword appears, where the number of documents to be performed is 1,
A keyword analysis unit selects a keyword that is a single word from among the keywords whose cumulative count obtained in the keyword extraction process is equal to or greater than a predetermined condition and whose unique count is equal to or greater than a predetermined condition. A keyword that is selected as a hierarchical keyword and has a cumulative count obtained in the keyword extraction process equal to or greater than a predetermined condition, or a keyword that is a compound word among the keywords whose unique count is equal to or greater than a predetermined condition is a keyword in a lower hierarchy. Keyword analysis process to select as,
A keyword hierarchy generation process in which a keyword hierarchy generation unit extracts the lower hierarchy keyword appearing in the same document as the highest hierarchy keyword as a second hierarchy keyword under the highest hierarchy keyword;
A keyword extraction method characterized by comprising: Computer
A keyword is extracted from a plurality of documents, the number of the documents in which each of the extracted keywords appears is counted to obtain a cumulative count, and for a plurality of documents in which the same combination of keywords appears, the number of documents that appear 1, a keyword extraction unit that obtains a unique count by counting the number of documents in which the extracted keyword appears;
Of the keywords whose cumulative count obtained by the keyword extraction unit is equal to or greater than a predetermined condition and whose unique count is equal to or greater than a predetermined condition, a keyword that is a single word is selected as a keyword in the highest hierarchy. In addition, a keyword analysis that selects a keyword that is a compound word as a lower-level keyword from among keywords whose cumulative count obtained by the keyword extraction unit is greater than or equal to a predetermined condition or whose unique count is greater than or equal to a predetermined condition. Part,
A keyword hierarchy generation unit that extracts the lower-level keyword appearing in the same document as the highest-level keyword as a second-level keyword under the highest-level keyword;
A program characterized by functioning as

Images