JP4385697B2 - Concept search method and system - Google Patents

Description

  The present invention relates to a concept search in which an electronic document is searched with a search condition registered by the user and a document satisfying the condition is returned to the user. In particular, by scanning the contents of the electronic document, the user can set the search condition as a search condition. The present invention relates to a document search method and system having an inter-document fitness level calculation function for calculating a fitness level for a registered document.

  In recent years, a large amount of digitized documents (hereinafter referred to as text) has been delivered to users from time to time by e-mail, electronic news, and the like. In addition, information sources that transmit information using the WWW (World Wide Web) are rapidly increasing, and a large amount of text is collected from these information sources using an information collecting robot or the like. For this reason, there is an increasing need to search for texts including information that the user really wants from among these texts.

  In the conventional search system, a search is performed by assembling a search expression for a word that the user thinks is necessary for the search according to a certain syntax and inputting it. However, it is difficult for a user unfamiliar with searching to enter a suitable word to obtain the desired information, or to extract only necessary information and assemble a complicated search formula to shake out unnecessary information. is there. For this reason, Patent Document 1 proposes a technique (hereinafter referred to as a concept search) in which a document including information desired by a user (hereinafter referred to as a seed document) is input instead of assembling a search expression. . In this technique, a word necessary for a search (hereinafter referred to as a feature term) is automatically extracted from a seed document, an appropriate weight is assigned to the extracted feature term, and the degree of matching of a search result sentence is calculated. A search result having a fitness value greater than a certain value is used.

  However, when the seed document is a long document, not only the document information desired by the user but also a seed document including an unnecessary concept may be input. In this case, since there is an error with the information desired by the user, the user is not satisfied with the search result, so he tries to perform a search again with some adjustments. In general, a search result document is referred to, and a document (or sentence) including the concept desired by the user is found more appropriately than the seed document. By repeating this operation, it is possible to approach the search result desired by the user. Alternatively, with respect to the weight for the feature term, the search is performed again by increasing the weight of the feature term that is considered necessary for the feature term, or by reducing the weight of the feature term that is deemed unnecessary.

Hereinafter, other related literature techniques related to the present invention will be described.
Non-Patent Document 1 mentions a technique for performing clustering (automatic classification) by comparing the number of appearances of the keyword of each classification item with the number of appearances of the keyword of each document and classifying the classification item to the closest classification item.

  Non-Patent Document 2 mentions a technique for extracting an important sentence in a document based on keyword weights.

JP-A-11-143902

Automatic document clustering system gnmz (http://icrouton.as.wakwak.ne.jp/pub/kks/cnamazu.html) "Technology for automatically summarizing texts-Part 1-Extracting important sentences in texts", computer science magazine bit February, Kyoritsu Shuppan, pp. 37-42, 2000.2

  In the concept search, the search is executed without determining whether the information contained in the document is necessary information or unnecessary information for the user. Therefore, when performing a concept search using a long document, only the conventional method is used. Then, the search accuracy may decrease. In order to prevent this search accuracy degradation, the following two points are problems.

1) Means for classifying information contained in a document 2) Means for allowing a user to accurately determine whether the information is necessary In the prior art, as means for 2), there are selection of feature terms and adjustment of weights. It may be difficult to determine whether or not the information is desired by the user only with feature terms lacking peripheral information.

  An object of the present invention is to provide a concept search method in which search accuracy is improved as compared with a conventional concept search by executing a more accurate search based on information necessary for a user.

  In order to achieve the above-mentioned object, the process consisting of the following steps automatically presents candidates for seed documents (sentences) suitable for the search without the user knowing the optimum feature terms for the search. By selecting an appropriate sentence, the above problem can be solved and the search accuracy can be improved. The processing procedure will be described below.

  Step 1: Extract feature terms and their weights from the input seed document.

  Step 2: The input seed document is decomposed into sentences.

  Step 3: Each sentence decomposed in Step 2 is regarded as one document, and each sentence is classified into equivalence classes using the feature terms extracted in Step 1 and their weights (hereinafter referred to as clustering). Is determined from the feature terms extracted in step 1.

  Step 4: Select a sentence (hereinafter referred to as an important sentence) that is most suitable as a feature of the equivalence class among the equivalence classes classified in Step 3 above.

  Step 5: Present the important sentence extracted in Step 4 to the user, and ask the user to select a sentence including necessary information.

  Step 6: A concept search is executed using the feature terms representing the equivalence classes classified in Step 3 and their weights corresponding to the sentence selected in Step 5 above.

  In executing the concept search, the concept search is executed using only information necessary for the user, so that a more accurate search can be executed. As a result, the search accuracy is improved as compared with the conventional concept search.

  FIG. 1 shows the system configuration of the present invention. The concept search device 30000 is a client server type search system that communicates with the client 10000 via the network 20000. When performing a search, the user inputs a seed document 40000 from the client 10000. The client 10000 transmits the input seed document 40000 to the concept search device 30000 through the network 20000, and executes the processing of the present invention.

  The concept search device 30000 has the following configuration. A feature term extraction unit 31000 that extracts a feature term and its weight from the seed document 40000 using the document information DB 38000, and a similarity that calculates the similarity between the document in the document DB 37000 and the seed document 40000 using the feature term and its weight Degree calculation unit 32000, sentence decomposition unit 33000 that decomposes seed document 40000 into sentences, clustering unit 34000 that classifies each sentence in seed document 40000 into equivalence classes with similar contents using feature terms and their weights, clustering unit 34000 The important sentence extraction unit 35000, the similarity calculation unit 32000, and the important sentence extraction unit that output the important sentence for the search condition (the feature term extracted by the feature term extraction unit and its weight) using the sentence classified as the equivalence class by the input sentence Screen data that receives 35000 results and sends to client 10000 Consisting screen data generating unit 36000 to generate. The concept search procedure of the present invention will be described below according to the processing flow of FIG.

  Step 60000: The user inputs the document 41000 having the contents illustrated in FIG. 3 into the seed document input BOX 11000 from the client screen shown in FIG. After the input, when the user presses the search start button 12000, the client 10000 transmits the input data as a seed document 40000 to the concept search device 30000 via the network 20000.

  Step 61000: The concept retrieval apparatus 30000 receives the seed document 40000 transmitted from the client 10000 and inputs it to the feature term extraction unit 31000. The feature term extraction unit 31000 uses the document information DB 38000 to extract the feature term 42000 and its weight 43000 shown in FIG. Here, as an example of the feature term extraction algorithm, the method of Patent Literature 1 can be used. In this case, the document information DB 38000 is assumed to store necessary data shown in Patent Literature 1. As another processing method, morphological analysis may be used for feature term extraction, and the number of appearances of feature terms in the seed document may be used for weight calculation.

  Step 62000: Next, the concept search device 30000 inputs the seed document 40000 to the sentence decomposition unit 33000, and obtains a sentence group 44000 in the seed document 40000 shown in FIG. The sentence decomposition unit 33000 operates according to the processing flow shown in FIG. When processing relatively free-form sentences such as business daily reports, the sentence length tends to be irregular, so the sentence disassembly unit 33000 combines several short sentences into one sentence or has a long document. You may adjust so that it may cut out at the break of the word which does not exceed fixed length.

  Step 63000: The feature term 42000 extracted by the above step, its weight 43000, and the sentence group 44000 are input to the clustering unit 34000, and each sentence is classified into equivalence classes, and the feature term group 45000 representing the classification shown in FIG. And the classification sentence group 46000 is obtained. Although there are various document clustering methods, the method of Non-Patent Document 1 can be used here. The similarity of each sentence may be calculated by using a term appearance frequency, a product of the term appearance frequency and the term uniqueness, or a product of the term appearance frequency and the number of types of terms. FIG. 7 shows the clustering result when the number of classifications is 3. Necessary information is acquired from the document information DB 38000. The feature term group 45000 representing the classified sentence group 46000 is assumed to be all included in each sentence group among the feature terms 42000 extracted in the above step. Further, the target sentence for executing clustering is only a sentence including the feature term shown in FIG.

  Step 64000: Next, an important sentence of each classification group is determined. The important sentence is determined by the important sentence extraction unit 35000 using the similarity calculation unit 32000. Each sentence in the classification is input to the similarity calculation unit 32000 using the feature term group 45000 and the weight obtained from the table of FIG. Here, data is acquired from the document information DB 38000 as necessary. The similarity calculation unit 32000 may obtain the similarity of each sentence in accordance with Patent Document 1 or may use the method of Non-Patent Document 2. However, in this case, it is assumed that the document information DB 38000 stores data necessary for Non-Patent Document 2. FIG. 9 shows a sentence having the highest similarity among the obtained similarities 47000 as the important sentence 48000 of the classification group.

  Step 65000: When the obtained important sentence 48000 is input to the output screen data generation unit 36000, the concept search device 30000 transmits necessary output data to the client 10000 via the network 20000, and the client 10000 outputs to the search result screen 13000. (FIG. 10).

  Step 66000: The search result screen 13000 displays the important sentence 48000 of each classification group and the number of sentences classified into the classification group as the score 15000 of the classification group. The higher the score, the more the concept of the classification group is included in the seed document 40000. Further, by using the detailed browsing button 18000, all documents in the classification group shown in FIG. 8 can be browsed. The user checks a necessary concept according to a screen instruction using a check box 17000, and when a search execution button 14000 is pressed, a selection number 49000 is transmitted. In FIG. 10, “2” is selected, and the client 10000 transmits “2”, which is the selection number 49000, to the concept search device 30000.

  Step 67000: After receiving the selection number 49000, the concept search device 30000 obtains a feature term group 45000 corresponding to the selection number 49000 and its weight 43000 from the table shown in FIG. 9 from the table shown in FIG.

  The feature term 45000 and its weight 43000 are input to the similarity calculation unit 32000, and the similarity to each document stored in the document DB 37000 is obtained. As an example of the similarity calculation method, Patent Document 1 is used here.

  Step 68000: After calculating the similarity, the document title is input to the output screen data generation unit 36000 in descending order of similarity. The output screen data generation unit 36000 transmits the output data to the client 10000.

  Step 69000: The client 10000 displays the search result screen shown in FIG. When the search condition storage button 19000 is pressed, the search condition is stored in the client using the selected sentence as a key. Thereby, the user can save the trouble of assigning an appropriate storage name, and whether or not the user can use it can be determined based on the contents of the text even when the user uses it later.

  Further, the sentence decomposition process shown in FIG. 12 will be described.

  Step 50000: The text operation unit 33100 reads the processed document into the work storage area 33200. The operation initial character position 33300 and the operation initial position (first character of the document) of the operation character position 33400 are set to 0 in the sentence number 33500, and the process proceeds to Step 50010.

  Step 50010: The text operation unit 33100 determines whether or not the character at the character position stored in the operation position storage area 33400 among the processed documents read into the work storage area 33200 is a delimiter (.). If it is a delimiter, the process of step 50020 is performed. If it is not a delimiter, the process proceeds to step 50060.

  Step 50020: The characters from the operation start character position 33300 to the operation character position 33400 are copied from the work storage area 33200 to the sentence group storage area 33600, and the process proceeds to Step 50030.

  Step 50030: The sentence number 33500 is incremented by 1, and the process proceeds to Step 50040.

  Step 50040: The operation character position 33400 is incremented by 1, and the process proceeds to Step 50050.

  Step 50050: The value of the operation character position 33400 is set in the operation start character position 33300, and the process proceeds to Step 50060.

  Step 50060: The value of the operation character position 33400 is incremented by 1, and the process proceeds to Step 50070.

  Step 50070: If the value of the operation character position 33400 is smaller than the number of characters of the processed document, the process proceeds to Step 50010. If the value of operation character position 33400 is equal to the number of characters of the processed document, the process proceeds to step 50080.

  Step 50080: The characters from the operation start character position 33300 to the operation character position 33400 are copied to the sentence group storage area 33600. The number of sentences 33500 is incremented by 1, each sentence in the sentence group storage area is output, and the process is terminated.

1 shows a system configuration of the present invention. The client screen in the Example of this invention is shown. The seed document input in the Example of this invention is shown. The feature terms and their weights are shown. The sentence decomposition part in the Example of this invention is shown. Indicates the sentence group output by the sentence decomposition unit. The classification sentence group in the Example of this invention is shown. The classification sentence group in the Example of this invention and the importance of each sentence are shown. Each classification and the important sentence in the Example of this invention are shown. The concept selection screen in the Example of this invention is shown. The search result screen in the Example of this invention is shown. The sentence decomposition part process flow in the Example of this invention is shown. The flow of the process in the Example of this invention is shown.

Explanation of symbols

10000: client, 20000: network, 30000: concept search device,
31000: Feature term extraction unit, 32000: Similarity calculation unit,
33000: sentence decomposition unit, 34000: clustering unit, 35000: important sentence extraction unit,
36000: output screen data generation unit, 37000: document DB, 38000: document information DB, 40000: seed document

Claims (4)

A concept search method for a server-type concept search device that is connected to a client device via a network and inputs a document as a search condition to search for a similar document ,
The concept search device
As the search condition, a document input from the client device and transmitted to the concept search device is read into a working storage area of the concept search device,
Extracting a keyword that is a feature of the document in the working storage area and the number of appearances of each of the keywords;
Disassemble the document in the working storage area into individual sentences ;
Copy the decomposed individual sentences to the sentence group storage area of the concept retrieval device,
From the keyword and the number of appearances and the decomposed individual sentences in the sentence group storage area, a keyword group consisting of a plurality of the keywords and a plurality of sentences corresponding to each of the keyword groups,
The importance level of each sentence is calculated from the keyword group, the plurality of sentences associated with the keyword group, and the number of appearances.
Extracting the sentence showing the highest value from the calculated importance as an important sentence from the plurality of sentences corresponding to each of the keyword groups, corresponding to the keyword group ,
Sending each of the keyword groups and the important sentence extracted corresponding to the keyword group to the client device,
After the plurality of important sentences received from the concept storage device in the client device are displayed on the screen of the client device, one important sentence selected in the client device from the client device is the concept retrieval device. Is notified for search processing,
Performing a concept search process of a document stored as an electronic document in a document DB connected to the concept search device, using the keyword group corresponding to the selected important sentence and the weight of the keyword group ; and
A concept search method for a server-type concept search device, wherein the result of the concept search processing is transmitted to the client device . In the method of decomposing the document into each sentence, when the length of the decomposed sentence exceeds a predetermined length, the document is further decomposed into a plurality of sentences so as not to exceed the predetermined length, or the length of the decomposed sentence is 2. The server type concept search according to claim 1, wherein if the predetermined length is not exceeded, a plurality of sentences are combined so as to exceed the predetermined length, and the document is decomposed by selecting one of them . Device concept search method. In the method of calculating the importance of each sentence,
A method of calculating the uniqueness of each keyword associated with the one keyword group from the one keyword group and the plurality of sentences associated with the one keyword group;
Add a method to calculate the product of the frequency of appearance of the keyword and the uniqueness in each sentence,
2. The concept search method for a server type concept search apparatus according to claim 1, wherein the importance of each sentence is calculated from the product of the frequency of appearance of the keyword and the uniqueness. A server-type conceptual search device that is connected to a client device via a network and inputs a document as a search condition to search for a similar document ,
Means for reading , as the search condition, a document input from the client device and transmitted to the concept search device into a working storage area of the concept search device;
Means for extracting a keyword that is a feature of the document in the working storage area and the number of appearances of each of the keywords ;
It means for decomposing the document of the working memory area in the individual statements,
Means for copying the decomposed individual sentences to a sentence group storage area of the concept search device;
From individual sentences the decomposing of the keywords and the appearance count and the Bungun storage region, and corresponding with to means with a plurality of said statements corresponding to each of the keyword group consisting of a plurality of the keywords the keyword group,
Means for calculating importance of each sentence from the keyword group, a plurality of the sentences associated with the keyword group, and the number of appearances;
Means for extracting the sentence showing the highest value from the calculated importance as an important sentence, and extracting the sentence corresponding to the keyword group from a plurality of sentences corresponding to each of the keyword groups ;
Means for transmitting each of the keyword groups and the important sentence extracted corresponding to the keyword group to the client device;
After the plurality of important sentences received from the concept storage device in the client device are displayed on the screen of the client device, one important sentence selected in the client device from the client device is the concept retrieval device. Means to be notified of for search processing,
Means for performing a concept retrieval process of a document stored as an electronic document in a document DB connected to the concept retrieval device, using the keyword group corresponding to the selected important sentence and the weight of the keyword group ; ,as well as
A server-type concept search device comprising means for transmitting the result of the concept search processing to the client device .

Images