JP5346045B2 - Document search apparatus, document search method, and document search program - Google Patents

Description

  The present invention relates to a technique for searching a document related to a search word input by a user from a document set.

  2. Description of the Related Art Conventionally, a search technique for obtaining a document related to a large collection of documents collected from the World Wide Web by inputting a search keyword is known. The search keyword is generally input from one word to several words.

  By the way, a document set collected from the word wide web or the like often includes many similar documents through reprints or quotations. For example, in Non-Patent Document 1, it is reported that 17.3% is almost duplicated even in a relatively small document set. In Non-Patent Document 2, 40% of all web pages Reported to be on the same page.

  If only similar documents are included in the search results, it is difficult to reach the target search results. In order to exclude these similar documents from the search results, for example, as shown in Non-Patent Document 3, etc., documents similar to each other are determined in advance and those similar to each other are included in the search results. There is no known method.

  The technique of KWIC (keyword in context) or query-biased summary used in the keyword context creation unit of the present invention described later is described in Non-Patent Document 4 below.

Yaniv Bernstein and Justin Zobel, “Redundant documents and search effects”, In Proceedings of the 14th ACM international conference on Information. Christopher D.C. Manning, Prabhakar Rhagavan and Hinrich Schutze, “Introduction to Information Retrieval”, Cambridge University Press, 2008. Andrew Broder, “Identifying and Filtering Near-Duplicate Documents”, Combined Pattern Matching, 2000, pp. 1-10. Anastasios Tombros and Mark Sanderson, "Advantages of query biased summaries in information retrieval", Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval, 1998, pp. 2-10.

  However, since the determination is based on the similarity of the whole document depending on the conventional method, even if the whole document is not similar, the document in which the partial portion is similar was not detected and excluded. For this reason, for example, when a part of another document is cited in the document, if the cited part is related to the keyword entered by the user, the cited document is also related in addition to the original document. That means it will be included in the search results.

  For this reason, in the two documents, even if almost the same content is described in the part including the target information that the searcher is searching for, there is a difference in other parts. Cannot be excluded. As a result, such a document is included in the search result, and as a result, there is a problem that the user has to spend time and effort to reach the target information.

  Based on these, there is a need for a method that prevents duplication of information related to the information requested by the user and allows the user to easily grasp the entire search result.

  The present invention solves the above-mentioned problems, and its object is to eliminate a similar search, thereby eliminating the trouble of arriving at a document containing information requested by a user, It is to provide a method and a program.

  A document search apparatus according to the present invention for solving the above-described problem is a keyword search means for searching a document including a keyword input by a user from a set of documents to be searched, and outputting a document number of the search result, and a search A keyword context creating means for creating a keyword context by calculating a part including a keyword input by a user for each document corresponding to a document number of a search result of the keyword searching means in the target document set; and the keyword Based on the similarity calculated by the keyword context similarity calculating means for calculating the similarity as a character string between each keyword context created by the context creating means, and the similarity calculated by the keyword context similarity calculating means, A list of search result documents to be presented to the user is determined by determining the presence or absence of duplication of the search result documents searched by the keyword search means. Serial overlap without the determined search result document is characterized in that and a result sort means for outputting rearranging the order in which the higher rank than the determined search result document that there overlap.

(1) According to the inventions described in claims 1 to 5, since the user can exclude similar searches among documents related to the specified keyword, the higher rank output by the search device The information requested by the user can be easily included in the result.

As a result, it is possible to save the trouble of reaching the document containing the information requested by the user, and to easily specify the document.
(2) According to the invention described in claims 2 and 4, the number of search results output by the keyword search means is n / (1-r) (where n is the number of search result documents to be presented to the user and r is the document If r is set to a known duplication rate for each document set, for example, an appropriate number of search result outputs can be determined according to the r.

  In addition, a person who presents the search result (for example, a service designer) can arbitrarily set r, so that the apparatus can be operated according to the situation. That is, even if the processing load of the apparatus becomes heavy, to reduce or reduce the number of duplicate documents to be presented to the user, r is set large, and conversely, it is presented to the user. Even if the number of overlapping documents increases, it is possible to reduce the processing load of the apparatus by setting r small.

The apparatus block diagram which shows the embodiment of this invention. The flowchart which shows the process which each part of the apparatus of FIG. 1 performs. Explanatory drawing which shows the example of an output of the keyword context creation part of FIG. The flowchart which shows the detail of the process which the result rearrangement part of FIG. 1 performs.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings, but the present invention is not limited to the following embodiments. FIG. 1 shows a configuration of a document search apparatus 100 according to an embodiment of the present invention, and FIG. 2 is a flowchart showing a flow of processing performed by each part of the apparatus of FIG.

  The document search apparatus 100 according to the present embodiment includes a keyword search unit 110 as a keyword search unit, a keyword context generation unit 120 as a keyword context generation unit, and a keyword context similarity calculation unit 130 as a keyword context similarity calculation unit. A result rearranging unit 140 and a document set DB (database) 150 as result rearranging means are provided.

  In the document set DB 150 of FIG. 1, a search target document set prepared through collection from the World Wide Web or the like is stored in advance.

  The keyword search unit 110 has a function of searching a document including a keyword input by a user from a search target document set stored in the document set DB 150 and outputting a document number of the search result.

  The keyword context creation unit 120 has a function of creating a keyword context by calculating a location including a keyword input by a user for each document searched by the keyword search unit 110 in the document set DB 150. .

  The keyword context similarity calculation unit 130 has a function of calculating a similarity as a character string between each keyword context created by the keyword context creation unit 120.

  The result sorting unit 140 determines whether or not the search result documents searched by the keyword search unit 110 are duplicated based on the similarity calculated by the similarity calculation unit 130, and presents the search result presented to the user A function of outputting a list of documents by rearranging the search result documents determined to be non-overlapping in an order higher than the search result document determined to be non-overlapping.

  The document search apparatus 100 of FIG. 1 is configured by a computer, for example, and includes hardware resources of a normal computer, such as a ROM, a RAM, a CPU, an input device, an output device, a communication interface, a hard disk, a recording medium, and a driving device thereof. Yes.

  As a result of the cooperation between the hardware resource and the software resource (OS, application, etc.), the document search apparatus 100, as shown in FIG. 1, calculates a keyword context similarity calculation unit 110, a keyword context creation unit 120, and keyword context similarity. The unit 130, the result sorting unit 140, and the document set DB 150 are mounted.

  It is assumed that the document set DB 150 is constructed in storage means / storage means such as a hard disk or RAM.

  Next, the operation of the apparatus configured as described above will be specifically described. The user inputs a keyword to the document search apparatus 100, and the output from the document search apparatus 100 is an ordered list of documents related to the keyword.

  The document search apparatus 100 stores a search target document set prepared in advance through collection from the World Wide Web in the document set DB 150, and performs processing on the document set in the document set DB 150 as a target.

  First, the keyword search unit 110 receives a keyword, searches the document set DB 150 for a document including the keyword, calculates a relevance score, and outputs a document number and a relevance score (step S110).

  Document search and relevance score calculation can be realized by combining a general transposed index and a conventionally known scoring method such as BM25 or PageRank.

  Here, the number of results output from the keyword search unit 110 will be described. When the number of search results to be presented to the user is n, and the duplication rate of documents predicted in the search target document set is r (0 ≦ r <1), the higher the relevance score of the search results n / (1-r) items are output from the keyword search unit 110. Here, as n, typically about 10 to 100 cases are used.

  Next, the keyword context creation unit 120 receives the document number output from the keyword search unit 110, and calculates a location including the keyword input by the user in each document (document in the document set DB 150) as a search result. A keyword context is generated (step S120). For this, a conventional technique called KWIC (keyword in context) or query-biased summary as shown in Non-Patent Document 4 can be used.

  The length of the keyword context to be generated may be arbitrarily set by the service designer according to the quality of the provided service. However, when creating a search service that finally presents a summary sentence to the user, it is preferable that the length is calculated so that the output of the keyword context creation unit 120 can be presented as it is as a summary sentence.

  The output of the keyword context creation unit 120 is a set of document number / keyword context pairs as shown in FIG. FIG. 3 shows an example in which “element” is assumed as a keyword.

  Next, the keyword context similarity calculation unit 130 receives the output of the keyword context creation unit 120 as an input, and calculates the similarity as a character string between each keyword context (step S130). Although the calculation method of the similarity is arbitrary, a commonly used cosine similarity, a Jaccard coefficient when each word is regarded as an element of the set, or the like can be used.

  Next, the result rearrangement unit 140 receives the output of the keyword search unit 110 and the output of the keyword context similarity calculation unit 130 as input, and performs a rearrangement process considering the similarity between the keyword contexts. The search result to be presented is output (step S140).

  The flow of processing performed by the result rearranging unit 140 is shown in FIG. In FIG. 4, a loop L140 represents that steps S143 to S146, which will be described later, are repeatedly executed from the highest search result to the n / (1-r) th item.

  In FIG. 4, first, the presentation result upper list D for including those not including duplication and the duplication includes, but for use when the number of D does not finally reach n. Each result auxiliary list S is initialized (step S141), and i = 1 is set (step S142).

Next, for n / (1-r) items from the keyword search unit 110, the following (1) and (2) are executed in order from the result with the highest relevance score.
(1) Whether or not the keyword context of the i-th search result overlaps with the keyword context of the higher-order search results up to the (i-1) -th search is greater than the predetermined similarity threshold th. The similarity (the calculated value of the keyword context similarity calculation unit 130) is checked to see if it is high (step S143).
(2) If there is no overlap in (1), that is, if all the similarities are less than the threshold th, the i-th result is added to the presentation result upper list D (step S144). If there is an overlap, it is determined that the rank of the i-th result should be lowered and added to the result auxiliary list S (step S145).

  After the processes in steps S144 and S145, i is incremented in step S146.

  Next, the lists D and S obtained as a result of the repetition of the above (1) and (2) are concatenated in this order, and n items are output from the top of the ranking (from the top of the concatenated list) (step S147). .

  As described above, according to the present embodiment, attention is paid to the information (summary sentence) presented to the user, not the document itself, and when the summary sentences are the same (or strongly similar), they are not displayed (presented). As a result, it is possible to increase the variety of appearance for the user and increase the chance of reaching the target information.

  Further, since the number of search result outputs in the keyword search unit 110 is n / (1-r) (n is the number of search result documents to be presented to the user, r is the document set duplication rate), r is, for example, a document set If the known duplication rate is set for each, an appropriate number of search result outputs can be determined according to the r.

  In addition, a person who presents the search result (for example, a service designer) can arbitrarily set r, so that the apparatus can be operated according to the situation. That is, even if the processing load of the apparatus becomes heavy, to reduce or reduce the number of duplicate documents to be presented to the user, r is set large, and conversely, it is presented to the user. Even if the number of overlapping documents increases, it is possible to reduce the processing load of the apparatus by setting r small.

  In addition, some or all of the functions of each means in the document search apparatus according to the present embodiment can be configured by a computer program, and the program can be executed using the computer to realize the present invention. It goes without saying that the procedure in the document search method can be constituted by a computer program and the program can be executed by the computer, and the program for realizing the function by the computer can be read by a computer-readable recording medium such as an FD. (Floppy (registered trademark) Disk), MO (Magneto-Optical disk), ROM (Read Only Memory), memory card, CD (Compact Disk) -ROM, DVD (Digital Versatile D) sk) -ROM, CD-R, CD-RW, HDD, and recorded in a removable disk, or stored, it is possible or distribute. It is also possible to provide the above program through a network such as the Internet or electronic mail.

DESCRIPTION OF SYMBOLS 100 ... Document search device 110 ... Keyword search part 120 ... Keyword context creation part 130 ... Keyword context similarity calculation part 140 ... Result rearrangement part 150 ... Document set DB

Claims (5)

A keyword search means for searching a document including a keyword input by a user from a search target document set and outputting a document number of the search result;
A keyword context creating means for creating a keyword context by calculating a part including a keyword input by a user for each document corresponding to a document number of a search result of the keyword searching means in the search target document set;
A keyword context similarity calculating means for calculating a similarity as a character string between each keyword context created by the keyword context creating means;
Based on the similarity calculated by the keyword context similarity calculation means, it is determined whether or not the search result documents searched by the keyword search means are duplicated, and a list of search result documents to be presented to the user is A document search apparatus, comprising: a result sorting unit that sorts and outputs a search result document determined to have no duplication in a higher order than a search result document judged to have duplication. The keyword search means includes a relevance score as a search result, wherein n is the number of search result documents to be presented to the user, and r is the duplication rate of the document set. -R) output the document number of the search results,
The result rearranging means includes all of the n / (1-r) search results searched by the keyword search means in the search results up to the (i−1) -th search result having a higher relevance score than the i-th search result. In addition, it is determined whether or not there is a similarity whose degree of similarity calculated by the similarity calculation means of the keyword context is higher than a similarity threshold, and the i-th search result when it is determined that the similarity does not exist is obtained. Rearranging in order of higher rank than the i-th search result when it is determined to exist, and outputting a list of the n sorted search result documents from the higher rank. Item 2. The document search device according to Item 1. A keyword search step for searching for a document including a keyword inputted by a user from a set of documents to be searched, and outputting a document number of the search result;
A keyword for creating a keyword context by calculating a location including a keyword input by a user for each document corresponding to a document number of a search result of the keyword search step in a set of documents to be searched by a keyword context creating unit A context creation step;
A keyword context similarity calculating unit that calculates a similarity as a character string between each keyword context created by the keyword context creating step;
The result sorting means determines whether or not the search result documents searched in the keyword search step are duplicated based on the similarity calculated in the keyword context similarity calculation step, and presents the search result to the user A result rearranging step for outputting a list of documents by rearranging the search result documents determined to be non-overlapping in an order higher than the search result document determined to be non-overlapping. Document search method. The keyword search step includes a relevance score as a search result, wherein n is the number of search result documents to be presented to the user, and r is the duplication rate of the document set, n / (1 -R) output the document number of the search results,
In the result sorting step, for all the n / (1-r) search results searched by the keyword search step, the search results are among the search results up to the (i-1) th that have a higher relevance score than the ith. In addition, it is determined whether or not there is an object whose similarity calculated by the similarity calculation step of the keyword context is higher than a similarity threshold, and the i-th search result when it is determined that there is not exists is Rearranging in order of higher rank than the i-th search result when it is determined to exist, and outputting a list of the n sorted search result documents from the higher rank. Item 4. The document search method according to Item 3. A document search program for causing a computer to function as each means according to claim 1.