JP4423841B2 - Keyword determination device, determination method, document search device, search method, document classification device, classification method, and program - Google Patents

Description

[0001]
BACKGROUND OF THE INVENTION
  The present invention is a keywordDecisionapparatus,DecisionMethod, document search apparatus, search method, document classification apparatus, classification method, and program, particularly keyword for hypertextDecisionapparatus,DecisionThe present invention relates to a method, a document search device, a search method, a document classification device, a classification method, and a program.
[0002]
Hypertext is a set of documents structured by hyperlinks (links), and has a structure in which documents are nodes and links are made between documents. Regarding the link from document A to document B, the character string in document A that is the link source of document B is called an anchor character string. A representative example of hypertext is WWW (World Wide Web). WWW is hypertext described in HTML (Hyper Text Markup Language) format, and links and anchor character strings are marked by <A> tags. Although WWW is a representative example of hypertext, the present invention is not limited to the WWW. Further, the hypertext can be described using not only HTML but also XML (Extensible Markup Language), SGML (Standard Generalized Markup Language), and the like.
[0003]
[Prior art]
With the recent spread of the Internet, it is possible to easily access a large number of documents structured by links. Documents structured with links usually form document clusters called sites. The unit of the document cluster that constitutes a site is not necessarily clear, but a document cluster with the same domain name can be regarded as one site, or a document cluster of the same theme by the same administrator can be regarded as one site. Many. Each site has an entrance document called a top page, from which a viewer can access a document of each content in the site by following a link.
[0004]
For example, in the example of FIG. 2 described later, the document 11, the document 21, the document 31, and the document 41 are the top pages of the site 1, the site 2, the site 3, and the site 4, respectively. The link includes a URL that indicates the storage location of the document to be cited and an anchor character string that indicates the content of the document to be cited. You can link freely to any accessible document, but due to Internet customs, links from different sites have the top page specified in the URL, and the anchor string represents the site title or site content. There is a tendency to use strings.
[0005]
On the other hand, links to documents of each content in the same site are often omitted in the assumption that the viewer follows the links in order from the top page. For example, if the site 1 is a site that deals with gourmet information across the country, the anchor character strings “L203” and “L302” of links from different sites such as the site 2 and the site 3 to the document 11 that is the top page are displayed. Is often used to express the site title itself and the site content, such as “Gourmet” page and “Restaurant search”.
[0006]
On the other hand, viewers such as “Kansai”, “Nara Prefecture”, and “Chinese” link to the anchor character strings “L101”, “L103”, and “L106” of links to the contents in the site in order from the top page. There is a tendency that the minimum necessary expression is used on the assumption that the document is followed. In many cases, the content of the linked document can be grasped only partially by the anchor character string alone. In addition, since the content of the document itself is described on the assumption that the viewer follows the links in order from the top page, for example, a document with an anchor character string “Nara Prefecture” contains the name of the prefecture. In many cases, a document alone cannot fully grasp the contents, such as only a list being described.
[0007]
As a conventional technique for searching and classifying a document structured by such a link, for example, Proceedings of the 1st International Conference on the World Wide Web, 1994), a search device shown in GENVL and WWWW: Tools for Taming the Web, and Japanese Patent No. 3108015 A hypertext search device described in JP-A-10-254899, a document classification system described in JP-A-10-254899, a keyword assignment method using link information described in a republished patent WO99 / 14690, and JP-A-2000-339320 Unlink source, such as the related document display device described And a method for the over string searched and classified.
[0008]
According to these search devices, document classification systems, and keyword assignment methods, in addition to the keywords included in the document body, by registering the link source anchor character string in the search index or adding it to the document feature vector, Using the property that the anchor character string of the link source describes the description of the link destination document, the search / classification accuracy is improved.
[0009]
[Problems to be solved by the invention]
However, the above-described conventional keyword extraction device, document search device, and document classification device have the following problems. That is, the first problem is that, even if the anchor character string of the link source of the document is searched and classified for the links in the site, the anchor character string is not necessarily a description that completely describes the document contents. is there. This is because the conventional keyword extraction device, document search device, and document classification device search only the text of a single page or the link source anchor character string. Because the content of links and pages within a site tends to be written assuming that the viewer follows the links in order from the top page, in the text of a single page or the link source anchor string, Search and classification that fully reflects the contents of the document may not be possible.
[0014]
  The present invention has been made in view of the above points, and a series of anchor character strings obtained by tracing back a link from outside the site representing the contents of the entire site and a link within the site representing the position of the document within the site. Can be searched and classified by reflecting the contents of the entire site and the position of the document within the site by extracting the word as a group of words characterizing the relationship between the document cluster and the documentDecisionapparatus,DecisionIt is an object to provide a method, a document search device, a search method, a document classification device, a classification method, and a program.
[0016]
[Means for Solving the Problems]
  In order to achieve the above object, the keyword of the present inventionDecisionapparatusIs the link information indicating the document body, the link destination document of the document, and the link source document for each of the plurality of hierarchized documents included in each of the document clusters formed by a plurality of documents structured by hyperlinks. And a storage unit storing at least an anchor character string that is a character string in a link source document of the document, and a document having the same condition as the top page that is the highest document in the document cluster determined from the positional relationship in the directory hierarchy Target specification means for specifying document cluster conditions including the conditions of documents included in the cluster, and the document text, link information, and anchor character strings of a plurality of documents are read from the storage means, and specified by the link information and target specification means. The given document is the top page or the top page Document cluster specifying means for specifying the document cluster to which the document cluster belongs based on the conditions of the documents included in the same document cluster, and link information of the document cluster specified by the document cluster specifying means Based on the above, for each of multiple documents in the same document cluster, when each document is a link destination document, each link obtained by going back to the link source document direction that is the upper layer up to the highest document in the document cluster It is a character string in a document in another document cluster that is determined as an in-site keyword of the document as a sequence of anchor strings, which is a character string in the original document, and is linked to the top-level document of the document cluster. Determining means for determining an anchor character string as an off-site keyword for each document in the document cluster.Features.
  In order to achieve the above object, the keyword determination method according to the present invention provides a data processing device having a plurality of hierarchies provided in each of document clusters formed by a plurality of documents structured by hyperlinks. For each document, a plurality of documents are stored from a storage means storing at least the document text, link information indicating the link destination document and link source document of the document, and an anchor character string that is a character string in the link source document of the document. A reading step for reading out the document body, link information, and anchor character string, and a data processing device for a document included in the same document cluster as the top page condition in the document cluster determined from the positional relationship in the directory hierarchy. A target specifying step for specifying a condition of a document cluster including the condition, and a data processing device, Based on the link information read in the extraction step and the document cluster conditions specified in the target specification step, if the given document is the top page or not the top page, the documents included in the same document cluster The document cluster specifying step for specifying the document cluster to which the document cluster belongs based on the conditions of the document cluster, and the data processing device uses the same document cluster based on the link information of the document cluster specified by the document cluster specifying step. For each of a plurality of documents, when each document is a link destination document, the character string in each link source document obtained by tracing back the link in the direction of the link source document that is the upper layer up to the top document in the document cluster Is determined as the keyword in the site of the document, and the maximum of the document cluster is determined. Position, characterized in that it comprises a determining step of determining the anchor character string is a character string in a document in another document clusters linked to a document as a site outside keywords for each document in the document clusters.
[0017]
  In order to achieve the above object, the document search apparatus of the present invention provides:An anchor that is a document text, link information, and a character string in a link source document of each of a plurality of hierarchized documents included in each of document clusters formed by a plurality of documents structured by hyperlinks Document cluster conditions including storage means storing at least a character string and conditions of a document included in the same document cluster as a top page condition in a document cluster determined from a positional relationship in a directory hierarchy The object designation means for designating the document body, the document text, link information and anchor character strings of a plurality of documents from the storage means are read out, and the given document is determined based on the link information and the document cluster condition designated by the object designation means. If it is the top page or not the top page, the conditions of the documents included in the same document cluster Based on the document cluster specifying means for specifying the document cluster to which the document cluster belongs, and for each of a plurality of documents in the same document cluster based on the link information of the document cluster specified by the document cluster specifying means When each document is a link destination document, a series of anchor character strings that are character strings in each link source document obtained by tracing back the link in the direction of the link source document up to the topmost document in the document cluster Decision to determine as an in-site keyword and an anchor character string, which is a character string in a document in another document cluster linked to the top document in the document cluster, as an off-site keyword for each document in the document cluster And the key of the keyword in the site and the keyword outside the site of each document determined by the determination unit Index creation and storage means for creating and storing in which document a document appears as an index, an input means for inputting search conditions, and a keyword used for the search from the search conditions input by the input means As a search result candidate based on the index stored in the index creation and storage means, the document in which the keyword used for the search appearsAnd a search means for searching.
[0018]
  In order to achieve the above object, the document search method of the present invention includes a data processing device,Document text for each of a plurality of hierarchized documents included in each of a plurality of document clusters formed by a plurality of documents structured by hyperlinks, link information indicating the link destination document and link source document of the document, and A reading step of reading out document texts, link information, and anchor character strings of a plurality of documents from a storage unit that stores at least an anchor character string that is a character string in a link source document of the document, and a data processing device in a directory hierarchy A target specifying step for specifying a condition of a document cluster including a condition of a document included in the same document cluster as a condition of a top page which is a top-level document in the document cluster determined from the positional relationship, and a reading step by the data processing device The link information read in step 1 and the document clip specified in the target specification step Based on the conditions of the star cluster, if the given document is the top page, or if it is not the top page, the document cluster to which the document cluster belongs is specified based on the conditions of the documents included in the same document cluster. The document cluster specifying step to be performed, and when the data processing apparatus sets each document as a linked document for each of a plurality of documents in the same document cluster based on the link information of the document cluster specified by the document cluster specifying step. A series of anchor character strings that are character strings in each link source document obtained by going back to the link source document direction that is the upper layer up to the top document in the document cluster is determined as a keyword in the site of the document, and the document An anchor statement that is a string in a document in another document cluster linked to the top document in the cluster A determination step for determining a column as an off-site keyword for each document in the document cluster, and a keyword that is included in which document among the in-site keyword and the off-site keyword for each document determined by the determination step. An index creation and storage step for creating and storing an appearance as an index, an input step for inputting a search condition by an input means, and a data processing device determining a keyword to be used for the search from the input search condition Then, the document in which the keyword used for the search appears is selected as a search result candidate based on the index stored by the index creation and storage step.And a search step for searching.
[0019]
  In order to achieve the above object, the document classification apparatus of the present invention provides:Document text for each of a plurality of hierarchized documents included in each of a plurality of document clusters formed by a plurality of documents structured by hyperlinks, link information indicating the link destination document and link source document of the document, and A storage means storing at least an anchor character string, which is a character string in a link source document of the document, and the same document cluster as the condition of the top page in the document cluster determined from the positional relationship in the directory hierarchy A document that is designated by the link information and the target designation unit by reading out the document text, link information, and anchor character strings of a plurality of documents from the storage unit Based on the cluster conditions, the given document is the top page or not the top page The document cluster specifying means for specifying the document cluster to which the document cluster belongs based on the conditions of the documents included in the same document cluster, and the link information of the document cluster specified by the document cluster specifying means. Based on this, for each of a plurality of documents in the same document cluster, when each document is a linked document, the characters in each link source document that can be obtained by going back to the link source document up to the topmost document in the document cluster A series of anchor character strings that are columns is determined as a keyword in the site of the document, and an anchor character string that is a character string in a document in another document cluster linked to the top-level document of the document cluster is a document. A determining means for determining as an off-site keyword for each document in the cluster, and an in-site keyword for each document determined by the determining means. Document vector creation means for creating a document vector that expresses the frequency or occurrence information of keywords and out-of-site keywords in each document as a multidimensional vector, and a characteristic keyword and its appearance for each category to be classified A category condition storage means whose frequency is stored as a feature vector, and a document vector of each document created by the document vector creation means, and a feature vector of each category stored in the category condition storage meansIt is characterized by having similarity calculation means for performing similarity calculation and classification means for classifying a plurality of documents according to the similarity calculation result.
[0020]
  In order to achieve the above object, the document classification method of the present invention includes:A data processing apparatus determines a document body, a link destination document and a link source document of each of a plurality of hierarchical documents included in each of document clusters formed by a plurality of documents structured by hyperlinks. A reading step for reading out document texts, link information and anchor character strings of a plurality of documents from storage means storing at least link information to be displayed and anchor character strings which are character strings in the link source document of the document, and a data processing device Is a target specifying step for specifying a document cluster condition including a condition of a document included in the same document cluster as a condition of a top page that is a top-level document in a document cluster determined from a positional relationship in a directory hierarchy, and data processing The device uses the link information read in the reading step and the target specifying step. Based on the specified document cluster conditions, the given document is the top page, or if it is not the top page, the document that belongs to which document cluster is based on the conditions of the documents included in the same document cluster A document cluster specifying step for specifying a cluster, and the data processing apparatus determines each document as a linked document for each of a plurality of documents in the same document cluster based on the link information of the document cluster specified by the document cluster specifying step. When this is done, a sequence of anchor strings, which is a character string in each link source document obtained by tracing back the link in the direction of the link source document, which is the upper layer up to the top document in the document cluster, is used as a keyword in the site of the document. A character in a document in another document cluster that is determined and linked to the top document in the document cluster A determination step for determining an anchor character string as an out-of-site keyword for each document in the document cluster, and the data processing apparatus causes the in-site keyword and off-site keyword for each document determined in the determination step to appear in each document. A document vector creation step for creating a document vector that expresses frequency or occurrence information as a multidimensional vector, and the data processing device stores a characteristic keyword and its appearance frequency as a feature vector for each category to be classified. Between the feature vector read from the category condition storage means and the document vector of each document created by the document vector creation step.A similarity calculation step for performing similarity calculation and a data processing apparatus include a classification step for classifying a plurality of documents according to the similarity calculation result obtained by the similarity calculation step.
[0021]
  In order to achieve the above object, the program of the present inventionThe above-described keyword determination method, document search method, and document classification method are executed by a computer in a data processing apparatus.
[0025]
DETAILED DESCRIPTION OF THE INVENTION
Next, embodiments of the present invention will be described in detail with reference to the drawings.
[0026]
  [First Embodiment]
  FIG. 1 shows a block diagram of a first embodiment of the present invention. As shown in FIG.SettingThe first embodiment of the device, the document search device, and the document classification device is configured to include a data processing device 1 that operates under program control and a storage device 2 that stores information.
[0027]
The storage device 2 includes a hypertext database 21 and a document keyword storage unit 22. The hypertext database 21 stores URLs, local addresses, body texts, linked documents and their anchor character strings, etc. of each document for a document group structured with hyperlinks as shown in FIG. An example of the hypertext database 21 corresponds to, for example, the Internet (Web) on the Internet or an intranet.
[0028]
The document keyword storage unit 22 stores a keyword determined by the document keyword determination unit 14 described later for each document. The keyword determined by the document keyword determining means 14 includes a series of anchor character strings (site keywords) obtained by tracing back links in the same site, and an anchor character string of links from different sites to the top page of the site ( There are two types of keywords.
[0029]
When the hypertext database 21 is a document group structured with hyperlinks as shown in FIG. 2, examples of keywords stored in the document keyword storage unit 22 are as shown in FIG. In FIG. 3, each document is stored in the document keyword storage unit 22 in association with the document name, the keyword outside the site, and the keyword inside the site. For example, “L203, L302” is set as the keyword outside the site of the document 15. It can be seen that “L104 ← L101” is stored as the keyword in the site.
[0030]
On the other hand, the data storage device 1 of FIG. 1 includes a hypertext access unit 11, a document cluster information acquisition unit 12, a target designation unit 13, and a document keyword determination unit 14.
[0031]
The hypertext access unit 11 reads a document stored in the hypertext database 21 and passes it to the document cluster information acquisition unit 12. When the hypertext database 21 is WWW, a document can be accessed via HTTP (Hyper Text Transfer Protocol). Such a function is conventionally realized in a web browser such as IE (Internet Explorer) or a web crawler (spider / robot).
[0032]
The document cluster information acquisition unit 12 extracts link information included in the document read by the hypertext access unit 11, specifies the document cluster that constitutes the site based on the conditions specified by the target specification unit 13, and the document reference relationship Generate tables and document cluster tables. An example of the document reference relation table is shown in FIG. 4, and an example of the document cluster table is shown in FIG.
[0033]
As shown in FIG. 4, the document reference relation table is a list in which an anchor character string, a link source document, and a link destination document are associated with each other. Indicates that a link has been established. As shown in FIG. 5, the document cluster table is a list in which a document cluster, a top page, and documents in the cluster are associated with each other. For example, the top page of the document cluster “site 1” is the document 11 and Indicates that documents 12 to 19 are included.
[0034]
The target designating unit 13 in the data processing apparatus 1 in FIG. 1 gives the document cluster information obtaining unit 12 the conditions for the document cluster to be regarded as the same site. The conditions given by the target designating means 13 include “site top page conditions” and “document conditions included in the same site”. For example, when a document cluster stored in a server having the same domain name is regarded as one site, “the URL of the document is“ http: // domain name / ”or“ http ” : // domain name / index.html ”and“ same domain name ”may be specified as“ a condition for documents included in the same site ”.
[0035]
The document keyword determination unit 14 in the data processing apparatus 1 refers to a series of anchor character strings obtained retrospectively in the same document cluster while referring to the document reference relation table and the document cluster table generated by the document cluster information acquisition unit 12. Then, an anchor character string of a link from a different document cluster is determined as a keyword of the document and stored in the document keyword storage unit 22.
[0036]
Next, the operation of the first embodiment will be described in detail with reference to the block diagram of FIG. 1 to the flowchart of FIG. First, the hypertext access unit 11 reads each document stored in the hypertext database 21 and passes it to the document cluster information acquisition unit 12. The document cluster information acquisition unit 12 extracts link information from the given document, and generates a document reference relation table as shown in FIG. 4 (step S1).
[0037]
Next, the document cluster information acquisition unit 12 determines whether the given document is the top page based on the “site top page condition” specified by the target specification unit 13. Here, the top page is the highest level document in the document cluster determined from the positional relationship in the directory hierarchy (in the case of FIG. 2, it is the document 11 at the site 1 and the document 31 at the site 3).
[0038]
If it is a top page, one line is added and registered in the document cluster table shown in FIG. 5 (step S2). For example, when “the URL of the document is“ http: // domain name / ”or“ http: // domain name / index.html ”” is specified as “the condition of the top page of the site”, The top page is registered in the document cluster table for each domain name.
[0039]
If the document cluster information acquisition unit 12 determines that the given document is not the top page, the document cluster information acquisition unit 12 determines that it is not the top page based on the “condition of the document included in the same site” specified by the target specification unit 13. It is determined to which site the recorded document belongs, and it is registered in the intra-cluster document of the document cluster table shown in FIG. 5 (step S3). For example, if “the domain name is the same” is designated as the “condition of documents included in the same site”, a document having the same domain name as the top page is registered in the intra-cluster document.
[0040]
Next, the document keyword determination unit 14 refers to the document reference relation table and the document cluster table generated by the document cluster information acquisition unit 12, and anchors the links placed from outside the site to the top page of each site. The character string is stored in the document keyword storage unit 22 as an off-site keyword (step S4).
[0041]
Further, the document keyword determination unit 14 refers to the document reference relation table and the document cluster table generated by the document cluster information acquisition unit 12, and for each intra-cluster document, the anchor character obtained by tracing back the link of the same intra-cluster document. The series of columns is stored in the document keyword storage unit 22 as an in-site keyword (step S5). At this time, the off-site keywords of the documents included in the same site are the same as the off-site keywords of the top page of the site. Therefore, the off-site keywords of the document 12 to the document 19 in FIG. 2 are the same as “L203, L302” as the off-site keyword of the document 11.
[0042]
Also, when going back to the link, remember the document that went back once, so that it doesn't go back in a loop. For example, when the link to the document 16 in FIG. An infinite number of anchor character strings are generated by a loop like L105 ← L109 ←. Therefore, if a document that has been traced once is not traced twice in the same anchor character string series, the in-site keyword of the document 16 is only “L105 ← L102”.
[0043]
On the other hand, when the same document is traced back by another anchor character string series, it is registered as a different keyword. For example, in the case of the document 19 in FIG. 2, “L108 ← L104 ← L101” and “L110 ← L105 ← L102” are both anchor character string sequences that go back to the document 11, but both are in the site because they are different sequences. Remember as a keyword. In this case, an anchor character string series such as “L110 ← L105 ← L109 ← L105 ← L102” can be considered. This is because the site 13 keyword and the document 16 are traced back twice in the same series. I don't remember.
[0044]
In this embodiment, the method in which the hypertext access unit 11 accesses the hypertext database 21 stored in the storage device 2 has been described. However, the hypertext database 21 is directly accessed from the Internet and the storage device 2 is accessed. And the present invention is not limited to the method described in this embodiment mode.
[0045]
In the present embodiment, “the URL of the document is“ http: // domain name / ”or“ http: // domain name / index ”as the“ site top page condition ”designated by the target designating means 13. .Html ”, and“ the domain name is the same ”as the“ condition of documents included in the same site ”. However, the “site top page condition” is “a document with a certain number of links from pages with different domain names”, and the “domain condition of the document included in the same site” is “directory hierarchy of top page and URL in the same domain” There is also a method of specifying “same or deep document”. Ordinarily, a directory name beginning with a tilde “~” can be regarded as the site of each user using the server.
[0046]
In addition, the “site top page condition” is “link destination document with an anchor character string that is considered to indicate the top page, such as“ Home Page ”,“ Top ”,“ return to the beginning ”, etc. There is also a method in which “a condition included in the same site” is “a link source document of a link having an anchor character string of an expression that is considered to indicate a top page, such as“ Home Page ”,“ Top ”, and“ Return to Top ”. Further, there is a method of using a list of URLs specified by hand in advance as the “site top page condition”, and the method is not limited to the method described in this embodiment.
[0047]
In the present embodiment, the document keyword determination means 14 uses an anchor character string series obtained by tracing back links of documents in the same cluster as keywords in the site. If it is, the anchor character string series that goes back one link may be stored as an in-site keyword. In addition, there is a method of limiting the number of links to the designated number of links without necessarily going back all the links to the top page, and the method is not limited to the method described in the present embodiment.
[0048]
In the present embodiment, the document keyword determination unit 14 excludes the anchor character string series of the looped link from the site keyword. However, other keywords such as “Back”, “Back”, “Top”, “Home Page”, “Previous”, “Next”, etc. that are not suitable for search / classification are stored in advance as a dictionary, and their character strings are included. There is also a method of not registering the anchor character string series as an in-site keyword. Also, a method of not registering an anchor character string series in which the number of documents going back as long as a certain value is registered as an in-site keyword, or a method of registering only a series of anchor character strings in the top s ways with a small number of documents going back as site keywords And the present invention is not limited to the method described in this embodiment mode.
[0049]
In the present embodiment, the document keyword determination means 14 determines a keyword based on the anchor character string, but in addition to the anchor character string, the document title, a fixed-length character string around the anchor character string, the anchor A character string enclosed in table tags around the character string, a character string enclosed in a list tag around the anchor character string, a character string enclosed in <BR> or <P> tags around the anchor character string, <H> There is a method of using a keyword including a character string in which a tag, font size, and color are emphasized, and the method is not limited to the method described in this embodiment.
[0050]
In the present embodiment, only the off-site keyword and the in-site keyword are stored in the document keyword storage unit 22, but the document title, body text, and the like may be stored as keywords. It is not limited to the method described. In this embodiment, the operation is described as executing step S3 for specifying a document cluster after step S2 for specifying the top page. However, after executing step S3 for specifying the document cluster, the top page is first executed. There is also a method of executing step S2 for specifying the method, and is not limited to the method described in the present embodiment.
[0051]
In the present embodiment, the operation has been described as executing step S5 for determining an in-site keyword after step S4 for determining an off-site keyword. However, after executing step S5 for determining an in-site keyword first. There is also a method of executing step S4 for determining an off-site keyword, and is not limited to the method described in the present embodiment.
[0052]
Next, effects of the first exemplary embodiment of the present invention will be described. In the present embodiment, a series of anchor character strings obtained by tracing back a link from outside the site representing the contents of the entire site and a link within the site representing the position of the document within the site are divided into a document cluster and a document. This is extracted as a word group characterizing the relationship. Therefore, for each document, a keyword reflecting the contents of the entire site and the position of the document in the site can be obtained.
[0053]
  [Second Embodiment]
  Next, a second embodiment of the present invention will be described with reference to the drawings. FIG. 7 shows a block diagram of the second embodiment of the present invention. As shown in FIG.SettingThe second embodiment of the apparatus, the document search apparatus, and the document classification apparatus includes a data processing device 5 that operates under program control, a storage device 6 that stores information, an input unit 3, and an output unit 4. . In the figure, the same components as those in FIG.
[0054]
In the second embodiment of the present invention, the data processing device 5 includes an index creating unit 15 and an index searching unit 16 in addition to the configuration of the data processing device 1 in the first embodiment shown in FIG. It is different in point. Moreover, the storage device 6 is different in that it includes a first index storage unit 23 in addition to the configuration of the storage device 2 in the first embodiment shown in FIG. Further, in addition to the first embodiment shown in FIG. 1, the difference is that an input means 3 such as a keyboard and an output means 4 such as a display device and a printing device are provided.
[0055]
In FIG. 7, the first index storage unit 23 in the storage device 6 stores an index generated by the index creating unit 15 based on the data in the document keyword storage unit 22. The index creation means 15 in the data processing device 5 reads the off-site keyword and the on-site keyword of each document stored in the document keyword storage unit 22, and which keyword appears as the off-site keyword or the on-site keyword of which document. Is created as an index and stored in the first index storage unit 23. The index search unit 16 in the data processing device 5 searches the first index storage unit 23 according to the search condition input from the input unit 3 and outputs the result to the output unit 4.
[0056]
Next, the operation of the second embodiment will be described in detail with reference to the drawings. In this embodiment mode, there are two types of processing with different operation timings, namely, registration processing according to the flowchart shown in FIG. 8A and search processing according to the flowchart shown in FIG. The search process is performed every time there is an input from the user, whereas the registration process only needs to be performed once in advance.
[0057]
First, registration processing according to the second embodiment will be described with reference to the flowchart of FIG. In FIG. 8A, the same processing steps as those in FIG. That is, in the flowchart of the registration process shown in FIG. 8A, the hypertext access unit 11, the document cluster information acquisition unit 12, the target designation unit 13, and the document keyword determination unit 14 in the present embodiment shown in steps S1 to S5. Since the operation of is the same as the operation of each of the means 11, 12, 13 and 14 of the first embodiment, the description thereof is omitted.
[0058]
In the first embodiment, the process is terminated when the in-site keyword is determined in step S5. In this embodiment, based on the document keyword generated as a result of step S5, the index creating means 15 creates an index indicating which word is registered in which document for the off-site keyword (step S6). Subsequently, the index creating means 15 creates an index indicating which word is registered in which document for the keyword in the site (step S7). This completes the registration process.
[0059]
Next, the search process will be described with reference to the flowchart of FIG. First, a search condition is input from the input means 3 (step T1). What is input as a search condition includes, in addition to keywords, a question sentence in a natural language, another document similar to a document to be searched, and the like.
[0060]
Next, the index search means 16 determines the keyword n word used for a search from the input search conditions (step T2). The keyword determination method includes two processes, sentence division and keyword selection. For example, there is a method in which morphological analysis is used for sentence division, and the remaining words excluding attached words such as “no” are used as keywords for keyword selection.
[0061]
Next, the index search means 16 checks whether there are any words that appear in the off-site keyword among the keywords divided into n words. If it appears, the keyword m word (1 ≦ m ≦ n−1), its appearance frequency, and the document in which the keyword appears are stored as search result candidates (step T3).
[0062]
Next, the index search means 16 additionally registers, in the search result list, a document in which the remaining nm words appear in the site keyword among the documents that are search result candidates and the appearance frequency of the keyword ( In step T4), the search result list is sorted by keyword appearance frequency, and the search result is displayed to the user using the output means 4 (step T5).
[0063]
In addition, this invention is not limited to 2nd Embodiment, The following various modifications are also included. That is, in the second embodiment, only the off-site keyword and the in-site keyword are stored in the document keyword storage unit 22, but the site title and the site text are also stored as keywords, and the search is performed after the search keyword is divided. The target method may be used. In this embodiment, the search result list is sorted by the appearance frequency of the keyword. However, the appearance frequency of the keyword outside the site and the appearance frequency of the keyword inside the site are multiplied by different weights, and sorted by the result. You may adopt the method of doing.
[0064]
In this embodiment, the search method / search model is assumed to be based on keyword matching. However, as the search method / search model, a vector space model, a probability model, a Boolean model for performing AND and OR operations, etc. It may be a method.
[0065]
Further, in this embodiment, step S7 for creating an index for in-site keywords is executed after step S6 for creating an index for keywords outside the site. Step S6 for creating an index of outside keywords may be executed.
[0066]
Further, in the present embodiment, after step S4 for determining the keyword outside the site and step S5 for determining the keyword within the site, step S6 for creating an index for the keyword outside the site and step S7 for creating the index for the keyword within the site, respectively. Although it is executed, step S6 for creating an index for the off-site keyword is executed after step S4 for determining the keyword outside the site, and step S7 for creating an index for the keyword inside the site after step S5 for determining the keyword within the site. May be executed.
[0067]
In the present embodiment, the method of dividing a sentence using morphological analysis in the step T2 for determining a search keyword has been described. However, it is divided into other character types such as kanji, alphanumeric, katakana, and hiragana. There are also methods such as dividing by a certain number of characters, dividing by spaces or punctuation marks, and dividing by an attached word such as “no”, and are not limited to the method described in this embodiment.
[0068]
In the present embodiment, the method of excluding ancillary words such as “no” in keyword selection in step T2 for determining a search keyword has been described, but other general information such as “information” and “method” are also used. Exclude words with high frequency in the document as unnecessary words, or add only a low score even if the search hits, and conversely, in the question text in comparison with the frequency in general documents There is a method of adding a high score when a search hits a word having a high appearance frequency as an important word, and is not limited to the method described in the present embodiment.
[0069]
In the present embodiment, the method of storing a document as a search result candidate when one or more keywords appear in step T3 of searching for an off-site keyword has been described. There is also a method of executing step T4 of searching for in-site keywords using all documents as search result candidates when no hit is found. If either the keyword outside the site or the keyword inside the site is hit, it is included in the search result list, and in step T5 for outputting the search result, the score of the document is determined depending on whether it is hit outside the site or inside the site. There is also a method of sorting by changing the weight.
[0070]
Next, the effect of the second embodiment will be described. In the present embodiment, a series of anchor character strings obtained by tracing back a link from outside the site representing the contents of the entire site and a link within the site representing the position of the document within the site are divided into a document cluster and a document. It is extracted as a group of words that characterize the relationship between them, and an index is created. Thereby, it is possible to perform a search reflecting the contents of the entire site and the position of the document in the site.
[0071]
Also, in this embodiment, when a keyword is divided by a space, connected by “no”, or entered as a compound word in a search condition, the keyword is divided to indicate the contents of the entire site. And a series of anchor character strings obtained by tracing back a link in the site and a link in the site representing the position of the document in the site. This makes it possible to perform an effective refined search that reflects the site structure.
[0072]
  [Third Embodiment]
  Next, a third embodiment of the present invention will be described in detail with reference to the drawings. FIG. 9 shows a block diagram of a third embodiment of the present invention. As shown in FIG.SettingThe third embodiment of the apparatus, the document search apparatus, and the document classification apparatus includes a data processing device 7 that operates under program control and a storage device 8 that stores information. In the figure, the same components as those in FIG.
[0073]
In the third embodiment of the present invention, as shown in FIG. 9, the data processing device 7 has a document vector creating means in addition to the configuration of the data processing device 1 in the first embodiment shown in FIG. 17 and the point of having similarity calculation means 18. In addition to the configuration of the storage device 2 in the first embodiment shown in FIG. 1, the storage device 8 includes a document vector storage unit 24, a category condition storage unit 25, and a classification result storage. It differs in that it has a portion 26.
[0074]
In the document vector storage unit 24, feature vectors of each document created by the document vector creation unit 17 based on the keywords stored in the document keyword storage unit 22 are stored. The document feature vector is, for example, each keyword appearing in the document and its appearance frequency expressed as a multidimensional vector.
[0075]
If a feature vector is determined for each of a plurality of documents, the similarity between documents can be calculated from the Euclidean distance between the feature vectors, the angle formed by the feature vectors, and the like. In addition, the sum or centroid of the feature vectors of a plurality of documents belonging to a certain category is regarded as the feature vector of the category, and by calculating the similarity between the feature vector of the category and the feature vector of the unclassified document, It is also possible to determine which category it belongs to.
[0076]
The category condition storage unit 25 stores characteristic keywords and their appearance frequencies as feature vectors for the categories to be classified. The classification result storage unit 26 stores the result of calculating the cosine of the document vector and the feature vector of each category by the similarity calculation means 18. As a result, it can be determined that the larger the value, the more the document belongs to the category.
[0077]
Based on the document keywords stored in the document keyword storage unit 22, the document vector creation means 17 determines which keyword is which part of each document (whether it is an off-site keyword, an in-site keyword, a title, or a text). Is stored in the document vector storage unit 24 as a document vector.
[0078]
The similarity calculation means 18 calculates the cosine of the feature vector of each category stored in the category condition storage unit 25 for the document vector of each document stored in the document vector storage unit 24, and classifies the result. The result is stored in the result storage unit 26.
[0079]
Next, the operation of the present embodiment will be described in detail with reference to the flowchart of FIG. In FIG. 10, the same processing steps as those in FIG. 6 are denoted by the same reference numerals, and the description thereof is omitted. That is, the operations of the hypertext access unit 11, the document cluster information acquisition unit 12, the target designation unit 13, and the document keyword determination unit 14 in the present embodiment shown in steps S1 to S5 in FIG. 10 are the same as those in the first embodiment. Since the operations are the same as those of the means 11, 12, 13 and 14, description thereof will be omitted.
[0080]
In the first embodiment, the process is terminated when the in-site keyword is determined in step S5. In the present embodiment, based on the document keyword generated as a result of step S5, the document vector creation means 17 determines which part of each document is which keyword (external keyword, in-site keyword, title, text) Or the like) is stored in the document vector storage unit 24 as a document vector (step S8).
[0081]
Next, the similarity calculation means 18 calculates the cosine of the feature vector of each category stored in the category condition storage unit 25 for the document vector of each document stored in the document vector storage unit 24, and the result Is stored in the classification result storage unit 26 (step S9).
[0082]
In this embodiment, keywords, their appearance parts (non-site keywords, in-site keywords, titles, body texts, etc.) and appearance frequencies are used as document vectors. A method of not performing, a method of weighting the appearance frequency according to the appearance part, a method of using only information on whether or not the appearance frequency is used instead of the appearance frequency may be employed.
[0083]
In the present embodiment, the cosine of a vector is taken as the similarity calculation of the document, but a method using the Euclidean distance between the vectors may be used as the similarity calculation. In this embodiment, the feature vector of each category is specified as the category condition. However, instead of specifying a category feature vector, a document actually included in a category is specified as teacher data, and a learning model generated as a result of learning using machine learning such as SVM is used as a category condition. A method of classifying unlearned documents using the learning model by the degree calculation means 18 may be used. For details on document classification using SVM, see Proceedings of 10th European Conference on Machine Learning, pp.137-I42, 1988, Proceedings of 10th European Conference on Machine Learning. , 1998).
[0084]
Next, the effect of the third embodiment will be described. In the present embodiment, a series of anchor character strings obtained by tracing back a link from outside the site representing the contents of the entire site and a link within the site representing the position of the document within the site are divided into a document cluster and a document. A word vector is extracted as a group of words that characterizes the relationship between and a document vector. Thereby, it is possible to perform classification that reflects the contents of the entire site and the position of the document in the site.
[0085]
  [Fourth Embodiment]
  Next, a fourth embodiment of the present invention will be described in detail with reference to the drawings. FIG. 11 shows a block diagram of a fourth embodiment of the present invention. As shown in FIG.SettingThe fourth embodiment of the apparatus, document search apparatus, and document classification apparatus includes a data processing device 9 that operates under program control, a storage device 11 that stores information, an input means 3 and an output means 4. . In the figure, the same components as those in FIG.
[0086]
As shown in FIG. 11, in the fourth embodiment of the present invention, the data processing device 9 has a hypertext access means 11 based on the configuration of the data processing device 5 in the second embodiment shown in FIG. , Except that the document cluster information acquisition unit 12, the target designation unit 13, the document keyword determination unit 14, and the index creation unit 15 are excluded. In addition, the storage device 10 has the configuration of the storage device 6 in the second embodiment shown in FIG. 7 except for the hypertext database 21 and the document keyword storage unit 22, and newly adds a second index storage unit 27. It is different in having.
[0087]
The first index storage unit 23 stores an off-site keyword and an on-site keyword index as meta information representing the content of the document. The second index storage unit 27 stores an index of keywords that appear in the text of the document.
[0088]
Next, the operation of the present embodiment will be described in detail with reference to the flowchart of FIG. In FIG. 12, the same processing steps as those in FIG. 8B are denoted by the same reference numerals, and the description thereof is omitted. The operation of the index search means 16 in the present embodiment shown in steps Tl, T2 and T5 in FIG. 12 is the same as the operation of the index search means 16 in the second embodiment, and a description thereof will be omitted.
[0089]
In the second embodiment, after step T2 for determining a keyword, the index search means 16 searches for an off-site keyword and an on-site keyword, respectively. In the present embodiment, the index search means 16 searches the first index that is the index of the keyword outside the site and the keyword within the site, and searches the second index that is the index of the keyword that appears in the text of the document. .
[0090]
First, the index search means 16 searches for a word (first index) registered in the first index storage unit 23 among the keywords determined to be n words in step T2. If registered, the keyword m words (1 ≦ m ≦ n−1), the appearance frequency and the document in which the keyword appears are stored as search result candidates (step U3).
[0091]
Next, the index search means 16 searches for the word (second index) registered in the second index storage unit 27 among the remaining nm words of each document that is a search result candidate. If registered, the registered document and the appearance frequency of the keyword are additionally registered in the search result list (step U4). Thereafter, the index search means 16 sorts the search result list by the appearance frequency of the keywords, and displays the search results to the user using the output means 4 (step T5).
[0092]
The present invention is not limited to this embodiment, and the following various modifications are possible. That is, in the fourth embodiment, the search result list is sorted by the appearance frequency of keywords, but the appearance frequency in the first index and the appearance frequency in the second index are multiplied by different weights, and the sum is obtained. And sort by the result. Further, in the present embodiment, the first index storage unit 23 registers the off-site keyword and the on-site keyword extracted from the hypertext, but appears in the meta information representing the content of the document. It may be a keyword. For example, when the search target is an academic paper, an introductory sentence in the cited paper corresponds to this meta information. In addition, when the search target is a book, bibliographic items and introductory articles of the book correspond to this meta information.
[0093]
In this embodiment, the search result list is sorted by the appearance frequency of the keyword. However, the appearance frequency in the first index and the appearance frequency in the second index are multiplied by different weights, and the result is obtained. You may make it sort by. In the present embodiment, the method of storing the document as a search result candidate when one or more keywords appear in step U3 for searching the first index has been described. There is also a method of executing the step U4 of searching the second index using all documents as search result candidates when the keyword is not hit.
[0094]
If either the first index or the second index is hit, it is included in the search result list, and in step T5 for outputting the search result, either the first index or the second index is hit. Depending on the method, there is a method of sorting by changing the weight of the document score, and the method is not limited to the method described in the present embodiment.
[0095]
Next, the effect of this embodiment will be described. In the present embodiment, a first index is created from a keyword included in meta information representing the content of a document, and this is preferentially searched. As a result, a search reflecting the contents of the document can be performed.
[0096]
In this embodiment, when a keyword is divided by a space, connected by “no”, or inputted as a compound word in a search condition, the keyword is divided, and the first index and the second index are divided. Each index is searched. As a result, an effective narrowing search that reflects the contents of the document can be performed.
[0097]
  [Fifth Embodiment]
  Next, a fifth embodiment of the present invention will be described in detail with reference to the drawings. FIG. 13 shows a block diagram of a fifth embodiment of the present invention. As shown in FIG.SettingThe fifth embodiment of the apparatus includes an input device 31, a data processing device 32, an output device 33, and a storage device 34. Further, the keyword of the first embodiment described above.SettingA recording medium 30 on which a program for realizing the apparatus is recorded is provided. The recording medium 30 may be a magnetic disk, a semiconductor memory, a CD-ROM, or other recording medium.
[0098]
  The input device 31 is a device for inputting instructions from an operator, such as a mouse and a keyboard. The output device 33 is a device that outputs the processing result of the data processing device 32, and is a display device, a printer, or the like, for example. keywordSettingA program for realizing the device is read from the recording medium 30 into the data processing device 32, controls the operation of the data processing device 32, and generates an input memory 35 and a work memory 36 in the storage device 34. The data processor 32 is a keywordSettingThe same processing as that of the first embodiment is executed under the control of a program for realizing the apparatus.
[0099]
The data processing device 1 in FIG. 1 corresponds to the data processing device 32 in FIG. 13, and the storage device 2 in FIG. 1 corresponds to the storage device 34 in FIG. However, the hypertext database 21 to be processed may be obtained by accessing the external database via the network (for example, the Internet) by the data processing device 32 in addition to the form read from the recording medium 30. Good.
[0100]
[Sixth Embodiment]
Next, a sixth embodiment of the present invention will be described in detail with reference to the drawings. The sixth embodiment uses the configuration shown in FIG. 13 as in the fifth embodiment. A program for realizing the document search device is read from the recording medium 30 into the data processing device 32 and controls the operation of the data processing device 32. The data processing device 32 executes the same processing as in the second embodiment by controlling a program for realizing the document search device.
[0101]
The data processing device 5 in FIG. 7 corresponds to the data processing device 32 in FIG. 13, and the storage device 6 in FIG. 7 corresponds to the storage device 34 in FIG. However, the hypertext database 21 to be processed may be obtained by accessing the external database via the network (for example, the Internet) by the data processing device 32 in addition to the form read from the recording medium 30. Good.
[0102]
Here, all of the hypertext access unit 11, the document cluster information acquisition unit 12, the target designation unit 13, the document keyword determination unit 14, the index creation unit 15, and the index search unit 16 in FIG. Although described in the form of being read by the data processing device 32, it may be divided and recorded on a plurality of recording media. For example, the programs of the hypertext access unit 11, the document cluster information acquisition unit 12, the target specification unit 13, and the document keyword determination unit 14 are read from the recording medium of the fifth embodiment, and the other index creation unit 15 And the program of the index search means 16 may be a separate recording medium. Furthermore, the program of the index creation means 15 and the program of the index search means 16 may be divided into separate recording media.
[0103]
[Seventh Embodiment]
Next, a seventh embodiment of the present invention will be described in detail with reference to the drawings. The seventh embodiment uses the configuration of FIG. 13 as in the fifth and sixth embodiments. A program for realizing the document classification device is read from the recording medium 30 into the data processing device 32 and controls the operation of the data processing device 32. The data processing device 32 executes the same processing as that of the third embodiment under the control of a program for realizing the document classification device.
[0104]
9 corresponds to the data processing device 32 in FIG. 13, and the storage device 8 in FIG. 9 corresponds to the storage device 34 in FIG. However, the hypertext database 21 to be processed may be obtained by accessing the external database via the network (for example, the Internet) by the data processing device 32 in addition to the form read from the recording medium 30. Good.
[0105]
Here, the hypertext access unit 11, the document cluster information acquisition unit 12, the target designation unit 13, the document keyword determination unit 14, the document vector creation unit 17, and the similarity calculation unit 18 in FIG. 9 are all in one recording medium. Although described in the form of being read from 30 to the data processor 32, it may be divided and recorded on a plurality of recording media.
[0106]
For example, the hypertext access means 11, document cluster information acquisition means 12, object designation means 13, and document keyword determination means 14 are read from the recording medium of the fifth embodiment, and other document vector creation means 17 and the program of the similarity calculation means 18 may be configured as separate recording media. Furthermore, the program of the document vector creation means 17 and the program of the similarity calculation means 18 may be configured separately on different recording media.
[0107]
[Eighth Embodiment]
Next, an eighth embodiment of the present invention will be described in detail with reference to the drawings. The eighth embodiment uses the configuration of FIG. 13 as in the fifth, sixth, and seventh embodiments. A program for realizing the document search device is read from the recording medium 30 into the data processing device 32 and controls the operation of the data processing device 32. The data processing device 32 executes the same processing as that of the fourth embodiment under the control of a program for realizing the document search device. 11 corresponds to the data processing device 32 in FIG. 13, and the recording device 10 in FIG. 11 corresponds to the recording device 34 in FIG.
[0108]
【Example】
[First embodiment]
Next, a first embodiment of the present invention will be described with reference to the drawings. This first example corresponds to the first embodiment of the present invention. In this embodiment, a personal computer is provided as the data processing device 1 shown in FIG. 1 and a magnetic disk storage device is provided as the storage device 2.
[0109]
The personal computer has a central processing unit that functions as the hypertext access unit 11, the document cluster information acquisition unit 12, the target designation unit 13, and the document keyword determination unit 14 shown in FIG. The magnetic disk storage device stores the hypertext database 21 and the document keyword storage unit 22 shown in FIG. An example of a hypertext group stored in the hypertext database 21 is shown in FIG.
[0110]
First, the hypertext access unit 11 reads each document stored in the hypertext database 21 and passes it to the document cluster information acquisition unit 12. The document cluster information acquisition unit 12 extracts link information from a given document, and generates a document reference relation table in which an anchor character string, a link source document, and a link destination document are associated with each other as shown in FIG. .
[0111]
Next, the document cluster information acquisition unit 12 uses the document cluster as shown in FIG. 16 based on the “site top page condition” and the “document condition included in the same site” specified by the target specifying unit 13. A document cluster table in which the top page and the in-cluster document are associated with each other is generated. In this embodiment, “the condition of the top page of the site” is “the URL of the document is“ http: // domain name / ”or“ http: // domain name / index.html ””, “The domain name is the same” as the “condition of documents included in the same site”.
[0112]
Next, the document keyword determination unit 14 refers to the document reference relation table and the document cluster table generated by the document cluster information acquisition unit 12, and anchors the links placed from outside the site to the top page of each site. A character string is used as an off-site keyword, and a series of anchor character strings obtained by tracing back links of documents in the same cluster is stored in the document keyword storage unit 22 as an on-site keyword for each intra-classic document. An example of the obtained document keyword is shown in FIG.
[0113]
In this embodiment, the off-site keyword of documents included in the same site is the same as the off-site keyword of the top page of the site. Accordingly, the off-site keywords of the document 112 to the document 119 in FIG. 14 are the same “gourmet information, restaurant search” as the off-site keyword of the document 111.
[0114]
Also, when going back to the link, remember the document that went back once, so that it doesn't go back in a loop. For example, when the link to the document 116 in FIG. 14 is simply traced back, in addition to the anchor character string series “Tokyo → Kanto”, “Tokyo → Kanto ← Back ← Tokyo → Kanto”, “Tokyo → Kanto ← An infinite number of anchor strings are generated by a loop like "Back ← Tokyo ← Kanto ← Back ← Tokyo ...". Therefore, a document that has been traced once is not traced twice within the same anchor character string series. Therefore, the keyword in the site of the document 116 does not include “return”.
[0115]
Further, in this embodiment, only one link from outside the site to a page that is not the top page is traced back and included in the in-site keyword. Accordingly, there are two types of keywords in the site of the document 116: “Tokyo ← Kanto” and “Recommended shops in Tokyo”. The document 119 shown in FIG. 14 is registered in the same manner. However, when the same document is traced back in another anchor character string series, it is registered as different keywords.
[0116]
That is, “Chinese ← Tokyo ← Kanto” and “Chinese ← Osaka Prefecture ← Kansai” are both anchor character strings that go back to the document 111, but both are stored as site keywords because they are separate series. . In this case as well, an anchor character string such as “Chinese ← Tokyo ← Kanto ← Back ← Tokyo ← Kanto” can be considered, but this is because the document 113 and the document 116 are traced back twice in the same series. It is not stored as an internal keyword. In addition, since only one link from outside the site to a page that is not the top page is included in the site keyword, “Chinese ← Tokyo recommended store” is also stored as the site keyword of document 119. .
[0117]
[Second Embodiment]
Next, a second embodiment of the present invention will be described with reference to the drawings. This second example corresponds to the second embodiment of the present invention. In this embodiment, a personal computer is provided as the data processing device 5 in the second embodiment shown in FIG. 7, and a magnetic disk storage device is provided as the storage device 6.
[0118]
The above central processing unit of the personal computer has the same function as that of the first embodiment, but in addition to this, it also functions as the index creating means 15 and index searching means 16 shown in FIG. Different from the embodiment. The second embodiment is different from the first embodiment in that a keyboard is provided as an input device and a display is provided as an output device. The magnetic disk storage device is different from the first embodiment in that the first index storage unit 23 shown in FIG. 7 is also stored. An example of a hypertext group stored in the hypertext database 21 of this embodiment is shown in FIG.
[0119]
In this embodiment, there are two types of processing with different timings of registration and search. The search is performed every time there is an input from the user, whereas the registration need only be performed once in advance. In the registration process, first, the hypertext access means 11 reads each document stored in the hypertext database 21 and passes it to the document cluster information acquisition means 12. The document cluster information acquisition unit 12 extracts link information from a given document, and generates a document reference relation table as shown in FIG.
[0120]
Next, the document cluster information acquisition unit 12 generates a document cluster table as shown in FIG. 16 based on the “site top page condition” and the “document condition included in the same site” designated by the target designating unit 13. Is generated. In this embodiment, “the condition of the top page of the site” is “the URL of the document is“ http: // domain name / ”or“ http: // domain name / index.html ””, “Conditions for documents included in the same site”.
[0121]
Next, the document keyword determination unit 14 refers to the document reference relation table and the document cluster table generated by the document cluster information acquisition unit 12, and anchors the links placed from outside the site to the top page of each site. A character string is used as an off-site keyword, and for each intra-cluster document, a series of anchor character strings obtained by tracing back the link of the same intra-cluster document is stored in the document keyword storage unit 22 as an intra-site keyword. An example of the obtained document keyword is shown in FIG.
[0122]
In this embodiment, the off-site keyword of documents included in the same site is the same as the off-site keyword of the top page of the site. Accordingly, the off-site keywords of the document 112 to the document 119 in FIG. 14 are the same “gourmet information, restaurant search” as the off-site keyword of the document 111.
[0123]
Also, when going back to the link, remember the document that went back once, so that it doesn't go back in a loop. For example, when the link to the document 116 in FIG. 14 is simply traced back, in addition to the anchor character string series “Tokyo → Kanto”, “Tokyo → Kanto ← Back ← Tokyo → Kanto”, “Tokyo → Kanto ← An infinite number of anchor strings are generated by a loop like "Back ← Tokyo ...". Therefore, a document that has been traced once is not traced twice within the same anchor character string series. Therefore, the keyword in the site of the document 116 does not include “return”.
[0124]
Further, in this embodiment, only one link from outside the site to a page that is not the top page is traced back and included in the in-site keyword. Accordingly, there are two types of keywords in the site of the document 116: “Tokyo ← Kanto” and “Recommended shops in Tokyo”. The document 119 in FIG. 14 is also registered in the same manner, but when the same document is traced back with another anchor character string series, it is registered as a different keyword.
[0125]
In other words, “Chinese ← Tokyo → Kanto” and “Chinese ← Osaka Prefecture ← Kansai” are both anchor character string sequences that go back to the document 111, but both are stored as in-site keywords. In this case as well, an anchor character string such as “Chinese ← Tokyo ← Kanto ← Back ← Tokyo ← Kanto” can be considered, but this is because the document 113 and the document 116 are traced back twice in the same series. It is not stored as an internal keyword. In addition, since only one link from outside the site to a page that is not the top page is included in the site keyword, “Chinese ← Tokyo recommended store” is also stored as the site keyword of document 119. .
[0126]
Next, the index creating means 15 creates an index indicating which words are registered in which document for the off-site keyword, and subsequently, an index indicating which word is registered in which document for the in-site keyword. Create
[0127]
Next, the search process will be described in detail. Assume that a search condition “Nara gourmet” is entered from the keyboard. Then, the index search means 16 divides the search condition into two keywords “Nara” and “Gourmet” by separating the search condition with a space or “no” or by performing morphological analysis.
[0128]
Next, the index search means 16 checks whether there is a word that appears in the keyword outside the site among “Nara” and “Gourmet”. If it appears, the keyword, its appearance frequency, and the document in which the keyword appears are stored as search result candidates. When the document keyword is shown in FIG. 17, “gourmet” appears once for each of the off-site keywords of the documents 111 to 119, so that it is set as a search result candidate.
[0129]
Next, the index search means 16 additionally registers, in the search result list, a document in which the remaining “Nara” appears as a keyword in the site among the documents that are search result candidates, and the appearance frequency of the keyword. When the document keyword is shown in FIG. 17, among the documents 111 to 119 that are search result candidates, the documents 114, 117, and 118 have “Nara” as the keyword in the site. The appearance frequency of the keyword “Nara” is once. Finally, the index search means 16 sorts the search result list by the appearance frequency of the keywords, and displays the search results to the user using the display.
[0130]
In the present embodiment, the appearance frequency of the keyword in the document 114, the document 117, and the document 118 is all once, but the appearance position (the keyword outside the site, the keyword within the site, or the keyword within the site is the first The search results may be sorted using a score that weights the appearance frequency according to whether it is the body or the body).
[0131]
It is also assumed that a search condition “Nara gourmet search” is input from the keyboard. Then, the index search means 16 divides the search condition into “Nara”, “Gourmet”, and “Search” by dividing the search condition by a space or “NO” or by performing morphological analysis.
[0132]
Next, the index search means 16 checks whether there is a word that appears in the keyword outside the site among “Nara”, “Gourmet”, and “Search”. If it appears, the keyword, its appearance frequency, and the document in which the keyword appears are stored as search result candidates. When the document keyword is shown in FIG. 17, “gourmet” and “search” appear once in the off-site keywords of the documents 111 to 119, respectively, and therefore are set as search result candidates.
[0133]
Next, the index search means 16 additionally registers, in the search result list, a document in which the remaining “Nara” appears as a keyword in the site among the documents that are search result candidates, and the appearance frequency of the keyword. When the document keyword is shown in FIG. 17, among the documents 111 to 119 that are search result candidates, “Nara” appears as the keyword in the site in the document 114, the document 117, and the document 118. The appearance frequency of the keyword “Nara” is once. Finally, the index search means 16 sorts the search result list by the appearance frequency of the keywords, and displays the search results to the user using the display.
[0134]
In this embodiment, the appearance frequency of the keywords in the document 114, the document 117, and the document 118 is all once, but the appearance position (the keyword outside the site, the keyword within the site, or the keyword within the site is the first one) Search results may be sorted using a score that weights the appearance frequency according to the text).
[0135]
Also assume that a search condition of “Nara Chinese” is entered from the keyboard. Next, the index search means 16 divides the search condition into “Nara” and “Chinese” by dividing the search condition with a space or “no” or performing morphological analysis.
[0136]
Next, the index search means 16 checks whether there is a word that appears in the off-site keyword among “Nara” and “Chinese”. If it appears, the keyword, its appearance frequency, and the document in which the keyword appears are stored as search result candidates. When the document keyword is shown in FIG. 17, neither “Nara” nor “Chinese” appears as off-site keywords.
[0137]
Next, the index search means 16 additionally registers in the search result list the documents in which “Nara” and “Chinese” appear in the site keywords among all the documents, and the appearance frequency of the keywords. When the document keyword is shown in FIG. 17, since “Nara” and “Chinese” appear once in the document 117, the document 117 is registered in the search result list. Finally, the index search means 16 sorts the search result list by the appearance frequency of the keywords, and displays the search results to the user using the display.
[0138]
Further, it is assumed that a search condition “Chinese restaurant” is input from the keyboard. Then, the index search means divides the search condition into “Chinese” and “Restaurant” by dividing the search condition with a space or “no” or performing morphological analysis.
[0139]
Next, the index search means 16 checks whether there is a word that appears in the keyword outside the site among “Chinese” and “Restaurant”. If it appears, the keyword, its appearance frequency, and the document in which the keyword appears are stored as search result candidates. In the case where the document keyword is shown in FIG. 17, “restaurant” appears once for each of the off-site keywords of the document 111 to the document 119, and is thus regarded as a search result candidate.
[0140]
Next, the index search means 16 additionally registers, in the search result list, the remaining “Chinese” in the site keyword among the documents that are search result candidates and the appearance frequency of the keyword. When the document keyword is FIG. 17, among the documents 111 to 119 that are search result candidates, “Chinese” appears in the site keyword in the documents 117 and 119, and the appearance of the keyword “Chinese” appears. The frequency is once and three times, respectively. Finally, the index search means sorts the search result list according to the appearance frequency of the keywords, and displays the search results to the user using the display.
[0141]
In this embodiment, the keyword in the site of document 119 is “Chinese ← Osaka Prefecture ← Kansai”, “Chinese ← Tokyo ← Kanto”, and “Chinese ← Tokyo recommended shops”. However, since all “Chinese Chinese” are derived from the same link, the appearance frequency may be counted as one. Alternatively, the keyword in the site of the document 119 may be stored as “Chinese ← Osaka Prefecture, Tokyo, Tokyo recommended store ← Kansai, Kanto”, and the appearance frequency of “Chinese” may be counted as one time.
[0142]
[Third embodiment]
Next, a third embodiment of the present invention will be described with reference to the drawings. This third example corresponds to the third embodiment of the present invention. As in the first example, this example has a configuration in which the data processing device 9 of the third embodiment shown in FIG. 9 is a personal computer and the storage device 8 is a magnetic disk storage device. The computer is different from the first embodiment in that the central processing unit of the computer also functions as the document vector creation means 17 and the similarity calculation means 18 shown in FIG. Further, the magnetic disk storage device is different from the first embodiment in that the document vector storage unit 24, the category condition storage unit 25, and the classification result storage unit 26 shown in FIG. 9 are also stored.
[0143]
Next, the operation of the present embodiment will be described. First, the hypertext access means (11 in FIG. 9) reads each document stored in the hypertext database (21 in FIG. 9) and passes it to the document cluster information acquisition means (12 in FIG. 9). An example of a hypertext group stored in the hypertext database 21 is shown in FIG. The document cluster information acquisition unit 12 extracts link information from a given document, and generates a document reference relation table as shown in FIG.
[0144]
Next, the document cluster information acquisition means 12 is shown in FIG. 16 based on the “site top page condition” and the “document condition included in the same site” designated by the target designation means (13 in FIG. 9). A document cluster table like this is generated. In this embodiment, “the condition of the top page of the site” is “the URL of the document is“ http: // domain name / ”or“ http: // domain name / index.html ””, “The domain name is the same” as the “condition of documents included in the same site”.
[0145]
Next, the document keyword determination means (14 in FIG. 9) refers to the document reference relation table and the document cluster table generated by the document cluster information acquisition means 12 and is applied to the top page of each site from outside the site. 9 is used as a keyword outside the site, and for each intra-cluster document, a series of anchor character strings obtained by tracing back the links of the documents in the same cluster as the intra-site keyword. 22). An example of the obtained document keyword is shown in FIG.
[0146]
In this embodiment, the off-site keyword of documents included in the same site is the same as the off-site keyword of the top page of the site. Accordingly, the off-site keywords of the document 112 to the document 119 in FIG. 14 are the same “gourmet information, restaurant search” as the off-site keyword of the document 111.
[0147]
Also, when going back to the link, remember the document that went back once, so that it doesn't go back in a loop. For example, when the link to the document 116 in FIG. 14 is simply traced back, in addition to the anchor character string series “Tokyo → Kanto”, “Tokyo → Kanto ← Back ← Tokyo → Kanto”, “Tokyo → Kanto ← An infinite number of anchor strings are generated by a loop like "Back ← Tokyo ← Kanto ← Back ← Tokyo ...". Therefore, a document that has been traced once is not traced twice within the same anchor character string series. Therefore, the keyword in the site of the document 116 does not include “return”.
[0148]
Further, in this embodiment, only one link from outside the site to a page that is not the top page is traced back and included in the in-site keyword. Therefore, there are two types of keywords in the site of the document 116, “Tokyo ← Kanto” and “Tokyo recommended shops”. The document 119 in FIG. 14 is registered in the same manner, but when the same document is traced back in another anchor character string series, it is registered as a different keyword.
[0149]
In other words, “Chinese ← Tokyo → Kanto” and “Chinese ← Osaka Prefecture ← Kansai” are both anchor character string sequences that go back to the document 111, but both are stored as in-site keywords. In this case as well, an anchor character string such as “Chinese ← Tokyo ← Kanto ← Back ← Tokyo ← Kanto” can be considered, but this is because the document 113 and the document 116 are traced back twice in the same series. It is not stored as an internal keyword. In addition, since only one link from outside the site to a page that is not the top page is included in the site keyword, “Chinese ← Tokyo recommended store” is also stored as the site keyword of document 119. .
[0150]
Next, the document vector creating means 17 stores the document vector as a document vector indicating how many times each keyword appears in which part (external keyword, in-site keyword, title, text, etc.). Store in the unit 24.
[0151]
Then, the similarity calculation means 18 calculates the cosine of the feature vector of each category stored in the category condition storage unit 25 for the document vector of each document stored in the document vector storage unit 24, and the result is Stored in the classification storage unit 26.
[0152]
[Fourth embodiment]
Next, a fourth embodiment of the present invention will be described with reference to the drawings. This fourth example corresponds to the fourth embodiment of the present invention. This embodiment is the same as the second embodiment in that the data processing device 9 of the fourth embodiment shown in FIG. 11 is constituted by a personal computer and the storage device 10 is constituted by a magnetic disk storage device. The central processing unit of the personal computer is different from the second embodiment in that it functions only as index search means. Further, the magnetic disk storage device is different from the second embodiment in that the second index storage unit 27 shown in FIG. 11 is stored instead of storing the hypertext database and the document keyword storage unit.
[0153]
FIG. 18 shows an example of an index created from document meta information stored in the first index storage unit 23 shown in FIG. In FIG. 18, keywords, appearing documents, and appearance frequencies are recorded. For example, documents 211, 212, and 214 are registered in the keyword “hotel”, and the keyword “hotel” appears. It can be seen that the frequency is 3 times, 1 time, and 5 times.
[0154]
An example of an index created from the text of the document stored in the second index storage unit 27 is shown in FIG. The format of the second index is the same as that of the first index, and keywords, appearing documents, and appearance frequencies are recorded. For example, documents registered with the keyword “Tokyo” are document 212, document 213, document In 214, document 217, document 218, and document 219, it can be seen that the appearance frequency of the keyword “Tokyo” is once, four times, six times, eight times, once, and twice.
[0155]
Assume that the search condition “Nara gourmet” is entered from the keyboard. Then, the index search means 16 divides the search condition into a keyword of “Nara” and “Gourmet” by dividing the search condition by a space or “no” or by performing morphological analysis.
[0156]
Next, the index search means 16 checks whether there is a word registered in the first index storage unit 23 among the keywords “Nara” and “Gourmet”. If registered, the keyword, its appearance frequency, and the document in which the keyword appears are stored as search result candidates. When the first index is FIG. 18, since “gourmet” is registered, the document 211, the document 212, the document 213, and the document 214 are search result candidates.
[0157]
Next, the index search unit 16 stores the remaining “Nara” documents registered in the second index storage unit 27 and the keyword appearance frequency in the search result list among the search result candidate documents. Register additional. When the second index is FIG. 19, among the search result candidate documents 211, 212, 213, and 214, “Nara” appears only in the document 213, so the document 213 is registered in the search result list. The
[0158]
Next, the index search means 16 sorts the search result list by the appearance frequency of the keywords, and displays the search results on the display. In this case, the document 213 registered in the search result list is output as the search result.
[0159]
Also assume that the search condition “Osaka Library” is entered from the keyboard. Then, the index search means 16 divides the search keyword into a keyword of “Osaka” and “library” by separating the search keyword with a space or “no” or by performing a morphological analysis.
[0160]
Next, the index search means 16 checks whether there is a word registered in the first index storage unit 23 among the keywords “Osaka” and “Library”. If registered, the keyword, its appearance frequency, and the document in which the keyword appears are stored as search result candidates. When the first index is shown in FIG. 18, the document 215, the document 216, the document 217, the document 218, and the document 219 in which “library” is registered are search result candidates. The appearance frequency of the keyword “library” is once, five times, twice, seven times, and four times, respectively.
[0161]
Next, the index search unit 16 stores the remaining “Osaka” in the second index storage unit 27 among the documents as search result candidates and the appearance frequency of the keyword in the search result list. Register additional. When the second index is FIG. 19, the keyword “Osaka” appears in the document 216, the document 217, and the document 219 among the search result candidate documents 215, 216, 217, 218, and 219. The appearance frequency of “Osaka” is 2 times, 4 times and 8 times, respectively.
[0162]
Next, the index search means 16 sorts the search result list by the appearance frequency of the keywords, and displays the search results on the display. The total frequency of occurrence of the keywords “library” and “Osaka” is 7 times for the document 216, 6 times for the document 217, and 12 times for the document 219, and therefore the document 219, document 216, and document 217 are displayed in the order. Is displayed.
[0163]
In this embodiment, the keywords are simply sorted according to the total appearance frequency of the keywords. However, the keyword appearance frequencies in the first keyword index and the keyword appearance frequencies in the second keyword index are multiplied by different weights, respectively. You may sort on the basis of the score obtained.
[0164]
【The invention's effect】
As described above, according to the present invention, the following various effects can be achieved.
(1) It is possible to perform search / classification on the hypertext group reflecting the contents of the entire site and the position of the document in the site. The reason for this is that the link between the document cluster and the document is a series of anchor strings obtained by tracing back the link from outside the site that represents the contents of the entire site and the link within the site that represents the position of the document within the site. This is because it is extracted as a word group that characterizes and is used as a search / classification target.
[0165]
(2) An effective refinement search for a plurality of keywords can be performed. The reason for this is that when a keyword is divided by a space, connected by “no”, or entered as a compound word in a search condition, the keyword is divided into a keyword as meta information representing the content of the document. This is because the keyword of the document body is searched separately.
[0166]
(3) A search that reflects the contents and meaning of a document can be performed for a general document. The reason is that a keyword in meta information representing the content of a document is distinguished from a keyword in the document, and a search is performed as separate indexes.
[Brief description of the drawings]
FIG. 1 is a block diagram showing a configuration of a first exemplary embodiment of the present invention.
FIG. 2 is a diagram showing an example of a hypertext group stored in a hypertext database according to the first embodiment of the present invention.
FIG. 3 is a diagram showing an example of a document keyword stored in a document keyword storage unit according to the first embodiment of the present invention.
FIG. 4 is a diagram illustrating an example of a document reference relationship table generated by a document cluster information acquisition unit according to the first embodiment of this invention.
FIG. 5 is a diagram showing an example of a document cluster table generated by a document cluster information acquisition unit according to the first embodiment of the present invention.
FIG. 6 is a flowchart showing the operation of the first exemplary embodiment of the present invention.
FIG. 7 is a block diagram showing a configuration of a second exemplary embodiment of the present invention.
FIG. 8 is a flowchart showing the operation of the second exemplary embodiment of the present invention.
FIG. 9 is a block diagram showing a configuration of a third exemplary embodiment of the present invention.
FIG. 10 is a flowchart showing the operation of the third exemplary embodiment of the present invention.
FIG. 11 is a block diagram showing a configuration of a fourth exemplary embodiment of the present invention.
FIG. 12 is a flowchart showing the operation of the fourth exemplary embodiment of the present invention.
FIG. 13 is a block diagram showing a configuration of fifth to eighth embodiments of the present invention.
FIG. 14 is a diagram showing an example of a hypertext group stored in a hypertext database according to the first embodiment of the present invention.
FIG. 15 is a diagram illustrating an example of a document reference relationship table generated by a document cluster information acquisition unit according to the first embodiment of this invention.
FIG. 16 is a diagram illustrating an example of a document cluster table generated by a document cluster information acquisition unit according to the first embodiment of this invention.
FIG. 17 is a diagram illustrating an example of a document keyword stored in a document keyword storage unit according to the first embodiment of this invention.
FIG. 18 is a diagram illustrating an example of an index stored in a first index storage unit according to the fourth exemplary embodiment of the present invention.
FIG. 19 is a diagram illustrating an example of an index stored in a second index storage unit according to the fourth embodiment of the present invention.
[Explanation of symbols]
1, 5, 7 Data processing device
2, 6, 8 storage device
3 Input means
4 Output means
11 Hypertext access means
12 Document cluster information acquisition means
13 Target designation means
14 Document keyword determination means
15 Index creation means
16 Index search means
17 Document vector creation means
18 Similarity calculation means
21 Hypertext database
22 Document keyword storage
23 First index storage unit
24 Document vector storage
25 Category condition storage
26 Classification result storage unit
27 Second index storage unit
30 storage media
31 Input device
32 Data processing equipment
33 Output device
34 Storage device
35 Input memory
36 Work memory

Claims (28)

Article for each body of a plurality of hierarchical documents with each document cluster that is formed by the structured plurality of documents in the hyperlink, the link information and indicates a link destination document and link source document of the document Storage means for storing at least an anchor character string that is a character string in a document from which the document is linked ;
A target designating unit for designating a condition of a document cluster including a condition of a document included in the same document cluster as a condition of a top page which is a top-level document in the document cluster determined from a positional relationship in a directory hierarchy;
The document body of said plurality of documents from the storage means, reads the link information and the anchor character string, the based on the condition of the specified the document clusters by link information and the object specifying means, a given document is the either a top page, a document cluster specific means for performing certain which document of document clusters is a document belonging to a cluster based on the condition of the documents included in the same document clusters and if not the top page,
Based on the link information of the document cluster specified by the document cluster specifying means , for each of a plurality of documents in the same document cluster, when each document is a linked document, the upper layer up to the highest level document in the document cluster the sequence of the anchor character string and determines the site keywords of the document is a character string of each link source document in the resultant is back links in the link source document direction is, link to the top-level document of the document clusters keyword determination apparatus characterized by having a determining means for determining anchors string is a character string in a document in another document clusters that as a site outside keywords for each document in the document clusters. The document clusters specifying means, the anchor text, links and document reference relationship table based on the document and the linked document is a list that is associated, said document clusters, top-level document and its top in its document clusters keyword determining apparatus according to claim 1, wherein the creating the document cluster table is a list of subordinate document clusters Uchibun manual upper document associated specifying the document clusters. The document cluster specifying means includes a range of the document cluster, a domain name of a server in which the document is stored, a number of hyperlinks to the document, an anchor character string, or a directory hierarchy in which the document is stored, or keyword determining apparatus according to claim 1, wherein the determining based on predesignated document list by the user. The determining means, when each of the plurality of documents of the same document cluster is a link destination document, when the link is traced back to the link source document direction that is a higher layer up to the highest document in the document cluster a keyword determining apparatus according to claim 1, wherein the storing once predated link source document so as not go back twice once predated link source document in the series of the same anchor character string. In addition to the anchor character string, the determination means includes a title of the document, a character string of a certain length around the anchor character string, a character string surrounded by tags around the anchor character string, or a font size and color. highlighted string, keyword determining apparatus according to claim 1, wherein the determining as the site keyword or the offsite keyword. An anchor that is a document text, link information, and a character string in a link source document of each of a plurality of hierarchized documents included in each of document clusters formed by a plurality of documents structured by hyperlinks Storage means storing at least a character string;
A target designating unit for designating a condition of a document cluster including a condition of a document included in the same document cluster as a condition of a top page which is a top-level document in the document cluster determined from a positional relationship in a directory hierarchy;
The document body of said plurality of documents from the storage means, reads the link information and the anchor character string, the based on the condition of the specified the document clusters by link information and the object specifying means, a given document is the either a top page, a document cluster specific means for performing certain which document of document clusters is a document belonging to a cluster based on the condition of the documents included in the same document clusters and if not the top page,
Based on the link information of the document cluster specified by the document cluster specifying means , when each document is a linked document for each of a plurality of documents in the same document cluster, the link source up to the highest level document in the document cluster determining the sequence of the anchor character string is a character string of each link source document in the resulting back links in the document direction as site keyword of the document, another linked with respect to the top-level document of the document clusters determining means for determining the anchors string as offsite keywords for each document in the document clusters is a character string in a document in the document cluster,
Index creation and storage means for creating and storing as an index an index which keyword appears in which document among the in- site keyword and the off-site keyword of each document determined by the determination means; ,
An input means for entering search conditions;
A keyword to be used for a search is determined from the search condition input by the input unit, and the document in which the keyword to be used for the search appears is searched based on the index stored by the index creation and storage unit. And a search means for searching as a candidate . The document clusters specifying means, the anchor text, links and document reference relationship table based on the document and the linked document is a list that is associated, said document clusters, top-level document and its top in its document clusters document search apparatus according to claim 6, characterized in that to create the document cluster table is a list of subordinate document clusters Uchibun manual upper document associated specifying the document clusters. The document cluster specifying means includes a range of the document cluster, a domain name of a server storing the document, a number of hyperlinks to the document, an anchor character string, or a directory hierarchy in which the document is stored, The document search apparatus according to claim 6 , wherein the document search apparatus is determined based on a document list designated in advance by a user. Article for each body of a plurality of hierarchical documents with each document cluster that is formed by the structured plurality of documents in the hyperlink, the link information and indicates a link destination document and link source document of the document Storage means for storing at least an anchor character string that is a character string in a document from which the document is linked ;
A target designating unit for designating a condition of a document cluster including a condition of a document included in the same document cluster as a condition of a top page which is a top-level document in the document cluster determined from a positional relationship in a directory hierarchy;
The document body of said plurality of documents from the storage means, reads the link information and the anchor character string, the based on the condition of the specified the document clusters by link information and the object specifying means, a given document is the either a top page, a document cluster specific means for performing certain which document of document clusters is a document belonging to a cluster based on the condition of the documents included in the same document clusters and if not the top page,
Based on the link information of the document cluster specified by the document cluster specifying means , when each document is a linked document for each of a plurality of documents in the same document cluster, the link source up to the highest level document in the document cluster determining the sequence of the anchor character string is a character string of each link source document in the resulting back links in the document direction as site keyword of the document, another linked with respect to the top-level document of the document clusters determining means for determining the anchors string as offsite keywords for each document in the document clusters is a character string in a document in the document cluster,
Document vector creation means for creating a document vector that expresses, as a multidimensional vector, the frequency of occurrence or occurrence of occurrence of the in-site keyword and the off-site keyword of each document determined by the determination means. When,
Category condition storage means for storing a characteristic keyword and its appearance frequency as a feature vector for each category to be classified,
Similarity calculation means for calculating similarity between the document vector of each document created by the document vector creation means and the feature vector of each category stored in the category condition storage means;
Classification means for classifying the plurality of documents according to the similarity calculation result;
Document classification apparatus characterized by having a. The document clusters specifying means, the anchor text, links and document reference relationship table based on the document and the linked document is a list that is associated, said document clusters, top-level document and its top in its document clusters document classification apparatus according to claim 9, wherein creating a document clusters table is a list of subordinate document clusters Uchibun manual of level document is associated and identifies the document clusters. The document cluster specifying means includes a range of the document cluster, a domain name of a server in which the document is stored, a number of hyperlinks to the document, an anchor character string, or a directory hierarchy in which the document is stored, The document classification apparatus according to claim 9 , wherein the document classification device is determined based on a document list designated in advance by a user. The determining means, when each of the plurality of documents of the same document cluster is a link destination document, when the link is traced back to the link source document direction that is a higher layer up to the highest document in the document cluster the document classification apparatus according to claim 9, wherein the storing once predated link source document so as not go back twice once predated link source document in the series of the same anchor character string. In addition to the anchor character string, the determination means includes a title of the document, a character string of a certain length around the anchor character string, a character string surrounded by tags around the anchor character string, or a font size and color. 10. The document classification apparatus according to claim 9 , wherein the emphasized character string is determined as the in-site keyword or the off-site keyword. A data processing apparatus determines a document body, a link destination document and a link source document of each of a plurality of hierarchical documents included in each of document clusters formed by a plurality of documents structured by hyperlinks. the document body of the link information and the plurality of document anchor character string from a storage unit operable to store at least a string of the link of the document based on the document in shown, a reading step of to read out the link information and the anchor character string ,
Target specification for specifying a condition of a document cluster including a condition of a top page that is the top document in the document cluster and a condition of a document included in the same document cluster determined by the data processing apparatus based on a positional relationship in a directory hierarchy Steps,
Wherein the data processing device, based on the condition of the document clusters designated by said object designating step and the link information read by said reading step, whether a given document is the top page, the top page If not , a document cluster specifying step of specifying a document cluster as to which document cluster belongs based on the conditions of the documents included in the same document cluster ;
When each of the data processing devices sets each document as a link destination document for each of a plurality of documents in the same document cluster based on the link information of the document cluster specified in the document cluster specifying step , A series of anchor character strings, which are character strings in each link source document obtained by going back to the link source document direction that is the upper layer up to the highest document, is determined as a keyword in the site of the document, and the top of the document cluster is determined. characterized in that it comprises a determining step of determining the anchors string is a character string in a document in another document clusters linked by relative level document as offsite keywords for each document in the document clusters Keyword decision method. The document clusters specifying step, the anchor text, links and document reference relationship table based on the document and the linked document is a list that is associated, said document clusters, top-level document and its top in its document clusters keyword determination method of claim 14, wherein the creating the document cluster table is a list of subordinate document clusters Uchibun manual upper document associated specifying the document clusters. In the document cluster specifying step, the scope of the document cluster is changed to a domain name of a server in which the document is stored, a number of hyperlinks to the document, an anchor character string, or a directory hierarchy in which the document is stored, 15. The keyword determination method according to claim 14 , wherein the determination is performed based on a document list designated in advance by a user. In the determination step, for each of the plurality of documents in the same document cluster , when each document is a link destination document, the link is traced back to the link source document direction that is a higher layer up to the highest document in the document cluster. the keyword determination method of claim 14, wherein the storing once back links the original document so as not go back twice in the sequence of the same anchor text once predated link source document. In the determination step, in addition to the anchor character string, a title of the document, a character string of a certain length around the anchor character string, a character string surrounded by tags around the anchor character string, or a font size and color The keyword determination method according to claim 14 , wherein the emphasized character string is determined as the in-site keyword or the off-site keyword. A data processing apparatus determines a document body, a link destination document and a link source document of each of a plurality of hierarchical documents included in each of document clusters formed by a plurality of documents structured by hyperlinks. the document body of the link information and the plurality of document anchor character string from a storage unit operable to store at least a string of the link of the document based on the document in shown, a reading step of to read out the link information and the anchor character string ,
Target specification for specifying a condition of a document cluster including a condition of a top page that is the top document in the document cluster and a condition of a document included in the same document cluster determined by the data processing apparatus based on a positional relationship in a directory hierarchy Steps,
Based on the link information read in the reading step and the condition of the document cluster specified in the target specifying step, the given data processing device is the top page or not the top page a document clusters specifying step of performing any specific document or document clusters is a document belonging to a cluster based on the condition of the documents included in the same document clusters when,
When the data processing apparatus sets each document as a link destination document for each of a plurality of documents in the same document cluster based on the link information of the document cluster specified in the document cluster specifying step , A series of anchor character strings, which are character strings in each link source document obtained by tracing back the link in the direction of the link source document, which is the upper layer up to the top document, is determined as a keyword in the site of the document, and the top of the document cluster is determined. a determination step of determining a anchors string is a character string in a document in another document clusters linked by relative level document as offsite keywords for each document in the document clusters,
The data processing device creates and stores as an index an index which keyword appears in which document among the in- site keyword and the off-site keyword of each document determined in the determining step. Indexing and storage steps;
An input step for inputting a search condition by an input means;
The data processing device determines a keyword to be used for a search from the inputted search condition, and the document on which the keyword to be used for the search appears is based on the index stored by the index creation and storage step. A document search method comprising: a search step for searching as a search result candidate . The document clusters specifying step, the anchor text, links and document reference relationship table based on the document and the linked document is a list that is associated, said document clusters, top-level document and its top in its document clusters document retrieval method according to claim 19, wherein the creating the document cluster table is a list of subordinate document clusters Uchibun manual upper document associated specifying the document clusters. A data processing apparatus determines a document body, a link destination document and a link source document of each of a plurality of hierarchical documents included in each of document clusters formed by a plurality of documents structured by hyperlinks. the document body of the link information and the plurality of document anchor character string from a storage unit operable to store at least a string of the link of the document based on the document in shown, a reading step of to read out the link information and the anchor character string ,
Target specification for specifying a condition of a document cluster including a condition of a top page that is the top document in the document cluster and a condition of a document included in the same document cluster determined by the data processing apparatus based on a positional relationship in a directory hierarchy Steps,
Based on the link information read in the reading step and the condition of the document cluster specified in the target specifying step, the given data processing device is the top page or not the top page a document clusters specifying step of performing any specific document or document clusters is a document belonging to a cluster based on the condition of the documents included in the same document clusters when,
When each of the data processing devices sets each document as a link destination document for each of a plurality of documents in the same document cluster based on the link information of the document cluster specified in the document cluster specifying step , A series of anchor character strings, which are character strings in each link source document obtained by going back to the link source document direction that is the upper layer up to the highest document, is determined as a keyword in the site of the document, and a determination step of determining a anchors string is a character string in a document in another document clusters linked by relative level document as offsite keywords for each document in the document clusters,
A document vector in which the data processing device expresses, as a multidimensional vector, information on the frequency or presence / absence of appearance of the in-site keyword and the off-site keyword of each document determined in the determining step. A document vector creation step to be created;
Each document created by the document vector creation step and the feature vector read from the category condition storage means in which the characteristic keyword and its appearance frequency are stored as a feature vector for the category to be classified by the data processing device. A similarity calculation step for calculating a similarity with the document vector of
A classification step in which the data processing device classifies the plurality of documents according to a similarity calculation result in the similarity calculation step;
Document classification method, which comprises a. The document clusters specifying step, the anchor text, links and document reference relationship table based on the document and the linked document is a list that is associated, said document clusters, top-level document and its top in its document clusters document classification method according to claim 21, wherein the creating the document cluster table is a list of subordinate document clusters Uchibun manual upper document associated specifying the document clusters. In the document cluster specifying step, the scope of the document cluster is determined based on the domain name of the server storing the document, the number of hyperlinks to the document, the anchor string, or the directory hierarchy in which the document is stored, The document classification method according to claim 21 , wherein the document classification method is determined based on a document list designated in advance by a user. In the determination step, for each of the plurality of documents in the same document cluster , when each document is a link destination document, the link is traced back to the link source document direction that is a higher layer up to the highest document in the document cluster. to, document classification method according to claim 21, wherein the storing once back links the original document so as not go back twice in the sequence of the same anchor text once predated link source document. In the determination step, in addition to the anchor character string, a title of the document, a character string of a certain length around the anchor character string, a character string surrounded by tags around the anchor character string, or a font size and color The document classification method according to claim 21 , wherein the emphasized character string is determined as the in-site keyword or the off-site keyword. The computer in the data processing device,
Article for each body of a plurality of hierarchical documents with each document cluster that is formed by the structured plurality of documents in the hyperlink, the link information and indicates a link destination document and link source document of the document a reading step of to read out the document body, link information and anchor character string of the plurality of documents from the storage means for storing at least an anchor character string is a string of link source document in the document,
A target designating step for designating a condition of a document cluster including a condition of a document included in the same document cluster as a condition of a top page that is a top-level document in the document cluster determined from a positional relationship in a directory hierarchy;
Based on the condition of the specified the document clusters by the target designation step and the link information read by said reading step, whether a given document is the top page, and if not the top page of the same a document clusters specifying step of performing any specific document is a document belonging to a cluster of document clusters based on the condition of the documents included in the document clusters,
Based on the link information of the document cluster specified by the document cluster specifying step , when each document is a linked document for each of a plurality of documents in the same document cluster, the upper layer up to the highest level document in the document cluster the sequence of the anchor character string and determines the site keywords of the document is a character string of each link source document in the resultant is back links in the link source document direction is, link to the top-level document of the document clusters a determination step of determining a anchors string is a character string in a document in another document clusters you are as off-site keywords for each document in the document clusters
A keyword determination program characterized by causing The computer in the data processing device,
Article for each body of a plurality of hierarchical documents with each document cluster that is formed by the structured plurality of documents in the hyperlink, the link information and indicates a link destination document and link source document of the document a reading step of to read out the document body, link information and anchor character string of the plurality of documents from the storage means for storing at least an anchor character string is a string of link source document in the document,
A target designating step for designating a condition of a document cluster including a condition of a document included in the same document cluster as a condition of a top page that is a top-level document in the document cluster determined from a positional relationship in a directory hierarchy;
Based on the link information read in the reading step and the condition of the document cluster specified in the target specifying step, if the given document is the top page or not the top page, the same document a document clusters specifying step of performing certain of document clusters is a document belonging to any document clusters based on the condition of the documents included in the cluster,
Based on the link information of the document cluster specified in the document cluster specifying step , when each document is a linked document for each of a plurality of documents in the same document cluster, the uppermost layer up to the topmost document in the document cluster the sequence of the anchor character string and determines the site keywords of the document is a character string of each link source document in the resultant is back links in the link source document direction is, link to the top-level document of the document clusters a determination step of determining a anchors string as offsite keywords for each document in the document clusters is a character string in a document in another document clusters you are,
Index creation that creates an index as an index indicating which keyword appears in which document among the in-site keyword and the off-site keyword of each document determined in the determination step, and stores it in an index storage means And a memory step;
An input step for entering search criteria;
A search step of determining a keyword to be used for a search from the input search condition and searching for the document in which the keyword to be used for the search appears as a search result candidate based on the index read from the index storage unit ;
A document search program characterized in that is executed . In the computer in the data processor,
Article for each body of a plurality of hierarchical documents with each document cluster that is formed by the structured plurality of documents in the hyperlink, the link information and indicates a link destination document and link source document of the document a reading step of to read out the document body, link information and anchor character string of the plurality of documents from the storage means for storing at least an anchor character string is a string of link source document in the document,
A target designating step for designating a condition of a document cluster including a condition of a document included in the same document cluster as a condition of a top page that is a top-level document in the document cluster determined from a positional relationship in a directory hierarchy;
Based on the link information read in the reading step and the condition of the document cluster specified in the target specifying step, if the given document is the top page or not the top page, the same document a document clusters specifying step of performing certain of document clusters is a document belonging to any document clusters based on the condition of the documents included in the cluster,
Based on the link information of the document cluster specified by the document cluster specifying step , when each document is a linked document for each of a plurality of documents in the same document cluster, the upper layer up to the highest level document in the document cluster the sequence of the anchor character string and determines the site keywords of the document is a character string of each link source document in the resultant is back links in the link source document direction is, link to the top-level document of the document clusters a determination step of determining a anchors string as offsite keywords for each document in the document clusters is a character string in a document in another document clusters you are,
A document vector creation step of creating a document vector that expresses information on the frequency or occurrence of occurrence of the in-site keyword and the off-site keyword of each document determined in the determination step as a multidimensional vector. When,
For each category to be classified, a characteristic keyword and its appearance frequency are stored as a feature vector between the feature vector read from the category condition storage means and the document vector of each document created by the document vector creation step. A similarity calculation step for calculating the similarity,
A classification step for classifying the plurality of documents according to a similarity calculation result obtained by the similarity calculation step;
Features and to Rubun document classification program that is running.

Images