JP5265610B2 - Related word extractor - Google Patents

Description

  The present invention relates to a technique for retrieving data related to a query.

  In a search technique using a query, various contents existing on the web are collected in advance and stored in a database of a search engine by a technique called crawling. At the time of accumulation, each data is indexed (indexed) and ranked.

  The types of content to be searched cover a wide variety of media such as text (character strings), still images, moving images, and audio, and processing such as data collection is performed by a search engine corresponding to each content.

  The user inputs a query (keyword) to a search engine corresponding to the content to be searched, and obtains a search result of the content related to the query. If a desired search result cannot be obtained, the search is performed again by adding or changing a keyword.

  Related keyword presentation is known as a function that supports user query input. That is, there is known an information processing apparatus that stores a thesaurus dictionary in which words are classified according to the relationship between words and extracts related words for an input query from the thesaurus dictionary (for example, Patent Document 1).

JP 2008-192110 A

  However, a thesaurus dictionary as shown in Patent Document 1 requires artificial maintenance in which words are associated with each other in advance. In addition, since a wide variety of keywords such as new words and abbreviations are input as keywords searched in the web search, it is very complicated to sequentially maintain a dictionary of related terms corresponding to such keywords. .

  The present invention has been made in view of the above-mentioned problems, and an object of the present invention is to extract related terms while reducing the burden caused by human maintenance.

In order to solve the above problems, the first aspect of the present invention provides:
Search content acquisition means for outputting different queries to each of a plurality of search engines that search for different media, and acquiring content data searched by the search engines for each query,
Similarity calculation means for calculating the similarity of the content data acquired from each search engine with respect to the output of the different queries, between the queries;
Related word specifying means for specifying the queries as related words based on the calculated similarity between different queries output to the search engine;
Each of a plurality of queries specified as the related term, and a related term storage unit that stores the plurality of queries in association with the type of the search engine from which each of the plurality of queries is output;
It is characterized by having.

According to the first aspect, since the different queries are identified as related words based on the similarity of the content data for each different query searched by various search engines, the semantic similarity between the queries is Therefore, it is possible to extract related words while reducing the burden caused by human maintenance. In addition, since the query specified as the related term and the type of the search engine in which the query is searched are stored in association with each other, the type of the search engine is also presented to the user when the related term is presented to the user. become able to.

Further, in the second aspect of the present invention, the apparatus further comprises storage means for accumulating and storing the history of queries searched by each of the plurality of search engines for each search engine,
The search content acquisition means is characterized by extracting one query from the stored query history for each search engine and outputting it to the corresponding search engine.

  According to the second aspect, the related word is specified using the query history in each search engine. Therefore, the related word can be specified using the query input by the user in the past.

Further, acquired in the third aspect of the present invention, prior to texture Deer, text, still images, including videos, at least one of audio, the similarity calculation means, from each search engine to the different queries The similarity is calculated by comparing feature values of text, still image, moving image, or audio included in the content data, between the content data.

  According to the third aspect, the similarity is calculated based on the text, still image, moving image, and audio feature amounts included in the searched content data, and thus related terms that take into account the content of the content represented by each query Can be identified.

  In the fourth aspect of the present invention, the content data searched by the search engine is ranked, and the similarity calculation means weights the similarity by the rank given to the content data. It is characterized by doing.

  According to the fourth aspect, the similarity is weighted by using the rank given to the content data searched in each query. Therefore, the similarity between contents highly relevant to the query is regarded as important. Since the related word extraction is performed, the related word extraction accuracy can be improved.

Further, in the fifth aspect of the present invention, a page for generating a page that allows a plurality of queries specified as the related words stored in the related word storage means to be searched by the output search engine. It further has a generation means .

According to the fifth aspect, when the related word is presented to the user, the type of the search engine can be presented to the user.

  According to the present invention, it is possible to extract related terms while reducing the burden caused by artificial maintenance.

The block diagram which shows an example of a function structure of a search server. The figure which shows the data structural example of an index, a query log, and related word DB. The flowchart which shows the specific process content of a related word extraction process. The conceptual diagram for the illustration of calculation of multimedia similarity. The figure which shows the combination pattern of the query in the case of calculation of multimedia similarity. Display example of search screen.

[Apparatus configuration of this embodiment]
An embodiment when the related word extracting device of the present invention is applied to the search server shown in FIG. 1 will be described with reference to the drawings. The apparatus of this embodiment performs a web search using a query.

  A block diagram showing an example of the functional configuration of the search server 1 of the present embodiment is shown in FIG. The search server 1 is connected to the user terminal T via a communication network such as the Internet so as to be able to communicate with each other.

  The user terminal T is a terminal having an input function for a user to input a query for search and an output function for displaying and outputting a search result corresponding to the query, and includes a CPU, an input device, a display device, and the like. This is realized by a personal computer or a portable terminal.

  The search server 1 performs a search based on the query input at the user terminal T, and returns the search result to the user terminal T. Further, the user terminal T is configured such that the content to be searched can be specified by the user. For example, by selecting the content tab TB on the search screen W as shown in FIG. 6, it is possible to specify content such as web, still image (hereinafter simply referred to as “image”), moving image, and product information.

  The search server 1 is configured to include a search engine E that searches various contents, and generates a search result for each content in response to a query input at the user terminal T and returns the search result to the user terminal T.

  As shown in FIG. 1, the search server 1 includes a query receiving unit 10, a search result acquiring unit 20, a search result output unit 30, a web search engine E1, an image search engine E3, and a video search as various search engines E. An engine E5, a product search engine E7, a related word extraction unit 40, and a related word DB 50 are provided.

  The query receiving unit 10 receives and receives a search query from the user terminal T. The query is composed of a single keyword or a combination of a plurality of keywords. In addition, the query receiving unit 10 receives and receives information on the type of content specified as the search target (content specifying information).

  The search result acquisition unit 20 selects a search engine E corresponding to the content designation information received by the query reception unit 10 and outputs a query to the search engine E, thereby retrieving the search result corresponding to the query. To get.

  Specifically, when the web is designated based on the content designation information, a query is output to the web search engine E1. Further, the query is output to the image search engine E3 when an image is specified, to the video search engine E5 when a moving image is specified, and to the product search engine E7 when product information is specified.

  The search result output unit 30 is for outputting the search result obtained by the search result acquisition unit 20 to the user terminal T. Specifically, the search result output unit 30 is display data that can access the image data of the search result (for example, by HTML). Web page data) is generated and transmitted to the user terminal T.

  The various search engines E compare the keyword included in the query with the keyword stored in the index D to obtain a search result for the query. Here, the web search engine E1 searches for web content including text, images, moving images, and the like. The image search engine E3 searches for still images, and the moving image search engine E5 searches for moving images. The product search engine E7 searches product information posted on the sales site.

  Each search engine E collects content on the Internet by crawling in so-called robot search, and indexes the content. For example, the web search engine E1 collects web pages by circulating URLs posted on the web pages, and indexes the URLs using keywords included in the web pages.

  The image search engine E3 collects image data from the image posting site, and indexes the URL of the image posting site based on tag information (keyword) given to the image data. At this time, the tag information may be set by dividing text information described in the vicinity of the image into words by morphological analysis or the like on the web page on which the image is posted. Similarly, the video search engine E5 and the product search engine E7 perform crawling and indexing to create the index D.

  As shown in FIG. 1, each search engine E includes an index D (D1, D3, D5, D7) and a query log L (L1, L3, L5, L7).

  The index D is a so-called transposed index. As shown in FIG. 2A, the index is a keyword, the URL of the web page (content data) on which the content is posted, and the relationship between the keyword and the content data. It is a database that stores weight values indicating sex and the content data in association with each other. At the time of the above crawling, each search engine E indexes and stores a keyword (including tag information) extracted from the content data in the URL and content data collected by crawling.

  Further, a weight value indicating the degree of association between the keyword and the content data is calculated by TF / IDF or the like and stored in association with it.

  When a query is input from the search result acquisition unit 20, each search engine E searches the index D based on the query, and generates a list of URLs as the search result. When generating this list of search results, ranking is given to the URLs of the search results in descending order of the weight value, which is the degree of association of the web page with the query.

  As a ranking algorithm of the search engine E, the importance of the web page by analyzing the link relation between the web pages as well as the method of using the importance of the keyword in the web page as in the above TF / IDF or the like. It may be a method used, or a combination thereof.

  The query log L is a database for accumulating and storing queries searched by each search engine E, and stores the query and the number of searches in association with each other as shown in FIG. When a query is input from the search result acquisition unit 20, each search engine E stores the query in each query log L and updates (adds 1) the number of searches.

  Note that the data configuration of FIG. 2 is an example, and can be changed as appropriate according to the implementation. For example, although the data example in which the total number of searches that have been aggregated for the query is illustrated and described, a database that stores and stores the date and time when the search is performed for the query is provided separately, and this database is regularly The query log L is updated by summing up automatically. In addition, as the index D, only the document ID indicating the keyword, the content data, and the weight value are stored in association with each other, and the URL and the content data are separately stored in association with the document ID in another database. Also good.

  The related word extracting unit 40 is a functional unit that extracts related words based on the query log L of each search engine E. As shown in FIG. 1, the search content acquiring unit 42, the similarity calculating unit 44, and the related word specifying unit 46 is comprised. The functions of these means will be described later.

  The related word DB 50 is a database that stores keywords extracted as related words by the related word extraction unit 40 in association with each other, and stores a combination of keywords and target engines in association with each other as shown in FIG. To do. The target engine is data indicating the search engine E in which the keyword is actually searched.

  By storing this target engine in association with each keyword, for example, when a related word is presented to the user, a search engine E that is often searched by the query, such as the query candidate Q in FIG. Can be presented.

[Detailed explanation of related word extraction unit]
Next, a specific operation of the related word extracting unit 40 will be described with reference to FIGS. The related word extraction processing performed by the related word extraction unit 40 is performed on the query logs L of the two search engines E, and all combinations of the search engines E (for example, web search and image search, web search and video search,.・ ・ Perform image search and video search. Moreover, it is preferable to perform it for all the combinations of queries stored in each query log L, but it is also possible to perform the search for the higher number of searches associated with the query.

  First, the search content acquisition unit 42 of the related word extraction unit 40 extracts different queries one by one from the query logs L of the two search engines E (step S11). Then, the extracted query is output to each search engine E, and a predetermined number of content data corresponding to the search result group (search result list) searched by each search engine E (for example, top N, N Is an arbitrary natural number) (step S12).

  For example, in the illustration of FIG. 4, “Tokyo” is extracted from the query log L1 of the web search engine E1, and the search result group A as a result of searching the web search engine E1 in “Tokyo” is acquired.

  Further, “Rainbow Bridge” is extracted from the query log L3 of the image search engine E3, and a search result group B as a result of searching the image search engine E3 with “Rainbow Bridge” is acquired.

  Next, the similarity calculation means 44 of the related word extraction unit 40 calculates the similarity between the search result groups acquired from the two search engines E (step S13). The similarity calculated here is referred to as “multimedia similarity”.

  The multimedia similarity is an index that comprehensively considers the similarity between media such as text, images, and audio included in content data. Specifically, it is calculated based on the following equation.

[Formula 1]
Multimedia similarity = (text similarity + image similarity + video similarity) / (number of types of media included in content)

  The number of types of media included in the content is set according to the search target of each search engine E. For example, if the content includes text, images, and moving images as in web search, it is “3”. It becomes '1'. The number of types of media may be the value of the smaller number of types of media between search targets in the combination of search engines E that calculate multimedia similarity, or the number of media that is common between search targets may be set May be.

  The text similarity is a similarity between texts included in the content. For example, the similarity is obtained from the cosine distance between the vectors by expressing the feature amount of the text by a multidimensional vector based on the number of appearances of the keyword included in the text.

  Regarding the search result group A of the web search engine E1 and the search result group B of the image search engine E3, assuming that N contents are included in the search result group, each content in the search result group Are represented by A1, A2, A3,... AN, B1, B2, B3,.

  The text similarity between the search result group A and the search result group B is obtained as follows.

[Formula 2]
Text similarity = [MAX {similarity (text A1, text B1), similarity (text A1, text B2), ..., similarity (text A1, text BN)} + MAX {similarity (text A2, Text B1), similarity (text A2, text B2), ..., similarity (text A2, text BN)} + ... + MAX {similarity (text AN, text B1), similarity (text AN , Text B2), ..., similarity (text AN, text BN)}] / N

  MAX {} means a function for selecting the maximum value, and similarity () means a function for calculating the similarity. That is, the text similarity is obtained by averaging the maximum similarities between the most similar texts in the search result group.

  The image similarity is the similarity between images included in the content. This similarity is obtained, for example, from the Euclidean distance between the vectors by expressing the feature quantities such as colors, shapes, and patterns extracted from the images by multidimensional vectors. If the image similarity is obtained for the search result group of the web search engine E1 and the image search engine E3, it is obtained by the following equation.

[Formula 3]
Image similarity = [MAX {similarity (image A1, image B1), similarity (image A1, image B2), ..., similarity (image A1, image BN)} + MAX {similarity (image A2, Image B1), similarity (image A2, image B2), ..., similarity (image A2, image BN)} + ... + MAX {similarity (image AN, image B1), similarity (image AN , Image B2), ..., similarity (image AN, image BN)}] / N

  The moving image similarity is a similarity between moving images included in the content. This similarity is represented by, for example, image features such as colors, shapes, and patterns extracted from a moving image, and feature amounts such as object motion and audio signals expressed by multidimensional vectors. Calculated by distance. When the moving image similarity is obtained for the search result group of the web search engine E1 and the image search engine E3, the following equation is obtained.

[Formula 4]
Video similarity = [MAX {similarity (video A1, video B1), similarity (video A1, video B2), ..., similarity (video A1, video BN)} + MAX {similarity (video A2, Video B1), similarity (video A2, video B2), ..., similarity (video A2, video BN)} + ... + MAX {similarity (video AN, video B1), similarity (video AN , Video B2), ..., similarity (image AN, image BN)}] / N

  As described above, the text similarity, the image similarity, and the video similarity between the search result groups are obtained and divided by the number of types of media included in the content, so that the multimedia similarity that is an average value thereof is calculated. calculate. Based on the contents of the contents of the search result group, the similarity between the queries from which the search results are derived can be calculated based on the multimedia similarity. Further, by selecting the maximum value using MAX {} for calculating each similarity, the two search result groups are compared, and the multimedia similarity can be calculated based on the most similar content data.

  In the case where a plurality of media of the same type are included in the content as in the search results A1 and A2 in FIG. 4, the maximum similarity is extracted for each of them as described above, and the average is extracted. A value may be used. Moreover, it is preferable that the value of each similarity is normalized with respect to the maximum similarity in each medium (for example, according to the type of text or image).

  The search content acquisition means 42 of the related word extraction unit 40 determines whether or not queries have been extracted in all combinations between the query logs L of the two search engines E (step S14). For example, as shown in FIG. 5, when the total number of queries in the web search query log L1 is M and the total number of queries in the image search query log L3 is L, queries in a combination of L × M Is extracted. Therefore, when there are three or more search engines E, the multimedia similarity between queries is calculated for each search engine E combination.

  When the similarity calculation unit 44 calculates the multimedia similarity between queries extracted from the query log L of each search engine E, the multimedia similarity is stored in the memory in association with the combination of the queries. That is, the multimedia similarity is stored according to the correspondence as shown in FIG.

  The related word specifying means 46 extracts a related word based on the multimedia similarity calculated in step S13 and stores it in the related word DB 50 (step S15). Specifically, among the multimedia similarities calculated with respect to a certain query Q1, a predetermined upper number (for example, up to the 10th) is selected, and the query for which the multimedia similarity is calculated is referred to as query Q1. Are identified as related words and stored in the related word DB 50 in association with each other. In this storage, the identification information of the search engine E that searched for the query is stored in association with the target engine.

  For example, as shown in FIG. 5, among the queries 1 to L in the image search query log L3, queries having the top ten multimedia similarity to the web search query “Tokyo” are specified as related words.

  For example, if the query “Rainbow Bridge” of the image search engine E3 is specified as a related word for the query “Tokyo” of the web search engine E1, it is a target engine of “Tokyo” and “Rainbow Bridge”. “Web search engine E1” and “image search engine E3” are stored in association with each other.

  When the search result output unit 30 creates a list page of search results obtained by the search result acquisition unit 20, the search result output unit 30 extracts related words associated with the query from the related word DB 50, and stores them as query candidates on the page. Create by embedding. In order to display the query candidates, it is preferable to set a search instruction for the search engine E as a link. The search screen W shown in FIG. 6 is a display example of the query candidate Q. By selecting the query candidate Q, the user can input a re-search with the query.

  In addition, when displaying a query candidate, a search engine E that is preferably searched for the query candidate may be displayed by a target engine stored in association with a related word. In this case, the search instruction for the search engine E associated with the related word is set as a link to the related word as a query candidate. For example, when the user selects a query candidate Q on the search screen W in FIG. 6, a search instruction in the search engine E corresponding to the query can be input.

  As described above, according to the present embodiment, the search result is calculated by calculating the similarity (multimedia similarity) between the search result groups in each search engine E based on the query stored in the query log L. The related words can be extracted in consideration of the similarity of media such as text, still images, and moving images included in. Therefore, it is possible to extract related terms while reducing artificial costs.

  In general, it is easy to extract a query (for example, “Tokyo Station”) that includes a part of a query (for example, “Tokyo”) input by the user as a related word extraction method. It is possible to extract related words in consideration of not only the similarity on the character string but also the meaning (contents) of the query from the content data.

[Modification]
The embodiment described above is an example to which the present invention is applied, and the applicable range is not limited to the above example.

  For example, a related word may be extracted from a query log in the voice search engine by adding a voice similarity to the multimedia similarity using a voice search engine that searches voice data such as music and recorded voice. Search engines for other media such as music, news, blogs, maps, etc. can also be used.

  In addition, when calculating the multimedia similarity, the ranking given to the search result may be used as a weight. For example, when ranks 1 to N of the search results are ranked, the above-described text similarity is obtained by the following equation.

Text similarity =
(MAX {similarity (text A1, text B1) x (1 / (1 * 1)), similarity (text A1, text B2) x (1 / (1 * 2)), ..., similarity ( Text A1, text BN) x (1 / (1 * N))}
+ MAX {similarity (text A2, text B1) x (1 / (2 * 1), similarity (text A2, text B2) x (1 / (2 * 2), ..., similarity (text A2 , Text BN) × (1 / (2 * N)}
+ ...
+ MAX {similarity (text AN, text B1) x (1 / (N * 1), similarity (text AN, text B2) x (1 / (N * 2), ..., similarity (text AN , Text BN) × (1 / (N * N)}] / N
= [MAX {similarity (text A1, text B1) x 1, similarity (text A1, text B2) x 2, ..., similarity (text A1, text BN) x N} / 1
+ MAX {similarity (text A2, text B1) x 1, similarity (text A2, text B2) x 2, ..., similarity (text A2, text BN) x N} / 2
+ ...
+ MAX {similarity (text AN, text B1) x 1, similarity (text AN, text B2) x 2, ..., similarity (text AN, text BN) x N} / N] / N

  In this way, by weighting the ranking of search results in calculating similarity, it is possible to calculate multimedia similarity that focuses on content included in search results that are highly relevant to the query. More related words can be extracted.

  In addition, the related words are extracted using the queries stored in the query log L of each search engine E. For example, a database (for example, a national language dictionary DB) storing a plurality of keywords is provided and stored in this database. It is also possible to extract the related keywords as described above by extracting the different keywords that have been extracted and outputting them to each search engine E.

  In addition, as an expression for calculating the multimedia similarity, the average value of the text similarity, the image similarity, and the moving image similarity has been described as in Expression 1, but the maximum value or the minimum value among them is used. Also good.

  Specifically, the maximum value among the calculated text similarity, image similarity, and moving image similarity may be set as the multimedia similarity. According to this, it is possible to extract related words based on a multimedia similarity that places importance on a medium having a high similarity.

  Further, the minimum value among the text similarity, the image similarity, and the moving image similarity may be set as the multimedia similarity. According to this, it is possible to increase the accuracy of extracting related words by strictly setting a value as a reference for extracting related words.

  A value obtained by multiplying the text similarity, the image similarity, and the moving image similarity may be set as the multimedia similarity. According to this, unless all the similarity values are increased, the multimedia similarity value is not increased, so that related words in which the respective media are generally similar are extracted.

  Further, in the calculation of text similarity, image similarity, and video similarity, it has been explained that MAX {} selects the maximum similarity between the content data of each search result, but MIN {} is used. By selecting the minimum value, the criteria for determining the similarity may be tightened, or each similarity may be comprehensively considered using AVE {}.

  The operation of the embodiment can be performed by incorporating appropriate computer software into the computer. The contents of the present invention are not limited to the above embodiment. In the present invention, various modifications can be made to the specific configuration within the scope of the claims.

  For example, each component only needs to exist as a functional block, and does not need to exist as independent hardware. As a mounting method, hardware or computer software may be used. Furthermore, one functional element in the present invention may be realized by a set of a plurality of functional elements, and a plurality of functional elements in the present invention may be realized by one functional element.

  Moreover, the functional element may be arrange | positioned in the position physically separated. In this case, the functional elements may be connected by a network. It is also possible to realize functions or configure functional elements by grid computing.

D Index E Search engine E1 Web search engine E3 Image search engine E5 Video search engine E7 Product search engine L Query log Q Query candidate 1 Search server 10 Query reception unit 20 Search result acquisition unit 30 Search result output unit 40 Related word extraction unit 50 Related Word DB

Claims (7)

Search content acquisition means for outputting different queries to each of a plurality of search engines that search for different media, and acquiring content data searched by the search engines for each query,
Similarity calculation means for calculating the similarity of the content data acquired from each search engine with respect to the output of the different queries, between the queries;
Related word specifying means for specifying the queries as related words based on the calculated similarity between different queries output to the search engine;
Each of a plurality of queries specified as the related term, and a related term storage unit that stores the plurality of queries in association with the type of the search engine from which each of the plurality of queries is output;
A related word extraction device comprising: Storage means for accumulating and storing the history of queries searched by each of the plurality of search engines for each search engine;
The search content acquisition means includes
2. The related word extraction apparatus according to claim 1, wherein one query is extracted from the stored query history for each search engine and is output to the corresponding search engine. Before texture Deer include text, still images, moving images, at least one of voice,
The similarity calculation means includes:
The similarity is calculated by comparing text, still image, video, or audio feature amounts included in content data acquired from each search engine with respect to the different queries between the content data. The related word extraction device according to claim 1 or 2. The content data searched by the search engine is ranked,
The similarity calculation means includes:
The related word extraction device according to any one of claims 1 to 3, wherein the similarity is weighted according to a rank assigned to the content data. The system further comprises page generation means for generating a page that allows a plurality of queries specified as the related words stored in the related word storage means to be searched by the output search engine. The related word extraction apparatus as described in any one of 1-4. A search content acquisition step of outputting a different query to each of a plurality of search engines that search different media, and acquiring content data searched by the search engine for each query;
A similarity calculation step of calculating the similarity of content data acquired from each search engine with respect to the output of the different queries, between the queries;
A related word specifying step of specifying the queries as related words based on the calculated similarity between different queries output to the search engine;
A related word storage step of associating each of the plurality of queries specified as the related word with the type of search engine from which each of the plurality of queries is output in a related word storage unit;
The related word extraction method characterized by performing computer.   A program for causing a computer to execute the related word extracting method according to claim 6.

Images