JP6985189B2 - Data collection equipment, data collection methods, and programs - Google Patents

Description

The present invention relates to a data collection device, a data collection method, and a program.

Conventionally, a crawler that collects data such as texts and images from the Web and automatically creates a database of the collected data is known (see, for example, Patent Document 1). Crawlers collect data from web pages of various hosts by following links in web pages. The data collected by the crawler is stored in the web information database. The data stored in the web information database is used for web page search services and the like.

On the other hand, when the search engine receives the search word input by the user, the information of the web page or application page related to the received search word (for example, URL: Uniform Resource Locator) is acquired from the web information database and acquired. Output a list of information as a search result.

Japanese Unexamined Patent Publication No. 2015-167039

In order to improve user satisfaction, the search engine may be operated to output text data and image data related to the search word input by the user together with the search result. In order to realize such a function, a database in which an entity and text data or image data related to the entity are associated with each other and stored may be used.

In order to expand the database as described above, a method for efficiently collecting image data of an entity is particularly required. In addition, in order to expand Japanese content, there is a need for a method of preferentially crawling hosts with a high percentage of Japanese pages.

The present invention has been made in consideration of such circumstances, and one of the objects of the present invention is to provide a data collection device, a data collection method, and a program capable of improving the efficiency of data collection.

One aspect of the present invention includes a first collection unit that collects content corresponding to reference information belonging to the host to be evaluated from a device accessible via a network, and a content collected by the first collection unit. It is a data collecting device including an evaluation unit that evaluates the host to be evaluated based on the number of associated image data different from each other.

According to one aspect of the present invention, it is possible to improve the efficiency of data collection.

It is a figure which shows the use environment of the crawl server 1 which concerns on embodiment. It is a figure which shows an example of the knowledge graph which concerns on embodiment. It is a functional block diagram which shows the structure of the crawl server 1 which concerns on embodiment. It is a flowchart which shows an example of the flow of the host evaluation process of the crawl server 1 which concerns on embodiment. It is a figure which shows an example of the URL list information D1 which concerns on embodiment. It is a figure which shows an example of the content information D2 which concerns on embodiment. It is a figure which shows an example of the host information D3 which concerns on embodiment. It is a figure which shows an example of the host information D3a when the host evaluation is performed in consideration of the path structure which concerns on embodiment. It is a flowchart which shows an example of the flow of the image collection process of the crawl server 1 which concerns on embodiment. It is a figure which shows an example of the image information D4 which concerns on embodiment.

[Overview]
Hereinafter, a data collection device, a data collection method, and an embodiment of a program will be described with reference to the drawings. In this embodiment, it is assumed that the data collection device is a crawl server. A crawl server is a server that automatically collects data from a plurality of devices that can be accessed via a network such as the Internet. The crawl server of the present embodiment evaluates the host regarding the collection of the representative image of the entity, and concentrates the crawl on the excellent host having a high evaluation.

The crawl server may be realized by one processor, or may be realized by distributed processing by a plurality of processors. In the present embodiment, the host constitutes a part of reference information (for example, URL) for referring to data stored in a device accessible via a network, and specifies a group of a plurality of reference information. Affiliation information. Hereinafter, embodiments will be described.

[overall structure]
FIG. 1 is a diagram showing a usage environment of the crawl server 1 (data collection device) of the present embodiment. The crawl server 1 collects page data (contents) including at least one of image data and HTML (HyperText Markup Language) data from a plurality of external servers S1 (devices) accessible via the network NW. The content is stored in the external server S1 and is page-based data that can be viewed by a browser. However, the content is not limited to the browser, and may be data played by the application program. The network NW includes the Internet, WAN (Wide Area Network), LAN (Local Area Network) and the like.

[Knowledge Graph Configuration]
In this embodiment, the data collected by the crawl server 1 is used, for example, to generate a knowledge graph. The Knowledge Graph is data in which an entity, class, or property is associated with the location information of a web page in which related data related to the entity is described. An entity may represent, for example, the entity of an object (eg, an object that exists in the real world) or is defined within the concept of an object (eg, in the real world or virtual world). It may represent a concept). For example, when the object is the concept of "building", the entity may represent an entity such as "○○ tower" or "○○ building". Further, for example, when the object is the concept of "economics", the entity may represent an insubstantial abstract concept such as "microeconomics" or "macroeconomics".

FIG. 2 is a diagram showing an example of the Knowledge Graph according to the present embodiment. The things described in the Knowledge Graph are defined by the ontology. An ontology is a definition of a class and a property of an object, and is a collection of constraints that hold between the class and the property.

A class is a group of things that have the same properties in an ontology. What the nature of an object is, that is, which class the object belongs to, is determined by the properties described below.

For example, things that have a beak, an oviparous vertebrate, and forelimbs that are wings are classified in the "bird" class. Also, in the class of "birds", things that have the property of not being able to fly are classified into lower classes such as "penguins" and "ostriches". In this way, the class system may have a hierarchical structure having a higher-lower relationship. The nature of the upper class is inherited by the lower class. In the example above, the "bird" class's "beak-bearing, oviparous vertebrate with winged forelimbs" property is a lower class property of "penguins" and "ostriches". Will also be included. For example, if the class name for identifying the class is a class "bird", the class name may be represented by the character string "bird". The class name does not necessarily have to represent a meaning, and for example, even in the class "bird", a character string indicating mere identification information such as "information 1" or "C1" may be assigned. .. The above-mentioned entities, or things, shall belong to any of the classes contained within the class system defined by the ontology.

Properties are attributes that describe the nature and characteristics of things and the relationships between classes. For example, a property may be an attribute that has the property of "having ~ as a component of the body" or the property of "living in ...", or "a class is a higher class and a certain class is a lower class". It may be an attribute indicating the relationship between the upper and lower levels of "class". As for the property name for identifying the property, the property name itself may or may not represent the meaning, as in the class name described above.

A knowledge graph is a directed graph in which the above-mentioned classes are represented as nodes and the above-mentioned properties are represented as labeled and directional edges. With such a graph structure, information about things can be discriminated by nodes, and relationships between things can be discriminated by edges.

In the example knowledge graph shown in FIG. 2, the baseball player A "entity E1" and the baseball team B "entity E2" belong to the class with the property name "affiliated team". Further, the baseball player A "entity E1" and the "baseball player AN1" belong to the class of the property name "name". Further, the baseball player A "entity E1" and the baseball player A "image P1" belong to the class with the property name "representative image". Further, the baseball team B "entity E2" and "baseball ream BN2" belong to the class of the property name "name". The data collected by the crawl server 1 is used, for example, as a representative image of an entity in the knowledge graph as described above.

[Crawl server configuration]
Hereinafter, the configuration of the crawl server 1 will be described. FIG. 3 is a functional block diagram showing the configuration of the crawl server 1. The crawl server 1 includes, for example, a data collection unit 10 (first collection unit), a host evaluation unit 20, an image collection processing unit 30, and a storage unit 40. For example, the URL list information D1, the content information D2, the host information D3, and the image information D4 are stored in the storage unit 40.

The URL list information D1 stores a list of a large number of URLs to be crawled. The URL list information D1 may be a list of URLs extracted from the text data of the content collected by the crawl. Alternatively, the URL list information D1 may be manually set by, for example, the administrator of the crawl server 1. The content information D2 stores information on the content collected by the crawl. The host information D3 stores the evaluation result of the host to be evaluated. The image information D4 stores a representative image of the entity collected by the crawl.

The data collection unit 10 (fetcher) collects (fetchs) content from a plurality of external servers S1. The collected content includes, for example, HTML data and image data. The data collecting unit 10 stores the collected content in the content information D2 of the storage unit 40.

The host evaluation unit 20 evaluates the host based on the presence or absence of a specific tag set as specific information in the content. The "specific tag" is a tag containing text indicating the content of the content, such as an OGP (Open Graph Protocol) tag. The OGP tag is a collection of the title of the linked content (hereinafter, OGP title), the URL of the image data related to the linked content (hereinafter, OGP image), the text data indicating the outline of the linked content, and the like. Information. The host evaluation unit 20 evaluates the host based on the presence or absence of the OGP image included in the OGP tag, for example.

The OGP image is an image set as a representative image of the link. Therefore, the OGP image can correspond to an image that can be used as a representative image of an entity. The OGP image is designated by the reference information (for example, URL) as the meta information of each content. The OGP image is specified by the OGP tag "<meta property =" og: image "content =" URL "/>.

Further, the OGP image is highly related to the OGP title included in the OGP tag specified as the meta information of each content page. Therefore, this OGP title can correspond to a character string that can be used as an entity name (name information). The OGP title is specified by the OGP tag "<meta property =" og: title "content =" title "/>.

The host evaluation unit 20 includes, for example, a sampling unit 21 (first collection unit), a unique OGP rate calculation unit 23 (evaluation unit), and a Japanese language determination unit 25 (evaluation unit). The sampling unit 21 extracts the URL belonging to the host to be evaluated from the URL list information D1. The number of URLs to be extracted is arbitrary. The sampling unit 21 inputs the extracted URL into the data collection unit 10. As a result, the data collection unit 10 collects content from the plurality of external servers S1 based on the URL input from the sampling unit 21.

The unique OGP rate calculation unit 23 analyzes the content collected by the data collection unit 10 and stored in the storage unit 40. The unique OGP rate calculation unit 23 evaluates the host based on the number of different image data associated with the collected content. For example, the unique OGP rate calculation unit 23 determines whether or not an OGP tag (OGP image) is included in each header portion of HTML data which is a content. The unique OGP rate calculation unit 23 calculates the unique OGP rate for each host to be evaluated. The unique OGP rate is an index value for evaluating the variety of OGP images included in the collected content for each host. Even when a large number of OGP images are collected, if the OGP images overlap each other, it is not possible to efficiently increase the images that can be used as the entity representative image. For example, when the same logo image representing the host is set as an OGP image in all the collected contents, only one image can be used as the entity representative image. Therefore, in the present embodiment, the host evaluation is performed based on the unique OGP rate that can determine the variety of OGP images. The unique OGP rate is expressed by, for example, the following equation (1).

Unique OGP rate = Unique number of collected OGP images / Total number of collected OGP images ... Equation (1)

For example, if the total number of OGP images (with duplication) collected for the host to be evaluated is 10, and the unique number of OGP images excluding duplication is 5, the unique OGP rate is 0.5 (with duplication). = 5/10). The unique OGP rate indicates that the larger the value, the higher the variety and the higher the evaluation of the host, and the smaller the value, the lower the variety and the lower the evaluation of the host.

Whether or not the collected OGP images are duplicated is determined, for example, based on the URL of the link destination of the OGP images included in the OGP tag. The unique OGP rate calculation unit 23 determines that the OGP images having the same character string of the URL of the link destination of the OGP image included in the OGP tag are duplicated with each other. The unique OGP rate calculation unit 23 stores the calculated unique OGP rate in the host information D3 stored in the storage unit 40 in association with the host to be evaluated.

That is, the unique OGP rate calculation unit 23 evaluates the host based on the ratio of the total number of image data associated with the collected content to the number of image data different from each other. Further, the unique OGP rate calculation unit 23 evaluates the host so that the evaluation increases as the ratio of the number of different image data to the total number of image data increases.

The Japanese determination unit 25 determines whether or not each of the collected contents is Japanese content (specific language content). Further, the Japanese language determination unit 25 calculates the Japanese content rate for each host to be evaluated. The Japanese content rate is an index value indicating the ratio of Japanese content contained in the collected content. The Japanese content rate is expressed by, for example, the following equation (2).

Japanese content rate = Number of Japanese contents collected / Total number of collected contents ・ ・ ・ Equation (2)

For example, if the total number of contents collected for the host to be evaluated is 10, and the number of Japanese contents is 9, the Japanese content rate is 0.9 (= 9/10). .. As for the Japanese content rate, the larger the value, the higher the ratio of Japanese content and the higher the evaluation of the host, and the smaller the value, the lower the ratio of Japanese content and the lower the evaluation of the host.

Whether or not the collected content is Japanese content is determined based on, for example, a character code, dictionary data for each language, or the like. The Japanese language determination unit 25 stores the calculated Japanese content rate in the host information D3 stored in the storage unit 40 in association with the host to be evaluated.

That is, the Japanese determination unit 25 evaluates the host based on the ratio of the total number of collected contents to the number of predetermined language contents described in the predetermined language included in the contents. Further, the Japanese language determination unit 25 evaluates the host so that the evaluation increases as the ratio of the predetermined language content to the total number of collected contents increases.

The image collection processing unit 30 selects a good host suitable for collecting the entity representative image based on the evaluation result by the host evaluation unit 20, and intensively crawls the URL belonging to the selected host to represent the entity. Collect images that can be images. That is, the image collection processing unit 30 intensively crawls hosts that are expected to collect many types of OGP images. The image collection processing unit 30 includes, for example, a host selection unit 31 (second collection unit) and an image URL selection unit 33 (third collection unit).

The host selection unit 31 selects an excellent host (hereinafter referred to as an excellent host) suitable for collecting entity representative images based on the evaluation result by the host evaluation unit 20. For example, the host selection unit 31 selects a host whose Japanese content rate is equal to or higher than a predetermined threshold and whose unique OGP rate is equal to or higher than a predetermined threshold as a good host. For example, the host selection unit 31 selects a host having a Japanese content rate of "0.9" or more and a unique OGP rate of "0.95" or more as an excellent host.

The host selection unit 31 extracts the URL belonging to the excellent host from the URL list information D1 and inputs it to the data collection unit 10. As a result, the data collection unit 10 collects the content from the plurality of external servers S1 based on the URL input from the host selection unit 31, and stores the content in the content information D2 of the storage unit 40. The number of URLs extracted by the host selection unit 31 is arbitrary.

The image URL selection unit 33 determines whether or not an OGP tag (OGP image) is included in each header portion of the content collected based on the URL extracted by the host selection unit 31. When the image URL selection unit 33 determines that the OGP image is included, the image URL selection unit 33 selects the URL of the link destination of the OGP image as the URL to be collected and inputs it to the data collection unit 10. As a result, the data collection unit 10 collects content that is an image from the external server S1 based on the URL input from the image URL selection unit 33.

The image URL selection unit 33 stores in the storage unit 40 a set of an OGP title included in the OGP tag and an image collected based on the linked URL of the OGP image included in the OGP tag. It is stored in the image information D4. The OGP title stored in the image information D4 can be used as an entity name. Further, the image stored in the image information D4 can be used as a representative image of the entity.

Each functional unit of the crawl server 1 is realized, for example, by executing a program (software) by a hardware processor such as a CPU (Central Processing Unit) in a computer. In addition, some or all of these components are hardware (circuits) such as LSI (Large Scale Integration), ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array), GPU (Graphics Processing Unit), etc. It may be realized by the part; including circuitry), or it may be realized by the cooperation of software and hardware.

The storage unit 40 of the crawl server 1 is, for example, a RAM (Random Access Memory), a ROM (Read Only Memory), an HDD (Hard Disk Drive), a flash memory, or a hybrid storage device in which a plurality of these are combined. It will be realized. Further, a part or all of the storage unit 40 may be an external device such as NAS or an external storage server that can be accessed by the crawl server 1.

[Host evaluation process]
Hereinafter, the host evaluation process of the crawl server 1 will be described. FIG. 4 is a flowchart showing an example of the flow of the host evaluation process of the crawl server 1. The process according to this flowchart may be executed by a batch process executed at a predetermined timing such as daily or weekly.

In the following example, it is assumed that the URL list information D1 is stored in the storage unit 40 in advance. FIG. 5 is a diagram showing an example of URL list information D1 stored in the storage unit 40. As shown in FIG. 5, the URL list information D1 includes URL information relating to a plurality of hosts.

First, the sampling unit 21 selects a host to be evaluated (S101). For example, the sampling unit 21 refers to the URL included in the URL list information D1 and selects a host to be evaluated. Alternatively, the sampling unit 21 may select a host designated via a reception unit (not shown) that accepts input by an administrator or the like as a host to be evaluated. The number of hosts selected is arbitrary.

Next, the sampling unit 21 selects one of the hosts to be evaluated and extracts the URL belonging to the host from the URL list information D1 (S103). The number of URLs to be extracted is arbitrary. Next, the sampling unit 21 inputs the extracted URL into the data collection unit 10. The data collection unit 10 collects content from a plurality of external servers S1 based on the URL input from the sampling unit 21 (S105). The data collecting unit 10 stores the collected content in the content information D2 of the storage unit 40.

FIG. 6 is a diagram showing an example of the content information D2 collected by the data collection unit 10. As shown in FIG. 6, the URL and the content collected based on the URL are associated and stored in the content information D2. In the content information D2, for example, the URL "host 1 / directory 1 / file 1" and the content "HTML data 1" are stored in association with each other.

Next, the unique OGP rate calculation unit 23 calculates the unique OGP rate (S107). The unique OGP rate calculation unit 23 stores the calculated unique OGP rate in the host information D3 stored in the storage unit 40 in association with the host to be evaluated.

Next, the Japanese determination unit 25 determines whether or not each of the contents is Japanese content, and calculates the Japanese content rate (S109). The Japanese language determination unit 25 stores the calculated Japanese content rate in the host information D3 stored in the storage unit 40 in association with the host to be evaluated.

FIG. 7 is a diagram showing an example of host information D3. As shown in FIG. 7, the host information D3 includes the host to be evaluated, the unique OGP rate calculated by the unique OGP rate calculation unit 23, and the Japanese content rate calculated by the Japanese determination unit 25. It is associated and stored. In the host information D3, for example, the host "host 1", the unique OGP rate "0.7", and the Japanese content rate "1.0" are stored in association with each other.

The unique OGP rate calculation unit 23 may calculate the unique OGP rate in consideration of the path structure of the URL belonging to the host to be evaluated. Further, the Japanese language determination unit 25 may calculate the Japanese content rate in consideration of the path structure of the URL belonging to the host to be evaluated. The accuracy of host evaluation can be improved by performing evaluation in consideration of the path structure of the URL belonging to the host to be evaluated.

FIG. 8 is a diagram showing an example of host information D3a when host evaluation is performed in consideration of the path structure of the URL belonging to the host to be evaluated. As shown in FIG. 8, the host information D3a stores the path structure, the unique OGP rate, and the Japanese content rate in association with each other. In the host information D3a, for example, the unique OGP rate of the content arranged directly under the path structure "host1 / directory 1 /" is "0.8", and the Japanese content rate is "1.0". Is stored. Further, in the host information D3a, for example, the unique OGP rate of the content arranged directly under the path structure "host1 / directory 3 /" is "0.5", and the Japanese content rate is "0.9". Some data is stored. In this way, even if the host is the same, the evaluation result can be given superiority or inferiority according to the path structure. In the example of the host information D3a, it can be seen that the unique OGP rate of the content arranged directly under "host1 / directory 1 /" is higher than that of the content arranged directly under "host1 / directory 3 /". In this way, the unique OGP rate calculation unit 23 groups the reference information based on the path structure of the reference information belonging to the host, and evaluates the host for each set of the reference information.

Next, the sampling unit 21 determines whether or not the evaluation for all the hosts to be evaluated is completed (S111). When the sampling unit 21 determines that the evaluation for all the hosts to be evaluated has not been completed, the sampling unit 21 selects one host to be evaluated and repeats the processing after S103. On the other hand, when it is determined that the evaluation for all the hosts to be evaluated is completed, the sampling unit 21 ends the processing of this flowchart.

[Image collection process]
Hereinafter, the image collection process of the crawl server 1 will be described. FIG. 9 is a flowchart showing an example of the flow of the image collection process of the crawl server 1. The process according to this flowchart may be executed by a batch process executed at a predetermined timing such as daily or weekly.

First, the host selection unit 31 selects an excellent host suitable for collecting entity representative images based on the evaluation result by the host evaluation unit 20 (S201). For example, the host selection unit 31 refers to the host information D3 stored in the storage unit 40, and sets a host whose Japanese content rate is equal to or higher than a predetermined threshold and whose unique OGP rate is equal to or higher than a predetermined threshold as an excellent host. Select as.

Next, the host selection unit 31 selects one of the selected excellent hosts, and extracts the URL belonging to the host from the URL list information D1 (S203). The number of URLs to be extracted is arbitrary. As shown in FIG. 9, when the result of the host evaluation considering the path structure is obtained, the URL having a specific path structure with high evaluation may be preferentially extracted.

Next, the host selection unit 31 inputs the extracted URL into the data collection unit 10. The data collection unit 10 collects content from a plurality of external servers S1 based on the URL input from the host selection unit 31 (S205).

Next, the image URL selection unit 33 determines whether or not an OGP image is included in each header portion of the collected content (S207). When the image URL selection unit 33 determines that the OGP image is included, the image URL selection unit 33 selects the URL of the link destination of the OGP image as the URL to be collected and inputs it to the data collection unit 10. As a result, the data collection unit 10 collects the content (image) from the external server S1 based on the URL input from the image URL selection unit 33 (S209). On the other hand, when the image URL selection unit 33 determines that the OGP image is not included, the image URL selection unit 33 does not collect the above image.

Next, the image URL selection unit 33 stores a set of an OGP title included in the OGP tag and an image collected based on the linked URL of the OGP image included in the OGP tag. It is stored in the image information D4 stored in 40 (S211). FIG. 10 is a diagram showing an example of image information D4. As shown in FIG. 10, the image information D4 is collected based on the title (entity name) acquired from the character string included in the OGP title of the OGP tag and the URL of the link destination of the OGP image of the OGP tag. It is stored in association with the image. In the image information D4, for example, the title "entity 1" and the image "image data 10" are stored in association with each other.

Next, the host selection unit 31 determines whether or not the processing for all the hosts selected as excellent hosts has been completed (S213). When the host selection unit 31 determines that the processing for all the hosts selected as the excellent hosts has not been completed, the host selection unit 31 selects one unprocessed host and repeats the processing after S203. On the other hand, when the host selection unit 31 determines that the processing for all the hosts selected as the excellent hosts has been completed, the host selection unit 31 ends the processing of this flowchart.

According to the embodiment described above, the data is collected by the first collection unit that collects the reference information belonging to the host to be evaluated and the corresponding content from the device accessible via the network, and the first collection unit. By providing an evaluation unit that evaluates the host to be evaluated based on the number of different image data associated with the content, the efficiency of data collection can be improved.

In the above embodiment, the example of collecting the entity representative image has been described, but the information of the official page (official content) of the entity may be collected. For example, the crawl server 1 may be provided with a collection unit (fourth collection unit, not shown) that collects information on the official page from the contents collected by the data collection unit 10. For example, if the HTML data of the collected content includes a link including a predetermined character string such as "official page" or "official page" in the link name, the crawl server 1 sets the URL of the link destination of this link. , Collect as the location information of the entity. Further, the crawl server 1 may further collect the information of the entity from the official page of the link destination.

Further, the crawl server 1 may collect the collected contents again when the collected contents collected in the past are updated. For example, the crawl server 1 may store the latest content in the storage unit 40 when there is a difference between the content collected in the past and the latest content. The presence or absence of the difference may be determined based on, for example, the difference in file size and the result of text comparison. Further, when the site map of the site to be crawled contains information on the update frequency, the crawl server 1 may recollect the contents based on the information on the update frequency.

Although the embodiments for carrying out the present invention have been described above using the embodiments, the present invention is not limited to these embodiments, and various modifications and substitutions are made without departing from the gist of the present invention. Can be added.

1 ... Crawl server, 10 ... Data collection unit, 20 ... Host evaluation unit, 21 ... Sampling unit, 23 ... Unique OGP rate calculation unit, 25 ... Japanese language judgment unit, 30 ... Image collection processing unit, 31 ... Host selection unit, 33 ... Image URL selection unit, 40 ... Storage unit

Claims (11)

The first collection unit that collects the reference information belonging to the host to be evaluated and the corresponding content from the device that can be accessed via the network.
An evaluation unit that evaluates the host to be evaluated based on the number of different image data associated with the content collected by the first collection unit .
A second collection unit that determines the host to be collected based on the evaluation result by the evaluation unit and collects the content corresponding to the reference information belonging to the determined host.
It is provided with a third collection unit that collects image data associated with the content collected by the second collection unit.
The evaluation unit evaluates the host to be evaluated based on the ratio of the total number of image data associated with the content collected by the first collection unit to the number of image data different from each other.
Data acquisition device. The evaluation unit evaluates the host to be evaluated so that the evaluation increases as the ratio of the number of image data different from each other to the total number of image data increases.
The data collection device according to claim 1. The evaluation unit of the host to be evaluated is based on the ratio of the total number of contents collected by the first collection unit to the number of predetermined language contents described in the predetermined language included in the contents. Make an evaluation,
The data collection device according to claim 1 or 2. The evaluation unit evaluates the host to be evaluated so that the evaluation increases as the ratio of the predetermined language content to the total number of contents collected by the first collection unit increases.
The data collection device according to claim 3. The evaluation unit groups the reference information based on the path structure of the reference information belonging to the host to be evaluated, and evaluates the host to be evaluated for each set of the reference information.
The data collection device according to any one of claims 1 to 4. The image data collected by the third collection unit is the image data described in the meta information included in the header unit of the content collected by the second collection unit.
The data collection device according to any one of claims 1 to 5. The third collecting unit collects the name information of the image data described in the meta information included in the header part of the content collected by the second collecting unit.
The data collection device according to claim 6. A fourth collection unit that collects information on the official content of the object of the image data based on a predetermined character string included in the content collected by the second collection unit is further provided.
The data collection device according to any one of claims 1 to 7. When the second collecting unit determines that the collected content has been updated, the second collecting unit collects the collected content again.
The data collection device according to any one of claims 1 to 8. The computer
From the device accessible via the network, the reference information belonging to the host to be evaluated and the corresponding first content are collected.
Based on the number of different image data associated with the collected first content, the evaluation target host is evaluated.
The host to be collected is determined based on the result of the evaluation, and the reference information belonging to the determined host and the corresponding second content are collected.
Collecting image data associated with the collected second content,
It ’s a data collection method .
Evaluating the host to be evaluated is based on the ratio of the total number of image data associated with the collected first content to the number of image data different from each other. Including making an evaluation,
Data collection method. On the computer
From the device accessible via the network, the reference information belonging to the host to be evaluated and the corresponding first content are collected.
The host to be evaluated is evaluated based on the number of different image data associated with the collected first content.
The host to be collected is determined based on the result of the evaluation, and the reference information belonging to the determined host and the corresponding second content are collected.
Collecting image data associated with the collected second content,
It ’s a program ,
To evaluate the host to be evaluated is to evaluate the host to be evaluated based on the ratio of the total number of image data associated with the collected first content to the number of image data different from each other. Including having the evaluation done,
program.

Images