JP7529006B2 - Program, search device, and search method - Google Patents

Description

The present invention relates to a program, a search device, and a search method for searching for a specific person from image data.

One application is to sell images taken at an event attended by many people using a web-based system. In such an application, a user must review a large number of images in which many people appear, one by one, and select images in which a specific person appears (hereafter, referred to as target images). However, when the number of images is large, such a selection process becomes time-consuming. Therefore, a method is required to efficiently and accurately extract target images from a large number of images.

When extracting images in which a specific person appears from a large number of images, if candidate images in which the target person appears (hereinafter referred to as candidate images) could be automatically extracted using facial recognition technology, it would reduce the user's efforts. However, when extracting candidate images using general facial recognition technology, there is a possibility that the target image will not be extracted. For example, in person searches using facial recognition technology, faces that are not facing forward are likely to be missed from the search results due to low recognition accuracy. In addition, the target person may not be reliably detected due to the influence of factors such as the brightness and shadows of the photo. For this reason, there is a demand for technology that can identify people using not only images but also some other information besides images.

Patent Document 1 discloses a method of using social connections obtained from various applications (hereinafter referred to as apps) to identify and rank people, in addition to visual similarity obtained from images. Patent Document 1 discloses an example of using social connection metrics obtained from communication apps, SNS (Social Networking Service) apps, calendar apps, collaborative apps, etc. to identify and rank people.

Patent document 2 discloses a method for predicting the strength of human relationships using attributes and classifications such as a person's date of birth, place of employment, educational background, hobbies, special skills, and clubs in addition to facial images, and for evaluating people with whom content should be shared.

Patent Document 3 discloses a device that identifies a search target person who appears in a specific target image. The device of Patent Document 3 estimates the movement pattern of a person who appears in multiple images based on the image capture times and image capture locations of those images, and estimates a location where the person is likely to be present based on the movement pattern of the person who appears in the images and the image capture time of the specific target image. Then, based on the estimation result of the location where the person is likely to be present, the device of Patent Document 3 estimates a person who is likely to have been present at the image capture location at the image capture time of the specific target image, and identifies the search target person by comparing the person who appears in the captured image with the search target person.

Patent No. 5557911 Patent No. 5477017 Patent No. 6139364

The method of Patent Document 1 uses social connection metrics obtained using apps such as SNS apps and calendar apps to facilitate the identification of people in images, and therefore requires collaboration with apps constructed as such external systems. Therefore, the method of Patent Document 1 has the problem that people cannot be identified in environments where collaboration with external systems is not possible. In addition, the method of Patent Document 1 uses personal information such as social connection metrics to identify people, and therefore is not sufficiently secure from the perspective of privacy.

The method in Patent Document 2 estimates the strength of human relationships using information about people's attributes and classifications, so there is a problem in that if the information registered in advance is incorrect or outdated, it may not be possible to accurately search for human relationships.

In the method of Patent Document 3, in order to identify the person to be searched who appears in the specific target image, it is necessary to extract candidates for the person to be searched for using an image taken before the specific target image. Therefore, if a person in an image taken before the specific target image is looking to the side or diagonally, it is not possible to estimate the movement pattern of that person, which is a problem in that it may not be possible to accurately extract candidates for the person to be searched for.

The object of the present invention is to solve the above-mentioned problems and provide a search device that can search for a specific person safely and with high accuracy based on human relationships without linking with an external system.

A program according to one aspect of the present invention causes a computer to execute a first input process in which first information indicating a first target, which is a person to be detected, or second information indicating a second target, which is a person different from the first target, is input; a process of outputting a candidate image of the first information or the second information when the first information or the second information is input as the first input process; a process of displaying a user interface for selecting the first information or the second information from among the candidate images and designating the candidate image selected via the user interface as either the first information or the second information; a second input process in which the first information and the second information are input; and a process of outputting image data including the first target and the second target from an image database when the first information and the second information are input as the second input process.

A search device according to one embodiment of the present invention includes an input unit that executes a first input process in which first information indicating a first target, which is a person to be detected, or second information indicating a second target, which is a person different from the first target, is input, an output unit that outputs a candidate image of the first information or the second information when the first information or the second information is input as the first input process, and a designation unit that displays a user interface for selecting the first information or the second information from among the candidate images and designates the candidate image selected via the user interface as either the first information or the second information, and the input unit executes a second input process in which the first information and the second information are input, and the output unit outputs image data including the first target and the second target from an image database when the first information and the second information are input as the second input process.

A search method according to one aspect of the present invention is a search method executed by a computer, which executes a first input process in which first information indicating a first target, which is a person to be detected, or second information indicating a second target, which is a person different from the first target, is input, and when the first information or the second information is input as the first input process, a candidate image of the first information or the second information is output, a user interface is displayed for selecting the first information or the second information from among the candidate images, the candidate image selected via the user interface is designated as either the first information or the second information, and a second input process is executed in which the first information and the second information are input, and when the first information and the second information are input as the second input process, image data including the first target and the second target is output from an image database.

The present invention makes it possible to provide a search device that can search for a specific person safely and with high accuracy based on human relationships without linking with an external system.

1 is a block diagram showing an example of a configuration of a search device according to a first embodiment of the present invention. 4 is a table illustrating an example of a search result table stored in a similarity DB of the search device according to the first embodiment of the present invention. FIG. 2 is a conceptual diagram showing an example of displaying output data of the search device according to the first embodiment of the present invention on a monitor. FIG. 11 is a conceptual diagram showing another example of displaying output data of the search device according to the first embodiment of the present invention on a monitor. 4 is a flowchart illustrating an operation of the search device according to the first embodiment of the present invention. FIG. 11 is a block diagram showing an example of a configuration of a search device according to a second embodiment of the present invention. FIG. 11 is a conceptual diagram showing an example of displaying candidate images output by a search device according to a second embodiment of the present invention on a monitor. FIG. 11 is a conceptual diagram showing another example of candidate images output by the search device according to the second embodiment of the present invention being displayed on the monitor. 10 is a flowchart illustrating an operation of a search device according to a second embodiment of the present invention. FIG. 2 is a block diagram showing an example of a hardware configuration for implementing a search device according to each embodiment of the present invention.

Below, the embodiments for carrying out the present invention are described with reference to the drawings. However, the embodiments described below have limitations that are technically preferable for carrying out the present invention, but the scope of the invention is not limited to the following. In addition, in all the drawings used to explain the embodiments below, the same reference numerals are used for similar parts unless there is a special reason. In addition, in the embodiments below, repeated explanations of similar configurations and operations may be omitted.

First Embodiment
First, a search device according to a first embodiment of the present invention will be described with reference to the drawings. The search device of this embodiment searches for image data in which a person to be searched for is included among at least one selection target image data, using images and similarities stored in a database (DB).

(composition)
Fig. 1 is a block diagram showing an example of the configuration of a search device 10 according to the present embodiment. As shown in Fig. 1, the search device 10 includes a face recognition unit 11, an image DB 12, a similarity calculation unit 13, a similarity DB 14, and a search result output unit 15. The search device 10 is connected to an input device 110 and an output device 150.

The search device 10 searches image data stored in the image DB 12 for image data containing the person to be searched for, using a target image 111 and a group of auxiliary images 112 input from the input device 110. The search device 10 generates output data (also called search results) including the searched image data, and outputs an output data group 151, which is a collection of the generated output data, to the output device 150.

The input device 110 is connected to the face recognition unit 11. A user inputs a target image 111, which is an image showing a person to be searched for, and an auxiliary image group 112, which is a collection of at least one auxiliary image that assists in searching for the target image 111, to the input device 110. The input device 110 outputs the input target image 111 and auxiliary image group 112 to the face recognition unit 11. The auxiliary image is an image showing a person related to the person to be searched for. For example, the input device 110 is realized by a terminal device having a monitor and peripheral devices such as a keyboard, mouse, and touch panel connected to the terminal device.

The face recognition unit 11 is connected to the input device 110. The face recognition unit 11 is also connected to the image DB 12 and the similarity calculation unit 13. A target image 111 and a group of auxiliary images 112 are input to the face recognition unit 11 from the input device 110.

When the target image 111 and the group of auxiliary images 112 are input, the face recognition unit 11 sequentially retrieves at least one image data to be selected from the image DB 12. The face recognition unit 11 detects a face from each piece of image data retrieved from the image DB 12. The face recognition unit 11 calculates the degree of match between the face detected from the image data and the target image 111 and the group of auxiliary images 112. The face recognition unit 11 outputs the calculation result (also called the evaluation result) of the degree of match between the face detected from the image data and the target image 111 and the group of auxiliary images 112 to the similarity calculation unit 13. It should be noted that general technology may be used for face detection and face recognition.

For example, a user uses the input device 110 to input a person image of a person to be searched for as a target image 111, and inputs at least one person image to assist the search as an auxiliary image to the search device 10. For example, assuming a site that sells photos of school field trips, an image of the user's child is input to the search device 10 as the target image 111, and images of the child's friends who were in the same group as the child are input as auxiliary images.

Image DB 12 (also called the first database) is connected to face recognition unit 11, similarity calculation unit 13, and search result output unit 15. Image DB 12 is a database in which image data to be selected is stored. For example, if search device 10 is used to sell photos taken on a school field trip, image data of images taken on the field trip is stored in image DB 12 as image data to be selected.

For example, image DB 12 stores image data to be selected and the shooting conditions (metadata) when the image data was captured in association with each other. For example, image DB 12 stores image data in the Exchangeable image file format (Exif). For example, the metadata of the image data to be selected includes the shooting date and time when the image data was captured, and GPS (Global Positioning System) information equivalent to the location information of the shooting location.

The similarity calculation unit 13 is connected to the face recognition unit 11, the image DB 12, and the similarity DB 14. The similarity calculation unit 13 acquires the face recognition evaluation result from the face recognition unit 11. When the similarity calculation unit 13 acquires the face recognition evaluation result, it uses the metadata of the image data stored in the image DB 12 to determine whether the search target person is included in the image data. The similarity calculation unit 13 stores the determination result in the similarity DB 14.

The similarity DB 14 (also called the second database) is connected to the similarity calculation unit 13 and the search result output unit 15. The similarity DB 14 is a database in which the results of the similarity determination by the similarity calculation unit 13 are stored.

The search result output unit 15 is connected to the image DB 12 and the similarity DB 14. The search result output unit 15 acquires the judgment result by the similarity calculation unit 13 from the similarity DB 14. The search result output unit 15 acquires image data from the image DB 12 based on the judgment result acquired from the similarity DB 14. The search result output unit 15 then acquires image data from the image DB 12 corresponding to the judgment result acquired from the similarity DB 14. The search result output unit 15 uses the image data acquired from the image DB 12 to generate output data (also called search results) in a format that can be processed by the output device 150. Note that the search result output unit 15 may include metadata of the image data corresponding to the judgment result in the output data. The search result output unit 15 outputs an output data group 151, which is a collection of the generated output data, to the output device 150.

The output device 150 is connected to the search result output unit 15. The output device 150 receives an output data group 151 from the search result output unit 15. The output device 150 is a device for presenting the search results of the search device 10 to the user. For example, the output device 150 is realized by a display device having a display. When the output device 150 is realized by a display device, the output device 150 displays the search results of the search device 10 on the monitor of the device itself. Also, for example, the output device 150 is realized by a printer. When the output device 150 is realized by a printer, the output device 150 prints the search results of the search device 10 on paper. Note that there are no particular limitations on the form of the method for outputting the search results of the search device 10, as long as the user can confirm the search results.

The above is an explanation of the configuration of the search device 10 of this embodiment. Note that the configuration of the search device 10 in FIG. 1 is an example, and the configuration of the search device 10 of this embodiment is not limited to this form.

〔judgment result〕
Next, an example of the determination results stored in the similarity DB 14 will be described. Fig. 2 shows a determination result table 140 as an example of the determination results stored in the similarity DB 14. In the determination result table 140, as an example, determination results regarding n pieces of image data (images 1 to n) to be selected are stored.

In the judgment result table 140, each column of the target image 111 and the auxiliary image group 112 (auxiliary image 1, auxiliary image 2, auxiliary image 3) is the calculation result of the degree of match between the target image 111 and the auxiliary image group 112 and each image data.

In the determination result table 140, the time information column indicates information about the shooting time of the image data included in the metadata of the image data to be selected. Note that in the determination result table 140, T1, T1', and T1'' are approximately the same time, and the other times are far apart from each other.

In the determination result table 140, each column of location information indicates information about the shooting location (geotag, etc.) of the image data to be selected, which is included in the metadata of the image data. In the determination result table 140, P1 and P1' are approximately in the same location, and the other locations are far from each other.

In the determination result table 140, the column for similarity determination indicates the result of the determination as to whether the search target person appears in each image, based on the degree of match and metadata. The similarity determination indicates the likelihood that the search target person appears in the image data. The determination result table 140 shows an example in which the determination result is determined in three levels: "high," "medium," and "low."

Referring to the judgment result table 140, image 1 has a matching degree of 90% (percent) with the target image 111. It is judged that image 1 is highly likely to include the person being searched for, and the similarity judgment is set to "high."

Referring to the judgment result table 140, image 2 has a matching rate of 70% with the target image 111. Although the matching rate of image 2 is relatively high at 70%, the shooting locations of image 2 are different even though the shooting time T1 of image 1 and the shooting time T1' of image 2 are almost the same time, so it is judged that there is a low possibility that the person being searched for is included, and the similarity judgment is set to "low."

Referring to the judgment result table 140, image 3 has a similarity of 50% with the target image 111. Although image 3 has a low degree of match of 50%, it was taken at approximately the same time and in approximately the same position as image 1. In addition, people with high similarities to auxiliary images 1 to 3 are also captured at the same time. For example, in the case of an image taken on an elementary school field trip, if the group of children being searched for is active at that position at that time, there is a high possibility that they will appear together in the same image. For this reason, the similarity judgment for the image data of image 3 is set to "medium."

Referring to the judgment result table 140, image n has a low degree of match with auxiliary images 2 and 3, but is determined to have been taken at a different scene because both the time and the position are far from images 1 to 3. Therefore, for image n, the degree of match with the auxiliary image does not affect the similarity judgment, and the degree of match with the target image is high at 80%, so the similarity judgment is set to "high."

The search result output unit 15 refers to the similarity judgments stored in the similarity DB 14 and extracts image data with similarity judgments of "high" and "medium" from the image DB 12. The search result output unit 15 uses the image data extracted from the image DB 12 to generate output data that can be displayed on the monitor of the output device 150. The search result output unit 15 outputs the generated output data to the output device 150.

For example, the search result output unit 15 generates thumbnails of images with a similarity judgment of "high" and "medium," and output data in an html format that makes it possible to display full-size images from the thumbnails. Note that the full-size images referenced from the thumbnails do not have to be actual full-size images, but may be enlarged images that can be determined by the user.

The output device 150 acquires output data from the search result output unit 15. The output device 150 displays the search results of the search device 10 on the monitor of the device itself. For example, the output device 150 displays images with similarity judgments of "high" and "medium" in different display areas. A user looking at the monitor of the output device 150 can check the images according to the similarity judgment, and is relieved of the burden of checking all images to be selected. If the similarity judgment classification is subdivided into "high" and "medium", the user may feel a new burden in distinguishing between images with a similarity judgment of "high" and images with a similarity judgment of "medium". Therefore, the similarity judgment of the images to be checked by the user may be set to one level rather than two levels of "high" and "medium". On the other hand, there may be situations in which it is better to subdivide the similarity judgment classification in more detail. In that case, the classification of the similarity judgment can be set in more detail. In other words, the classification of the similarity judgment can be adjusted to an appropriate classification depending on the application.

Figures 3 and 4 show an example of a user interface 155 (also called a first user interface) that allows the user to select searched image data according to the similarity judgment, displayed on the monitor of the output device 150. Figures 3 and 4 show an example of a user interface 155 that allows the user to select image data that the user wishes to purchase, divided into a group of image data with a "high" similarity judgment and a group of image data with a "medium" similarity judgment.

As an example, the user interface 155 in FIG. 3 displays an enlarged view of image data with a similarity judgment of "high." In FIG. 3, as an example, seven pieces of image data with a similarity judgment of "high" are selected, and no image data with a similarity judgment of "medium" is selected. If image data can be purchased for 100 yen each, the total number of selected images is seven, and the purchase price is displayed as 700 yen.

As an example, the user interface 155 in FIG. 4 displays an enlarged view of image data with a similarity judgment of "medium." In FIG. 4, as an example, three pieces of image data with a similarity judgment of "medium" have been selected, and seven pieces of image data with a similarity judgment of "high" have already been selected. If image data can be purchased for 100 yen each, the total number of selected images is 10, so the purchase price is displayed as 1,000 yen.

In the above Figures 2 to 4, an example is shown in which three levels of similarity judgment, "high," "medium," and "low," are stored in the similarity DB 14 for all image data. However, the similarity judgment may be subdivided by percentages or the like instead of a gradual judgment. Subdividing the similarity judgment allows the search accuracy to be set more precisely.

For example, the search device 10 can be configured as a local system on a terminal device operated by a user. As another example, the input of the target image 111 and the auxiliary image group 112 and the display of the search results can be performed on a terminal device such as a personal computer or smartphone operated by the user. The search device 10 may also be configured as a computer resource connected to the terminal device operated by the user via the Internet.

(motion)
Next, the operation of the search device 10 of this embodiment will be described with reference to the drawings. Fig. 5 is a flowchart for explaining the operation of the search device 10. In the explanation following the flowchart of Fig. 5, the search device 10 will be the subject of the operation.

In FIG. 5, first, the search device 10 acquires the target image 111 and the auxiliary image group 112 input from the input device 110 (step S11).

Next, the search device 10 retrieves the image data to be selected from the image DB 12 (step S12). Note that when multiple pieces of image data are to be processed together, the multiple pieces of image data may be retrieved from the image DB 12 all at once.

Next, the search device 10 detects faces from the image data retrieved from the image DB 12 (step S13). Note that, when multiple selection items are processed together, faces may be detected from the multiple image data retrieved from the image DB 12 all at once.

Next, the search device 10 calculates the degree of match between the face detected from the image data to be selected and the target image 111 and the auxiliary image group 112 (step S14). Note that when multiple images are processed together, the degree of match between the face detected from the multiple image data and the target image 111 and the auxiliary image group 112 may be calculated all at once.

Next, the search device 10 uses the calculated degree of match and the metadata of the image data stored in the image DB 12 to determine whether the search target person is included in the image data (step S15). Note that when determining whether the search target person is included in the image data to be selected, if the metadata of the image data is not used, the search device 10 uses the calculated degree of match to make the determination.

Next, the search device 10 stores the result of the determination as to whether the person to be searched is included in the image data of the selected object in the similarity DB 14 (step S16).

If image data to be searched remains (Yes in step S17), the process returns to step S13. On the other hand, if image data to be selected does not remain (No in step S17), image data is obtained from image DB 12 based on the determination result stored in similarity DB 14 (step S18). Then, the search device 10 generates output data using the image data obtained from image DB 12, and outputs the generated output data to the output device 150 (step S19).

The above is an explanation of the operation of the search device 10 according to the flowchart in FIG. 5. Note that the operation of the search device 10 according to the flowchart in FIG. 5 is an example, and the operation of the search device 10 of this embodiment is not limited to this exact procedure.

As described above, the search device of this embodiment includes a first database, a face recognition unit, a similarity calculation unit, a second database, and a search result output unit.

The first database stores image data of at least one selection target. The face recognition unit detects a face from the image data of the selection target stored in the first database. When a target image showing the person to be searched for and a group of auxiliary images including at least one auxiliary image that assists in the search for the person to be searched for are specified, the face recognition unit evaluates whether the face detected from the image data of the selection target is included in the target image and the group of auxiliary images. The similarity calculation unit uses the evaluation result by the face recognition unit to determine whether the person to be searched for is included in the image data. The second database stores the determination result by the similarity calculation unit. The search result output unit acquires image data determined to include the person to be searched for from the first database based on the determination result stored in the second database, and generates and outputs output data including the acquired image data.

As one aspect of this embodiment, an image showing a person related to the person being searched for is input to the face recognition unit as an auxiliary image.

As one aspect of this embodiment, the face recognition unit calculates the degree of match between a face detected from the image data and the face of a person included in the target image and the group of auxiliary images. The similarity calculation unit determines whether the person to be searched for is included in the image data based on the degree of match calculated for the target image and the group of auxiliary images.

As one aspect of this embodiment, the first database stores metadata of the image data to be selected. The similarity calculation unit acquires the metadata of the image data to be selected from the first database. The similarity calculation unit determines whether the person to be searched for is included in the image data based on the acquired metadata of the image data to be selected and the degree of match calculated for the target image and the group of auxiliary images.

As one aspect of this embodiment, the similarity calculation unit assigns a similarity judgment to the image data, which indicates the likelihood that the person being searched for is included, based on the metadata of the image data to be selected and the degree of match calculated for the target image and the group of auxiliary images.

As one aspect of this embodiment, the search result output unit is connected to an output device having a monitor. The search result output unit outputs output data to the output device. The output device displays on the monitor a first user interface for selecting image data included in the output data.

The search device of this embodiment is input with a target image, which is an image showing the person to be searched for, and a group of auxiliary images including an auxiliary image, which is an image of at least one person that assists in searching for the target image. For example, a facial image of a user's child is input to the search device as the target image, and a facial image of the child's friend is input as the auxiliary image. When the target image and the group of auxiliary images are input, the search device of this embodiment performs face recognition on the image data to be searched for stored in the first database. Using the evaluation result of the face recognition and metadata of the image data of the image to be searched for, the search device of this embodiment determines whether the image data includes the person to be searched for, based on a similarity evaluation.

For example, in a case where photos of children's school trips are sold through a web-based system, there is a high possibility that the child who is the search target person will be photographed together with related people such as close friends or group members. According to this embodiment, even for image data in which the search target person appears but the facial recognition match is low, it is possible to verify whether the search target person appears based on the facial recognition match of related people.

In addition, according to this embodiment, it is possible to verify whether the person to be searched for appears in image data, including the meta information of the image data (time information and location information). For example, the person to be searched for will not appear in multiple image data taken at different locations at the same time. Therefore, it is possible to exclude from the search results any of the multiple image data taken at different locations at the same time.

In this embodiment, the judgment results from each of these evaluations are stored in a second database. According to this embodiment, images that include the person being searched for are selected from the image database and output according to the judgment results stored in the second database, allowing the user to select the images to be confirmed.

When extracting image data that includes a person to be searched for from a large number of image data, the search device of this embodiment also uses a determination result indicating the presence or absence of a person related to the person to be searched for, as well as metadata such as time information and location information. Therefore, the search device of this embodiment can improve search accuracy compared to searches that use only facial recognition of the person to be searched for.

For example, when purchasing an image taken at a school event in which a user's child participated, the user will have a parental instinct to want to purchase the image even if the child's face is not facing forward. In this embodiment, in addition to the facial image recognition results of the search target person, search results for related people who are likely to be in the same situation as the search target person, and behavioral history (time information and location information) are also referenced. Therefore, according to this embodiment, it is possible to highly accurately eliminate the possibility of overlooking an image in which the search target person appears, and provide more accurate search results.

In addition, in a social climate where privacy protection regulations are becoming stricter, there are limitations to improving search accuracy by using a social relationship diagram (social graph) that uses external systems such as social network services and shared calendar apps. According to this embodiment, it is possible to improve the search accuracy of relationships in a closed environment without the need for linking with external systems, which is safe from the perspective of privacy protection.

In other words, the search device of this embodiment can search for a specific person safely and with high accuracy based on human relationships without linking with an external system.

Second Embodiment
Next, a search device according to a second embodiment of the present invention will be described with reference to the drawings. This embodiment differs from the first embodiment in that it generates candidate images for a target image or an auxiliary image (also called candidate images) from image data stored in an image DB and proposes the generated candidate images to a user.

Fig. 6 is a block diagram showing an example of the configuration of the search device 20 of this embodiment. As shown in Fig. 6, the search device 20 includes a face recognition unit 21, an image DB 22, a similarity calculation unit 23, a similarity DB 24, a search result output unit 25, and a candidate image suggestion unit 26. The search device 20 is connected to an input device 210 and an output device 250.

The search device 20 detects a face from at least one image data of a selection target stored in the image DB 22. The search device 20 outputs an image including the detected face to the input device 210 as a candidate image for the target image or auxiliary image. The user selects a target image or auxiliary image from the candidate images displayed on the input device 210, and specifies the selected target image or auxiliary image. The search device 20 searches for image data including the person to be searched from among the image data stored in the image DB 22, using the target image 211 and auxiliary image group 212 specified by the user via the input device 210. The search device 20 generates output data including the searched image data, and outputs an output data group 251, which is a collection of the generated output data, to the output device 250.

The input device 210 is connected to the face recognition unit 21 and the candidate image suggestion unit 26. Candidate images for the target image and auxiliary image are input to the input device 210 from the candidate image suggestion unit 26. The input device 210 displays on a monitor a user interface for selecting the target image 211 and auxiliary image from the input candidate images. The input device 210 designates the candidate image selected via the user interface as the target image 211 or the auxiliary image group 212. At this time, the input device 210 outputs information indicating which candidate image has been designated as the target image 211 or the auxiliary image group 212 to the search device 20. The input device 210 may also be configured to output image data designated as the target image 211 and the auxiliary image group 212 to the search device 20.

Figure 7 is a conceptual diagram showing an example of a user interface 215 (also called a second user interface) for presenting candidate images to the user on the monitor of the input device 210. A plurality of candidate images are displayed on the user interface 215 in Figure 7. A check box for selecting a target image or an auxiliary image is displayed on the user interface 215 in association with each candidate image.

Figure 8 shows an example of the state after a user who has checked the user interface 215 in Figure 7 has selected a target image or auxiliary image. In Figure 8, as an example, candidate image No. 2 is selected as the target image 211, and candidate images No. 1, 3, and 7 are selected as auxiliary images. The input device 210 designates the candidate image selected in the user interface 215 as the target image 211 or an auxiliary image that constitutes the group of auxiliary images 212.

The candidate image suggestion unit 26 acquires images including a face detected from the image data to be selected from the face recognition unit 21. The candidate image suggestion unit 26 uses the acquired images to generate candidate images for the search target image and the auxiliary image. The candidate image suggestion unit 26 outputs the generated candidate images to the input device 210.

The face recognition unit 21 is connected to the input device 210. The face recognition unit 21 is also connected to the image DB 22, the similarity calculation unit 23, and the candidate image suggestion unit 26.

The face recognition unit 21 acquires image data of at least one selection target stored in the image DB 22, and detects a face from the acquired image data. The face recognition unit 21 outputs an image including a face detected from the image data of the selection target to the candidate image suggestion unit 26. The face recognition unit 21 may store the image including a face detected from the image data of the selection target in the image DB 22, or may delete it temporarily.

Furthermore, information indicating which candidate images have been designated as the target image 211 and the auxiliary image group 212 is input from the input device 210 to the face recognition unit 21. Upon receiving the designation of the target image 211 and the auxiliary image group 212, the face recognition unit 21 calculates the degree of match between the face detected from the image data to be selected and the target image 211 and the auxiliary image group 212. The face recognition unit 21 outputs the calculation result (also called the evaluation result) of the degree of match between the face detected from the image data to be selected and the target image 211 and the auxiliary image group 212 to the similarity calculation unit 23.

The image DB 22 (also called the first database) is connected to the face recognition unit 21, the similarity calculation unit 23, and the search result output unit 25. The image DB 22 is a database in which image data to be selected is stored.

The similarity calculation unit 23 is connected to the face recognition unit 21, the image DB 22, and the similarity DB 24. The similarity calculation unit 23 acquires the face recognition evaluation result from the face recognition unit 21. When the similarity calculation unit 23 acquires the face recognition evaluation result, it uses the metadata of the image data stored in the image DB 22 to determine whether the search target person is included in the image data. The similarity calculation unit 23 stores the determination result in the similarity DB 24.

The similarity DB 24 (also called the second database) is connected to the similarity calculation unit 23 and the search result output unit 25. The similarity DB 24 is a database in which the results of the similarity determination by the similarity calculation unit 23 are stored.

The search result output unit 25 is connected to the image DB 22 and the similarity DB 24. The search result output unit 25 acquires the judgment result by the similarity calculation unit 23 from the similarity DB 24. The search result output unit 25 acquires image data from the image DB 22 based on the judgment result acquired from the similarity DB 24. Then, the search result output unit 25 acquires image data from the image DB 22 corresponding to the judgment result acquired from the similarity DB 24. The search result output unit 25 uses the image data acquired from the image DB 22 to generate output data (also called search results) in a format that can be processed by the output device 230. Note that the search result output unit 25 may include metadata of the image data corresponding to the judgment result in the output data. The search result output unit 25 outputs an output data group 231, which is a collection of the generated output data, to the output device 230.

The output device 230 is connected to the search result output unit 25. The output device 230 receives an output data group 231 from the search result output unit 25. The output device 230 is a device for presenting the search results of the search device 20 to the user. There are no particular limitations on the form of the method for outputting the search results of the search device 20, as long as the user can confirm the search results.

The above is an explanation of the configuration of the search device 20 of this embodiment. Note that the configuration of the search device 20 in FIG. 6 is an example, and the configuration of the search device 20 of this embodiment is not limited to this form.

(motion)
Next, the operation of the search device 20 of this embodiment will be described with reference to the drawings. Fig. 9 is a flowchart for explaining the operation of the search device 20. In the explanation following the flowchart of Fig. 9, the search device 20 will be the subject of the operation.

In FIG. 9, first, the search device 20 generates candidate images from image data to be selected stored in the image DB 22 (step S21).

Next, the search device 20 outputs the generated candidate images to the input device 210 (step S22).

Next, the search device 20 acquires information from the input device 210 indicating which candidate images have been designated as the target image 211 and the auxiliary image group 212 (step S23).

Next, the search device 20 calculates the degree of match between the face detected from the image data of the selection target and the target image 111 and the auxiliary image group 112 (step S24).

Next, the search device 20 uses the calculated degree of match and the metadata of the image data stored in the image DB 22 to determine whether the search target person is included in the image data (step S25). Note that when determining whether the search target person is included in the image data to be selected, if the metadata of the image data is not used, the search device 20 uses the calculated degree of match to make the determination.

Next, the search device 20 stores the result of the determination as to whether the person being searched for is included in the image data of the selected object in the similarity DB 14 (step S26).

If image data to be searched remains (Yes in step S27), the process returns to step S24. On the other hand, if image data to be selected does not remain (No in step S27), image data is obtained from the image DB 22 based on the determination result stored in the similarity DB 24 (step S28). The search device 20 then generates output data using the image data obtained from the image DB 22, and outputs the generated output data to the output device 230 (step S29).

The above is an explanation of the operation of the search device 20 according to the flowchart in FIG. 9. Note that the operation of the search device 20 according to the flowchart in FIG. 9 is an example, and the operation of the search device 20 of this embodiment is not limited to this procedure as it is.

As described above, the search device of this embodiment includes a first database, a face recognition unit, a similarity calculation unit, a second database, and a search result output unit, as well as a candidate image suggestion unit.

The face recognition unit acquires at least one image data stored in the first database, detects a face from the acquired image data, and outputs image data including the detected face to the candidate image suggestion unit. The candidate image suggestion unit generates candidate images for the target image and auxiliary image using an image including the face detected by the face recognition unit, and outputs the generated candidate images.

As one form of this embodiment, the candidate image suggestion unit is connected to an input device having a monitor and outputs candidate images to the input device. The input device displays on the monitor a second user interface for selecting a target image and an auxiliary image from among the candidate images. The input device designates each of the candidate images selected via the second user interface as either a target image or an auxiliary image, and outputs information regarding the designated target image and auxiliary image to the face recognition unit.

In general, there are many cases where a user has image data of their child at hand, but does not have suitable image data of their friend to use as an auxiliary image. In this embodiment, candidate images generated using face images detected from image data of the selection target stored in the first database are presented to the user, and the user is prompted to select a target image and an auxiliary image from each of the candidate images. As a result, the user is saved the trouble of procuring auxiliary images that are not at hand.

For example, the search device of this embodiment retrieves multiple person images assumed to be of different people from the first database, and outputs and displays thumbnails and full-size images of those person images as links in HTML format data. The user looks at the images displayed on the monitor and selects the person to be searched for or a person related to that person by clicking or tapping on the photo. At this time, if there are many target people, the candidate images are displayed on multiple pages. For example, if person images determined to be closely related based on similarity calculations are preferentially displayed, it is possible to reduce the operational burden on the user.

The method of each embodiment of the present invention is effective when searching for images of people, where the interpersonal relationships are relatively closed but the search targets are large, such as when photos taken at school events such as field trips and school performances are sold on a web-based system.

For example, it is very time-consuming to search for a target image that contains a specific person while referencing a large number of image data that contain people other than the target person. Using face recognition technology can reduce the amount of work compared to searching for the target image by referencing all image data one by one. However, the detection accuracy of current face recognition technology is not high enough to accurately detect the target image when the specific person is looking to the side or diagonally.

For adults with accumulated personal relationships and behavioral histories, the target image can be searched for using the personal relationships and behavioral histories extracted from social network services and shared calendar apps, improving search accuracy. However, when searching for images of children who do not have accumulated personal relationships or behavioral histories, using social network services and shared calendar apps does not necessarily improve the search accuracy for the target image. In addition, due to the social climate in which privacy protection regulations are becoming stricter, it is becoming more difficult to publicly use information from social network services and shared calendar apps.

In addition, according to the method of each embodiment of the present invention, it is possible to search for target images using the face of the person to be searched for as well as the faces of people related to the person to be searched for, without using information from external systems such as social network services or shared calendar apps.

In other words, the search device of this embodiment can search for a specific person safely and with high accuracy based on human relationships without linking with an external system.

(hardware)
Here, a hardware configuration for executing the processing of the search device according to each embodiment of the present invention will be described by taking as an example an information processing device 90 in Fig. 10. Note that the information processing device 90 in Fig. 10 is an example of a configuration for executing the processing of the search device according to each embodiment, and does not limit the scope of the present invention.

As shown in FIG. 10, the information processing device 90 includes a processor 91, a main memory device 92, an auxiliary memory device 93, an input/output interface 95, and a communication interface 96. In FIG. 10, the interface is abbreviated as I/F (Interface). The processor 91, the main memory device 92, the auxiliary memory device 93, the input/output interface 95, and the communication interface 96 are connected to each other via a bus 99 so as to be able to communicate data with each other. In addition, the processor 91, the main memory device 92, the auxiliary memory device 93, and the input/output interface 95 are connected to a network such as the Internet or an intranet via the communication interface 96.

The processor 91 deploys a program stored in the auxiliary storage device 93 or the like in the main storage device 92, and executes the deployed program. In this embodiment, a software program installed in the information processing device 90 may be used. The processor 91 executes processing by the search device according to this embodiment.

The main memory device 92 has an area in which the program is expanded. The main memory device 92 may be, for example, a volatile memory such as a dynamic random access memory (DRAM). A non-volatile memory such as a magnetoresistive random access memory (MRAM) may also be configured or added as the main memory device 92.

The auxiliary storage device 93 stores various data. The auxiliary storage device 93 is configured with a local disk such as a hard disk or flash memory. Note that it is also possible to configure the main storage device 92 to store various data, and omit the auxiliary storage device 93.

The input/output interface 95 is an interface for connecting the information processing device 90 to peripheral devices. The communication interface 96 is an interface for connecting to external systems and devices through a network such as the Internet or an intranet based on standards and specifications. The input/output interface 95 and the communication interface 96 may be a common interface for connecting to external devices.

The information processing device 90 may be configured to connect input devices such as a keyboard, mouse, or touch panel as necessary. These input devices are used to input information and settings. When a touch panel is used as an input device, the display screen of the display device may be configured to also function as an interface for the input device. Data communication between the processor 91 and the input devices may be mediated by the input/output interface 95.

The information processing device 90 may also be equipped with a display device for displaying information. When a display device is equipped, the information processing device 90 is preferably equipped with a display control device (not shown) for controlling the display of the display device. The display device may be connected to the information processing device 90 via the input/output interface 95.

The information processing device 90 may also be equipped with a disk drive as necessary. The disk drive is connected to the bus 99. The disk drive mediates between the processor 91 and a recording medium (program recording medium) (not shown) for reading data and programs from the recording medium, writing the processing results of the information processing device 90 to the recording medium, and the like. The recording medium may be, for example, an optical recording medium such as a CD (Compact Disc) or a DVD (Digital Versatile Disc). The recording medium may also be a semiconductor recording medium such as a USB (Universal Serial Bus) memory or an SD (Secure Digital) card, a magnetic recording medium such as a flexible disk, or other recording medium.

The above is an example of a hardware configuration for enabling a search device according to each embodiment of the present invention. Note that the hardware configuration in FIG. 10 is an example of a hardware configuration for executing the computational processing of the search device according to each embodiment, and does not limit the scope of the present invention. In addition, a program that causes a computer to execute processing related to the search device according to each embodiment is also included in the scope of the present invention. Furthermore, a program recording medium on which a program according to each embodiment is recorded is also included in the scope of the present invention.

The components of the search device of each embodiment can be combined in any manner. In addition, the components of the search device of each embodiment can be realized by software or by a circuit.

The present invention has been described above with reference to the embodiments, but the present invention is not limited to the above embodiments. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present invention within the scope of the present invention.

10, 20 Search device 11, 21 Face recognition unit 12, 22 Image DB
13, 23 Similarity calculation unit 14, 24 Similarity DB
15, 25 Search result output unit 26 Candidate image suggestion unit 110, 210 Input device 150, 250 Output device

Claims (10)

A process of displaying a user interface for allowing a user to select second information including a second object that is a related person of a first object that is a person to be detected;
A process of acquiring a candidate image selected via the user interface as the second information indicating the second target;
and outputting image data including the first object from an image database based on first information indicating the first object and the second information. A process of acquiring an image showing a related person included in the selected candidate image as the second information ;
and outputting image data including the first object and the second object from an image database based on the first information and the second information . A process of calculating a degree of coincidence between a face detected from the image data and a face of a person included in the first information and the second information;
The program according to claim 1 or 2, which causes a computer to execute a process of determining whether or not the first object is included in the image data based on the degree of match with the first information and the degree of match with the second information. obtaining metadata from the image data;
The program according to any one of claims 1 to 3, which causes a computer to execute a process of outputting the image data including the first object from an image database based on the first information, the second information, and the metadata. The program according to any one of claims 1 to 4, which causes a computer to execute a process of outputting the candidate image using the image data. a process of connecting to an input device having a monitor and outputting the candidate image to the input device;
The program according to any one of claims 1 to 5, which causes a computer to execute a process of: displaying on the monitor a second user interface for selecting the first information and the second information from the candidate images; designating each of the candidate images selected via the second user interface as either the first information or the second information; and outputting information relating to the designated first information and second information. The program according to any one of claims 1 to 6, which causes a computer to execute a process of classifying the image data showing a subject in the second information and the first subject into multiple levels of priority based on the similarity between the image data and the first information and the similarity between the image data and the second information indicating a person related to the first subject. The program according to claim 7, which causes a computer to execute a process of outputting the image data showing the first target and the subject so as to display them according to the priority order. a designation means for displaying a user interface for allowing a user to select second information including a second object that is a related person of a first object that is a person to be detected;
an input means for acquiring a candidate image selected via the user interface as the second information indicating the second target;
and an output unit that outputs image data including the first object from an image database based on first information and the second information indicating the first object. 1. A computer-implemented search method, comprising:
displaying a user interface for allowing the user to select second information including a second object that is a related person of the first object that is a person to be detected;
acquiring a candidate image selected via the user interface as the second information indicating the second target;
A search method for outputting image data including the first object from an image database based on first information and the second information indicating the first object.

Images