JP7521339B2 - Determination device and program - Google Patents

Description

The present invention relates to a determination device and a program for determining whether the background of a facial image is appropriate.

When applying for a My Number card or a driver's license, a facial image is required. Patent Document 1 describes a system in which an application is made using a facial image taken with a passport photo machine.

Applications for My Number cards can be made online using a photo of one's face taken with a mobile device with a camera such as a smartphone. Applications for driver's licenses can also be made offline at a reception desk using a photo of one's face taken in advance with a mobile device with a camera.

Patent No. 6064569

Facial images used for these applications often need to have a solid, monochrome background. However, when a facial image is taken of oneself using a mobile device with a camera, people or objects may be caught in the background, making it undesirable to use such a facial image for an application. Therefore, there was a need for a mechanism to determine whether the background of a facial image is appropriate.

The present invention has been made in consideration of the above problems, and aims to provide a determination device etc. that determines whether the background of a facial image is appropriate.

The first invention for solving the above-mentioned problems is a judgment device that creates a histogram of brightness and hue of pixels in the background area of a facial image for application, and if there is a single histogram peak in both the brightness and hue histograms, it regards the background of the facial image as a solid color and judges that the background area is appropriate.

By using the determination device of the present invention, it is possible to determine whether the background of a facial image for application is appropriate. In other words, if there is a single peak in the histogram of the brightness and hue of pixels in the background area, the background of the facial image can be considered to be a solid color and is therefore appropriate.

The determination device of the first invention preferably includes a receiving means for receiving the face image from a user terminal, or the determination device of the first invention may be a user terminal.
In the former case, in an online application system, etc., a judgment device can judge a face image sent from a user terminal at the time of application, and automatically detect a face image with an inappropriate background and reject an application with that face image at the time of application, thereby reducing the administrative burden on the department that accepts applications. In the latter case, a pre-check of the face image for application can be performed on the user terminal, and the user will not use a face image with an inappropriate background for application. Therefore, there is no delay in the application procedure due to reapplication, etc.

It is preferable that the determination device of the first invention further comprises a notification means for notifying a user, when the background area is not appropriate, that the background of the face image is not appropriate.
When the background area is inappropriate, it is possible to process the image by filling the background area with a single color, but processing the image around the hair is difficult and the facial image is likely to become unnatural. Also, since the user intends to use the facial image they themselves took for the application, it is easier to obtain a satisfactory facial image including the background by notifying the user that the background of the facial image is inappropriate and encouraging the user to take the photo again.

The histogram is preferably generated for pixels excluding black and white pixels.
This makes it possible to improve the accuracy of the determination.

The second invention is a program for causing a computer to function as a judgment device having judgment means for creating histograms of brightness and hue of pixels in the background area of a facial image of a user for application, and if there is a single histogram peak in both the brightness and hue histograms, determining that the background of the facial image is a solid color and that the background area is appropriate.
A second aspect of the present invention is a program for causing a computer to function as the determination device of the first aspect of the present invention.

The present invention can provide a determination device that determines whether the background of a face image is appropriate.

FIG. 1 shows an online application system 1. FIG. 2 is a diagram showing a hardware configuration of a determination device 3. FIG. 2 is a diagram showing the hardware configuration of a user terminal 5. FIG. 2 is a diagram showing the functional configuration of a determination device 3. 1 is a flowchart showing the flow of a determination method. 2A and 2B are diagrams for explaining extraction of a face image 10 and a background region 13. FIG. Example of notification to the user. FIG. 2 is a diagram showing the functional configuration of a user terminal 5. 1 is a flowchart showing the flow of a determination method.

The following describes in detail a preferred embodiment of the present invention with reference to the drawings.

[First embodiment]
(1. Online Application System 1)
1 is a diagram showing an online application system 1 having a determination device 3 according to an embodiment of the present invention. The online application system 1 accepts various applications online using face images, and is configured by connecting the determination device 3 and a user terminal 5 via a network so that they can communicate with each other.

In the online application system 1, a facial image of the user is sent from the user terminal 5 to the determination device 3 along with other information required for the application, and the determination device 3 determines whether the background of the facial image is suitable for the application.

Figure 2 is a diagram showing the hardware configuration of the determination device 3. The determination device 3 can be realized by a computer configured by connecting a control unit 31, a storage unit 32, a communication unit 33, etc. via a bus or the like. However, the present invention is not limited to this, and various configurations can be used as appropriate.

The control unit 31 is composed of a CPU, ROM, RAM, etc. The CPU loads programs related to the processing of the determination device 3 stored in storage media such as the memory unit 32 and ROM into a work area on the RAM and executes them. The ROM is a non-volatile memory, and permanently stores programs such as the boot program and BIOS, as well as data. The RAM is a volatile memory, and temporarily stores programs and data loaded from the memory unit 32, ROM, etc., and also has a work area that the control unit 31 uses to perform various processes.

The storage unit 32 is a hard disk drive, solid state drive, flash memory, etc., and stores the programs executed by the determination device 3 during the processing described below, data necessary for program execution, the OS, etc. These programs and data are read and executed by the control unit 31 as necessary.

The communication unit 33 is a communication interface that mediates communication via a network, and communicates with the user terminal 5.

The user terminal 5 is a camera-equipped terminal carried by the user, which takes a picture of the user and transmits the face image and the like to the determination device 3. As the user terminal 5, for example, a mobile terminal such as a smartphone or a tablet terminal is used.

Figure 3 is a diagram showing the hardware configuration of the user terminal 5. As shown in Figure 3, the user terminal 5 can be realized by a computer configured by connecting a control unit 51, a memory unit 52, a display unit 53, an input unit 54, a communication unit 55, a camera 56, an audio input/output unit 57, etc., via a bus or the like. However, the present invention is not limited to this, and various configurations can be used as appropriate.

The control unit 51, the memory unit 52, and the communication unit 55 are substantially the same as the control unit 31, the memory unit 32, and the communication unit 33 described above. The display unit 53 has a display device such as a liquid crystal panel, and a touch panel is provided as the input unit 54 for inputting data to the user terminal 5. The audio input/output unit 57 includes a microphone and a speaker used for audio input and output.

Camera 56 is an area camera that is composed of an optical lens, an imaging element such as a CCD (Charge Coupled Device) or CMOS (Complementary Metal-Oxide Semiconductor), an A/D (Analog/Digital) converter, etc. Camera 56 photoelectrically converts the subject image input through the optical lens using the imaging element to generate an analog image signal. The A/D converter then converts the analog image signal into digital image data.

Figure 4 is a diagram showing the functional configuration of the determination device 3. The determination device 3 has a receiving means 301, a determination means 302, a notification means 303, etc.

The receiving means 301 is a means by which the control unit 31 of the determination device 3 receives a facial image of the user for application from the user terminal 5 via the communication unit 33.

The determination means 302 determines whether the background of the face image received by the control unit 31 of the determination device 3 from the user terminal 5 is appropriate. Details of this determination will be described later.

The notification means 303 is a means for notifying the user that the background of the facial image is inappropriate when the background of the facial image is inappropriate, by the control unit 31 of the determination device 3.

(2. Judgment Method)
Fig. 5 is a flowchart showing the flow of the face image determination method executed in the online application system 1. S1 to S2 and S7 to S8 in Fig. 5 are processes executed by the control unit 51 of the user terminal 5 by controlling each unit of the user terminal 5, and S3 to S6 are processes executed by the control unit 31 of the determination device 3 by controlling each unit of the determination device 3.

In this embodiment, first, the user terminal 5 captures a facial image of the user with the camera 56 in response to the user's operation (S1). The captured facial image is recorded in the memory unit 52 of the user terminal 5. FIG. 6(a) is an example of this facial image 10, where the facial image 10 on the left has a solid color background, and the facial image 10 on the right does not have a solid color background.

The user terminal 5 accesses the application website in response to user operation, and transmits the facial image 10 and other information required for the application (referred to as the facial image 10, etc.) to the determination device 3 (S2). The determination device 3 receives the facial image 10, etc. (S3), and determines whether the background of the facial image 10 is appropriate or not (S4).

In this embodiment, in S4, first, the person portion 11 is extracted from the face image 10 as shown in FIG. 6(b), and the person portion 11 of the face image 10 is masked. A known technique using AI (artificial intelligence) or the like can be used to extract the person portion 11. Here, in order to prevent the influence of noise on the boundary of the person portion 11, the person portion 11 is expanded outward to create a mask region 12 as shown in FIG. 6(c), and the person portion 11 of the face image 10 is masked by the mask region 12 as shown in FIG. 6(d), with the remaining portion being the background region 13.

Next, as shown in Figure 7, a histogram of brightness and hue is created for the pixels in the background region 13 of the face image 10. The brightness and hue of a pixel can be obtained, for example, from the RGB values of the pixel. Figures 7(a) and (b) are schematic examples of histograms of brightness and hue of the background region 13 for the left and right face images 10 in Figure 6(a), respectively. The center figures in Figures 7(a) and (b) are histograms of brightness, and the right figures are histograms of hue. In each histogram, the vertical axis represents frequency (number of pixels) and the horizontal axis represents brightness or hue.

In this embodiment, to improve the accuracy of the determination, a histogram of brightness and hue is created for the remaining pixels, excluding pixels whose brightness values are within a specified range from the minimum value (referred to as "black pixels") and pixels whose brightness values are within a specified range from the maximum value (referred to as "white pixels"). For example, a histogram of brightness and hue is created for pixels whose brightness values range from 5% to 95%, with the minimum brightness value set to 0% and the maximum value set to 100%. This is because white and black pixels often appear as noise, and even a small difference in RGB values for white and black pixels can cause a large change in hue, which can reduce the accuracy of the determination.

In this embodiment, if there is a single histogram peak in both the brightness and hue histograms, the background of the facial image 10 is deemed to be a solid color and is deemed appropriate. On the other hand, if there are multiple histogram peaks in at least either the brightness or hue histograms, the background of the facial image 10 is not a solid color and is deemed inappropriate.

Note that a histogram peak is at least a portion where the frequency of a certain brightness (or hue) is greater than the frequencies of the brightness (or hue) before and after it, but various conditions can be imposed to make the judgment more suitable. For example, as shown in the hue histogram in FIG. 7(b), only peaks with frequencies equal to or greater than α·Tmax, the maximum frequency value Tmax multiplied by a predetermined percentage α smaller than 1, can be treated as histogram peaks. In addition, it is also possible to set it so that a histogram peak is treated only when it has a certain range of spread, that is, only when the frequencies of brightness (or hue) in a certain range before and after a certain brightness (or hue) continue to decrease as they move away from that brightness (or hue).

When the determination device 3 determines in S4 that the background of the face image 10 is inappropriate (S5; NO), it notifies the user by sending a notification to the user terminal 5 that the background of the face image 10 is inappropriate (S6). When the user terminal 5 receives the notification (S7), it displays it on the display unit 53 (S8) and ends the process. Figure 8 is an example of a notification displayed on the display unit 53 of the user terminal 5.

On the other hand, if the determination device 3 determines in S4 that the background of the facial image 10 is appropriate (S5; YES), it completes the determination process and accepts the application based on the facial image 10 and other information received in S3.

As described above, by using the determination device 3 of this embodiment, it is possible to determine whether the background of the facial image 10 for application is appropriate. In other words, if there is one histogram peak in the brightness and hue histogram of pixels in the background region 13, the background of the facial image 10 can be regarded as a solid color and is therefore appropriate.

In addition, by creating the above histogram for pixels excluding black and white pixels, the accuracy of the judgment can be improved as mentioned above.

In addition, in this embodiment, an online application system 1 is constructed that includes a user terminal 5 and a judgment device 3, and the judgment device 3 can judge the face image 10 sent from the user terminal 5 at the time of application. By automatically detecting a face image 10 with an inappropriate background and rejecting an application using that face image 10 at the time of application, the administrative burden on the department that accepts applications can be reduced.

In addition, in this embodiment, if a facial image 10 is inappropriate, the user is notified and prompted to retake the image, and an application with a facial image 10 using a correct background can be accepted. If the background region 13 is inappropriate, one possible solution is to fill the background region 13 with a solid, single color through image processing, but image processing around the hair is difficult and this can easily result in an unnatural facial image 10. Furthermore, since the user intends to use a facial image 10 that they themselves have taken for their application, notifying the user that the background region 13 is inappropriate and prompting them to retake the image makes it easier to obtain a facial image 10 that is satisfactory, including the background.

However, the present invention is not limited to the above-described embodiment. For example, in this embodiment, the background region 13 is deemed appropriate when it can be considered to be a solid color, but it may also be deemed appropriate when it can be considered to be a solid color and is a specified color. In this case, in the judgment of S4, the background is deemed appropriate when there is a single histogram peak in both the lightness and hue histograms, and the peak of the hue histogram is within the range corresponding to the specified color.

Below, another example of the present invention will be described as the second embodiment. The second embodiment will be described in terms of the differences from the first embodiment, and the description of the similarities will be omitted.

Second Embodiment
The second embodiment differs from the first embodiment in that the user terminal 5 judges the face image 10. That is, in the second embodiment, the user terminal 5 functions as a judgment device and judges in advance whether the background of the face image 10 for application is appropriate or not. A pre-check program for the face image 10 for making the judgment is stored in the storage unit 52 of the user terminal 5.

Figure 9 is a diagram showing the functional configuration of a user terminal 5 according to the second embodiment. In the second embodiment, the user terminal 5 has the above-mentioned determination means 302 and notification means 303.

Figure 10 is a flowchart showing a method for determining a face image 10 according to the second embodiment, and each step in Figure 10 is executed by the control unit 51 of the user terminal 5 by controlling each part of the user terminal 5.

In this embodiment, the user terminal 5 first captures a facial image 10 of the user with the camera 56 in response to a user operation (S11). The captured facial image 10 is recorded in the memory unit 52 of the user terminal 5.

In response to a user's operation, the user terminal 5 starts a pre-check program for the face image 10 stored in the storage unit 52. The user terminal 5 then uses the program to determine whether the background of the face image 10 is appropriate (S12). The method of determination is the same as in the first embodiment.

If the user terminal 5 determines in S12 that the background of the facial image 10 is inappropriate (S13; NO), it displays a notification that the background of the facial image 10 is inappropriate on the display unit 53 in the same manner as in FIG. 8 (S14), thereby notifying the user.

On the other hand, if the user terminal 5 determines in S12 that the background of the facial image 10 is appropriate (S13; YES), it displays on the display unit 53 a message indicating that the background of the facial image 10 is appropriate (S15). The user can confirm that the background of the facial image 10 is appropriate by looking at this message, and can use the facial image 10 for an application (offline) to the application location. Alternatively, the facial image 10 can be used for an online application.

In the second embodiment, a pre-check of the facial image 10 for application can be performed on the user terminal 5, and the user will not use a facial image 10 with an inappropriate background for the application. This prevents delays in the application procedure due to reapplications, etc., and reduces the administrative burden on the department that accepts applications. Note that a pre-check of the facial image 10 for application can also be performed on a server (determination device) that is communicatively connected to the user terminal 5 via a network. The processing of the server in this case is substantially similar to the processing of the determination device 3 described in FIG. 5 etc., and differs only in that the server does not accept applications.

The above describes preferred embodiments of the present invention with reference to the attached drawings, but the present invention is not limited to these examples. It is clear that a person skilled in the art can come up with various modified or revised examples within the scope of the technical ideas disclosed in this application, and it is understood that these also naturally fall within the technical scope of the present invention.

1: Online application system 3: Determination device 5: User terminal 10: Face image 13: Background area 301: Receiving means 302: Determination means 303: Notification means

Claims (6)

A determination device characterized by having a determination means for creating a histogram of brightness and hue of pixels in the background area of a user's facial image for application, and if there is a single histogram peak in both the brightness and hue histograms, determining that the background of the facial image is a solid color and that the background area is appropriate. The determination device according to claim 1, further comprising a receiving means for receiving the face image from a user terminal. The determination device according to claim 1, characterized in that the determination device is a user terminal. The determination device according to any one of claims 1 to 3, further comprising a notification means for notifying a user that the background of the face image is inappropriate if the background area is inappropriate. The determination device according to any one of claims 1 to 4, characterized in that the histogram is created for pixels excluding white and black pixels. Computer,
A program for functioning as a judgment device having a judgment means for creating a histogram of brightness and hue of pixels in the background area of a user's facial image for application, and determining that the background area is appropriate by regarding the background of the facial image as a solid color if both the brightness and hue histograms have a single histogram peak.

Images