JP7707004B2 - Image processing device, image processing method, and program - Google Patents

Description

The present invention relates to a technology for estimating pulse information from an image.

One method for acquiring pulse information from an image is to assume a certain lighting environment, cut out a part of the face, find the change in skin color within that area, and measure the period of change. Patent Document 1 detects a face from an image, generates a color histogram from the area including the detected face, and identifies the range of skin color in the image (values in color space such as YUV, RGB, HSV) based on the histogram. Patent Document 2 repeats face detection and skin color area detection while relaxing the skin color conditions for the image, and identifies the range of skin color at the point when the ratio of the area of the face detection area to the skin color area exceeds a predetermined value and a face is detected.

Patent No. 5153660 JP 2020-102059 A

However, with the method of Patent Document 1, when identifying skin color within a large face detection frame, the colors of the hair and background are mixed in, and when identifying skin color within a small face detection frame, the colors of the outer cheeks and forehead are excluded, so the range of skin color may not be identified appropriately. Furthermore, with the method of Patent Document 2, face detection and detection of skin color areas are repeated while relaxing the skin color conditions for the image, so processing takes time. The present invention was made in consideration of the above problems, and aims to quickly set an appropriate skin color range.

The image processing device according to the present invention is an image processing device that estimates the pulse rate of a person detected from an image, and is characterized by having a setting means for setting an elliptical region that includes a face identified from an image, an acquisition means for acquiring color information of pixels included in the set elliptical region, a determination means for determining whether the acquired color information satisfies a predetermined condition, and a determination means for determining a threshold value indicating a range of skin color based on the color information in the elliptical region when the determination means determines that the acquired color information satisfies the predetermined condition.

This invention allows you to quickly set the appropriate skin color range.

FIG. 2 is a block diagram illustrating an example of a hardware configuration of the image processing device. 1 is a block diagram showing an example of a functional configuration of an image processing device; 11 is a flowchart illustrating a process executed by the image processing device. 11 is a flowchart illustrating a process executed by the image processing device. 1A to 1C are diagrams illustrating an example of an image to be processed. 13 is a table illustrating an example of a skin color threshold determination table. 11 is a flowchart illustrating a process executed by the image processing device. FIG. 13 is a diagram showing an example of a variation. 1 is a block diagram showing an example of a functional configuration of an image processing device; 11 is a flowchart illustrating a process executed by the image processing device. FIG. 13 is a diagram illustrating an example of a background mask. 11 is a flowchart illustrating a process executed by the image processing device. 11 is a flowchart illustrating a process executed by the image processing device. 13A to 13D are diagrams illustrating an example of a saturation threshold correction process. 11 is a flowchart illustrating a process executed by the image processing device. 13A to 13F are diagrams illustrating an example of a threshold correction process. 13A to 13G are diagrams illustrating an example of a threshold correction process. 13A to 13G are diagrams illustrating an example of a threshold correction process.

Below, a preferred embodiment of the present invention will be described with reference to the drawings.

<Embodiment 1>
As a means for acquiring pulse information from an image, a certain lighting environment is assumed, a part of the face is cut out, the change in skin color within the area is obtained, and the period of change is measured. However, when cutting out a part of the face, it is easily affected by the direction of the lighting and the direction of the face, so if the position of the subject cannot be determined in advance using a surveillance camera, etc., it is necessary to consider that the way the skin color is reflected varies. In addition, if the skin color conditions of the entire face are set in advance as fixed, there may be areas that deviate from the fixed value due to the influence of individual differences, differences in cameras, and the color temperature of the lighting. If the deviating areas flicker over time, noise will be introduced into the period of the change in the color of the target, making it impossible to estimate the pulse correctly. Based on these, a method for dynamically determining the definition of the skin color in an image (the range of RGB values) so that pulse estimation from an image is stable even in a situation where the shooting conditions are not constant will be described.

1 is a block diagram showing an example of the hardware configuration of an image processing device according to the present embodiment. The central processing unit (CPU) 11 reads and executes the OS and other programs stored in the ROM 12 and the storage device 14 using the RAM 13 as a work memory, controls each component connected to the system bus 19, and performs calculations and logical judgments for various processes. The processes executed by the CPU 11 include the image processing of the embodiment. The storage device 14 is a hard disk drive or an external storage device, and stores the programs and various data related to the image processing of the embodiment. The input unit 15 is an input device such as an imaging device such as a camera, a button for inputting user instructions, a keyboard, or a touch panel. The storage device 14 is connected to the system bus 19 via an interface such as SATA, and the input unit 15 is connected to the system bus 19 via a serial bus such as USB, but details of these are omitted. The communication I/F 16 communicates with external devices via wireless communication. The display unit 17 is a display. The sensor 18 is an image sensor or a distance sensor. The image processing device may not have some of the hardware described here. For example, if the image processing device is a network camera, it does not matter whether it has an input device such as a keyboard or a display unit 17.

Figure 2 is a block diagram showing an example of the functional configuration of the image processing device 1. The image processing device 1 estimates the pulse rate of a person detected from an image. The image processing device 1 has an image acquisition unit 101, a face detection unit 102, a skin color threshold determination unit 103, a skin color area detection unit 104, a hue calculation unit 105, a hue change management unit 106, a pulse rate estimation unit 107, and a pulse rate recording unit 108. Note that an imaging device that acquires an image, an output device that displays the results of the pulse rate estimation, etc. may be connected as an external device.

The image acquisition unit 101 acquires an image of a target person. For example, the image may be acquired by an imaging unit included in the image processing device 1, or the image may be acquired from a video stored in a storage unit.

The face detection unit 102 detects a face from the acquired image. The face may be tracked by detecting the face across multiple frames. The face detection method uses an existing image recognition method. Specifically, a face detector that has trained a neural network with face images may be used, or the face may be detected by template matching of face images. Alternatively, a screen may be generated to guide the target person into a specified angle of view, and the face image may be acquired by extracting a face area from an image when the person is reflected within the frame of the screen. Regardless of which method is used, it is sufficient that the size of the face in the image is equal to or larger than a certain size, and the face direction and lighting conditions do not necessarily need to be constant. Note that if the face direction and lighting conditions are constant, pulse rate estimation can be performed more accurately, but this embodiment can perform pulse rate estimation suitably even if the face direction and lighting conditions vary to a certain extent.

The skin color threshold determination unit 103 determines the threshold value (conditions for color information to be extracted as skin color) of the target color (skin color in this case) in the input video based on the color information in the face area. The skin color threshold determination unit 103 first sets an elliptical area including the face identified from the image. Next, color information such as hue is acquired from each pixel from the set elliptical area. The color information refers to hue, saturation, and brightness. Furthermore, based on the statistical value of the acquired color information, it is determined whether the color information satisfies a predetermined condition. The statistical value of the color information refers to the average value and standard deviation of the hue, saturation, and brightness. If the statistical value of the color information included in the set elliptical area satisfies the condition, the threshold value of the color information to be extracted as skin color is determined based on the color information. Details will be described later. Since the color information to be extracted as skin color differs depending on the lighting conditions of the video or the skin color of the target person, by specifying the skin color threshold value for each video, it becomes possible to perform pulse estimation that is robust to lighting conditions, face orientation, and individual characteristics.

The skin color area determination unit 104 acquires a partial image (skin color area) to be used for pulse rate estimation from the image based on the determined threshold value of the color of the target. Details will be described later. The hue calculation unit 105 calculates the hue based on the color information of each pixel in the determined skin color area. The hue change management unit 106 calculates the change in hue (frequency) based on the hue calculated in multiple frames. The pulse rate estimation unit 107 estimates the pulse rate of the target person from the frequency of the calculated hue. Estimating the pulse rate from an image has the advantage that the person's health condition can be confirmed even from a distance. Details will be described later. This concludes a brief explanation of the processing of each functional unit.

Next, the processing executed by each functional unit will be explained using the flowchart in FIG. 3. In the following explanation, each process (step) will be indicated by adding an S to the beginning, and the notation of the process (step) will be omitted. However, it is not necessary to perform all of the steps explained in this flowchart.

In S201, the image acquisition unit 101 acquires one frame of image from the video captured by the imaging device. In S202, the image acquisition unit 101 determines whether the imaging by the imaging device has ended. If the imaging is continuing, the process proceeds to S203. If the imaging is ended, the pulse estimation process ends. The continuation of the process may be determined based on a user's instruction. If the process is to continue, in S203, the face detection unit 102 detects a face from the acquired image. Note that since images of multiple frames are acquired, faces are detected from each image. If multiple faces are included, the faces of the same person are associated. In S204, the face detection unit 102 determines an image to be output to the next process for each detected person. The processes from S205 to S211 are performed for all detected faces. In S205, the skin color threshold determination unit 103 determines whether the input image is the first face image in which the target person has been detected. If it is the first face image, the skin color threshold is determined in S206, and the process proceeds to S207. The process of S206 will be described later. If it is not the first face image, the process proceeds directly to S207. The determination here may be made by other methods as long as it can determine whether or not a skin color threshold has been determined for the target person. For example, in the face detection unit 102, face authentication with registered persons is performed, and if a registered person is detected and a skin color threshold corresponding to that person is stored, the process may proceed to S207. In addition, if detection results of multiple frames are obtained for a target person, an image for determining the skin color threshold may be obtained from any image. Here, face detection is performed for each frame, and if a person who has not been detected in the past is detected, the process proceeds to S206, and if a person who has been detected in the past is detected, it is determined that the image is not the first image based on the corresponding tracking information, and the process proceeds to S207. The process of S206 will be described later, and the process of S207 will be described first.

In S207, the skin color region extraction unit 104 uses a skin color threshold determined for each target person to extract pixels having skin color information in the set face region as skin color regions. At this time, the detected face frame (rectangular bounding box) is enlarged, for example, by 1.5 times so that the entire skin color region of the face is included in the detection target frame, and skin color regions are detected for all pixels within the frame.

In S208, the hue calculation unit 105 finds and records the average hue of the detected skin color area. In S209, the hue change management unit 106 determines whether the number of frames in which the hue average value is recorded meets a predetermined number. When the predetermined number is met, the process proceeds to S210. In S210, the pulse estimation unit 107 performs pulse estimation using the change in the hue average value, and in S211, the pulse hypochondrium unit 108 records the pulse value. When pulse estimation for a certain person is completed, the process returns to S204, and pulse estimation is performed again for the next person. When pulse estimation processing has been completed for all detected face tracking, the process proceeds to S201, and processing of the next image continues.

Note that there are various pulse rate estimation methods, such as finding a moving average of the interval between peaks, performing frequency spectrum analysis, and performing principal component analysis, and the present invention does not limit the pulse rate estimation method.

Although in this example the skin color threshold is determined for the first face image of tracking, the skin color threshold may also be determined for images during tracking if they satisfy certain conditions. For example, the threshold may be determined based on when the size of the skin color area reaches its maximum. Alternatively, it may be when the tracking target moves a certain distance, when the change in hue exceeds a certain value, or when a means for detecting face direction is added and the face direction changes. Also, in this example face tracking is performed, but full body tracking may be performed and images with the same tracking ID may be considered to be the same person. Even if the face is not captured, if it is considered to be the same person using full body tracking, there is no need to change the skin color threshold the next time a face is captured.

Figure 4 is a sub-flowchart of S206, and Figure 5 shows an example of determining the skin color threshold.

In S301, the skin color threshold determination unit 103 sets a frame corresponding to the face area for the input image based on the face detected by the face detection unit 102. Here, the face detected by the face detection unit 102 is rectangular, whereas an elliptical frame is set as a frame corresponding to the face area. That is, the initial setting of the face frame ellipse is performed. For example, 401 in FIG. 5(A) is an image (input image) acquired by the image acquisition unit 101, 402 and 403 are the tracking results of face tracking, and 404 and 405 are face frames (rectangular bounding boxes) detected by the face tracking 102. In this example, the face detection frame tends to be detected smaller than the entire face, and needs to be enlarged to include the entire face. A frame about 1.5 times larger than the face detection frames 404 and 405 is set, as in 406 in FIG. 5(B) and 410 in FIG. 5(C). If the classifier of the face detection unit 102 has been trained to detect a larger face detection frame like 406, it is advisable to adjust the enlargement/reduction of the face detection frame according to the tendency of that detection frame. Next, an ellipse of a specified ratio is first placed within the face detection frame. This ellipse is set to be larger than the face detection frame and to include the face detection frame. In this example, an ellipse ratio of 0.85 (1 vertical, 0.85 horizontal) of the enlarged face frame is set as the initial face frame ellipse. Examples are 407 in FIG. 5(B) and 411 in FIG. 5(C). Note that this initial ellipse ratio is determined by investigating the tendency of face detection size in advance.

In S302, the skin color threshold determination unit 103 obtains HSV color information from the RGB values for all pixels within the set face frame ellipse. The method of converting from RGB to HSV is known, so a description thereof will be omitted. Next, in S303, the skin color threshold determination unit 103 calculates the average and standard deviation for each of the HSV values.

In S304, the skin color threshold determination unit 103 determines whether the standard deviation of H is equal to or less than the target value, and in S305, the skin color threshold determination unit 103 determines whether the standard deviation of V is equal to or less than the target value. If either is greater than the target value, the process proceeds to S308, where the face frame ellipse is reduced, and the average and standard deviation of the HSV of the pixels in the face frame ellipse reduced in S303 are calculated. Since the HSV values have been calculated for all pixels in the original face frame ellipse, there is no need to calculate them again. 408 and 409 in FIG. 5(B) and 412-412 in FIG. 5(C) are examples of reduced face frame ellipses. When reducing an ellipse, the vertical size may be reduced at a ratio of 1/2 the horizontal size. An ellipse ratio of 0.85 is 1 vertical size and 0.85 horizontal size, and an ellipse ratio of 0.80 is 0.975 vertical size and 0.80 horizontal size.

When the standard deviations of both H and V are below the target value, the process proceeds to S306, where the skin color threshold determination unit 103 sets the thresholds (range) for H and S. The thresholds are centered on the average value and use the standard deviation multiplied by a predetermined multiple A, with the value subtracted from the average set as the lower threshold (minimum value of the range) and the value added to the average set as the upper threshold (maximum value of the range). Note that when determining the threshold for S, a predetermined multiple A2 separate from the predetermined multiple A for H may be set to determine the range.

In S307, the skin color threshold determination unit 103 determines the threshold value of V. The threshold value may be determined in the same manner as for HS, but in face images, the standard deviation of V tends to be larger than that of H or S, and the range will be wider if the same multiple as H or S is used. When the average value of brightness V is within a range between a first predetermined value indicating the lower limit of the target value and a second predetermined value indicating the upper limit of the target value, a fixed value set in advance is used. If the average value is smaller than the first predetermined value, the threshold value is determined based on a statistical value. That is, the value obtained by multiplying the standard deviation by a predetermined multiple B is subtracted from the average value to determine the lower limit threshold. The upper limit threshold remains fixed. Also, if the average value of V is larger than the second predetermined value, the threshold value is determined based on a statistical value. That is, the value obtained by multiplying the standard deviation by a predetermined multiple B is subtracted from the average to determine the lower limit threshold, and the value obtained by multiplying the standard deviation by a predetermined multiple C is added to the average to determine the upper limit threshold. The predetermined multiples A, B, and C are determined experimentally.

Note that in S304 it is determined whether the standard deviation of H is below the target value, and in S305 it is determined whether the standard deviation of V is below the target value, but if either of them falls below the target value first, the average and standard deviation at that time may be retained, and the average and standard deviation at the time when they fell below the target value may be used in determining the threshold. Note that in S304 and S305 it is determined whether the standard deviations of H and V are below the target value, but a separate target value may also be set for S to determine whether it is below the target value.

Figure 6 shows an example of the ellipse ratio and the average and standard deviation of HSV when the face frame ellipse is being reduced. When it falls below the target value, the upper and lower limit values are found and a threshold is set for each tracking. In this example, the predetermined multiple A is 2.0, B is 0.6, and C is 2.0. The ellipse ratio to be reduced may be decreased at regular intervals. The reduction rate of the face frame ellipse may be increased by a predetermined proportion. Also, the reduction rate may be changed more greatly when the target value is far away, and less greatly when the target value is approached.

As described above, the skin color threshold can be determined for each person or shooting environment, so pulse rate can be estimated accurately from images even in situations where shooting conditions are not constant.

<Embodiment 2>
In the first embodiment, in S206, the average and standard deviation are calculated for all pixels within the ellipse. In this case, if the person has bangs covering the face or is wearing accessories such as glasses or a face mask, colors other than skin color may be mixed into the face frame ellipse, and each color information may not reach the target value. In that case, the shape of the ellipse may be replaced to calculate the average and standard deviation of each color information.

Figure 7 is a flowchart explaining the processing executed by the image processing device when replacing the shape of the face frame ellipse in S206. The processing from S301 to S308 is the same as the processing indicated by the corresponding symbols in Figure 4, so the explanation is omitted.

In S601, the skin color threshold determination unit 103 judges whether the ellipse ratio is equal to or less than a predetermined value (the reduction ratio is equal to or greater than a predetermined value). If it is equal to or less than the predetermined value, ellipse shape replacement is performed in S602. 701 to 703 in FIG. 7 are replacement candidate ellipse shapes. The pixels in the gray area are not included in the calculation of the average and standard deviation. 701 is an image with bangs, 702 is an image with a face mask, and 703 is an image with sunglasses. It is possible to create combinations of these shapes. The shape to be changed is selected arbitrarily, and if the ellipse ratio for that shape falls below the predetermined value, a deformation area of a different shape is set as the ellipse area. In S603, the skin color threshold determination unit 103 sets the ellipse ratio to return to the initial ellipse ratio, and proceeds to S303 to repeat the process.

Furthermore, a means for detecting attached items may be combined to complement face detection. In that case, a corresponding elliptical shape is selected from the beginning. For example, in the face detection unit 102, a detector that detects objects that obscure the face is used to detect specific objects such as attached items (masks, glasses, sunglasses, etc.) or bangs at the same time as face detection. When a specific object is detected, the skin color threshold determination unit 103 sets a different deformation area that corresponds to each specific object.

As described above, if an object is attached that obscures the skin-colored area of the face, the shape of the face frame ellipse to be set is replaced with another shape. This allows the skin-colored area to be set while suppressing colors other than skin color, making it easier to set the skin color threshold and improving the accuracy of pulse rate estimation.

<Embodiment 3>
If the color of the background is similar to skin color and is included in the range of color information defined as skin color, the hue average value may be calculated including the background when the skin color region is determined using the determined threshold. In that case, the accuracy of the pulse estimation may decrease. Therefore, when the skin color threshold is determined, the degree of confusion with the background color may be checked, and if confusion occurs, a background mask may be created to exclude the background region from the target of subsequent skin color region detection.

Figure 9 is a block diagram showing an image processing device 1 that executes processing to mask a background region. Components 101 to 108 have the same configuration as the image processing device in Figure 1, so a description thereof will be omitted. A background mask setting unit 801 sets a mask for the background region using a histogram based on color information in the image.

Figure 10 is a flowchart for explaining the processing executed by the image processing device 1. S201 to S211 are the same as in Figure 2, so their explanation will be omitted. In S901, the skin color area detection unit 104 performs skin color area detection using the determined skin color threshold value. The background mask setting unit 801 sets the background mask in S902. The background mask may be set when there is confusion between the background color and the skin color of the face, or may be set even when there is no confusion between the background and the face. The processing of S902 will be described in detail later.

In S903, the skin color region detection unit 104 determines whether or not a background mask has been set. If a background mask is present, in S904 the background mask is applied to the pixels within the face frame to exclude them from the detection of the skin color region, and the process proceeds to S208. In other words, the skin color region detection unit 104 extracts the skin color region from the region not included in the background mask. If no background mask is present, the process proceeds directly to S208.

Figure 11 shows an example of a background mask, and Figure 12 is a flowchart of the background mask setting process (S902). A face image 1001 in Figure 11 is an example where the background and the skin color of the face are similar.

In S1101 of FIG. 12, the background mask setting unit 801 generates a histogram showing the distribution of the number of skin-colored pixels in the horizontal direction of the image. 1002 in FIG. 10 is an example of the generated histogram. The x-axis is the position of the pixel in the horizontal direction of the image, and the number of skin-colored pixels present on each X-coordinate is the y-axis. Here, points 1003 and 1004 where the y value drops near both ends can be considered to indicate the x-coordinates outside the face. In S1102, the background mask setting unit 801 judges whether or not there is confusion between the background and the face color based on the generated histogram. For example, if there is a histogram shape where the y value is large outside the positions of the valleys on both sides, it is judged that there is confusion between the background color and the skin color. As the end points of the face, two points where the number of pixels is lower than a predetermined value may be selected, or an end point where the number of pixels is minimal based on the slope of the curve may be searched for. Specifically, when the number of pixels in the range with the X coordinate smaller than the left end point is a specified amount greater than the number of pixels at the end point, or when the slope is a negative value, it is possible that the left half of the face and the background are similar in color. The same is true for the right end point; by comparing the number of pixels in the range with the X coordinate larger than the right end point, or when the slope of the line graph in that range is a positive value, it can be determined that the right end point is the edge of the face and that the background is confused with a similar color.
Other methods may be used to determine whether the boundary between the face and the background is visible.

If it is determined in S1103 that background skin color confusion has occurred, the background mask setting unit 801 generates a background mask in S1104. Here, if it is determined that no confusion has occurred, no mask is generated, and the process of S901 is terminated.

The background mask is an elliptical region that excludes the points where the y value drops near both ends. In other words, the width of the background mask is determined based on a histogram. Note that the shape of the background mask is not limited to an ellipse, but may be a shape as shown in FIG. 8. 1005 in FIG. 11 is the created background mask, and 1006 in FIG. 11 is the face image to which the background mask has been applied, which is created in the background mask generation process of S904 in FIG. 9. Note that the shape of the background mask is not limited to an ellipse, but may be one that excludes both sides, as shown in 1007 in FIG. 11.

When various objects are captured in the background and are partially confused with the skin color of the face, a histogram like 1010 in Figure 11 is created. There are objects that are determined to be skin color on both sides of the valleys 1008 and 1009 on both sides. In this case too, it is determined that the background color and skin color are being confused. Also, since a fixed size background mask cannot be used when the size of the face detection frame changes, the position of the valley can be recorded as a ratio to the overall size, and an ellipse of that ratio can be recreated as the background mask to match the size of the face and then applied.

For example, if a person moves and the background changes, the decision to apply a background mask may be made for every frame. In that case, the decision on whether to confuse the background and face colors is made every time for the pulse estimation frame, not just when the skin color threshold is set. Specifically, after detecting a skin color area, the confusion decision of S1103 is made to determine whether to generate a background mask. By performing this process, pulse estimation from an image is possible even when the environment is changing.

<Embodiment 4>
In the first embodiment, statistical values are obtained until the standard deviations of hue H and lightness V fall below the target values, and the threshold value for the skin color range is determined based on the average and standard deviation. However, even if the standard deviations of H and V reach the target values, the standard deviation of saturation S may remain large, and if a threshold value is determined by obtaining a certain multiple of the standard deviation from the average value of S, it may exceed the maximum or minimum value that S can take. This occurs due to the influence of the type of camera, auto white balance, lighting color, background color, etc.

In this case, even if the threshold is set by limiting it to the highest or lowest value of S, the threshold range of S may become too wide, and even the background may be regarded as skin. If the standard deviation of S becomes too large and the threshold range becomes too wide, it is also possible to limit it to an upper margin value and a lower margin value that add a margin to the highest and lowest values.

Figure 13 is a flowchart for explaining the process executed by the image processing device 1. This process is related to the determination of S in the process of determining the thresholds for H and S in S306 in Figure 4. Figures 14 (B) to (D) are diagrams showing an example of the process.

In S1301, the skin color threshold determination unit 103 determines the threshold using the average value and standard deviation of saturation S. Specifically, the lower threshold (minimum value of the range) is the value obtained by subtracting the value obtained by multiplying the standard deviation of saturation S by a predetermined multiple A from the average value of saturation S. The upper threshold (maximum value of the range) is the value obtained by adding the value obtained by multiplying the standard deviation of saturation S by a predetermined multiple A to the average value of saturation S. FIG. 14(A) shows an example of determining the skin color threshold from the image in FIG. 14(B). This is a calculation example when the predetermined multiple A is set to 2.0, and the lower limit of saturation S is -6 as shown in 1401. The maximum and minimum values that saturation S can take are set differently depending on the calculation method, but in this example, the values are set to 0 to 255, which exceeds the minimum value that can be taken. Fig. 14(C) shows the result when the threshold is set to this value or limited to a value between 0 and 255, with skin tones displayed as white and non-skin tones displayed as black. There are also white areas of skin tones in the background.

In S1302, the skin color threshold determination unit 103 determines whether the upper limit of the determined threshold value of saturation S exceeds the upper limit margin value Uth. For example, the upper limit margin value Uth is determined by adding 75% of the difference between the upper limit of the possible values of saturation S, 255, and the average value to the average value. If the upper limit of the threshold value of saturation S is greater than Uth, the upper limit margin value Uth is determined as the upper limit of the threshold value of saturation S in S1303. If it is smaller, the upper limit determined in S1301 remains unchanged.

In S1304, the skin color threshold determination unit 103 determines whether the lower limit of the threshold determined in S1301 is smaller than the lower limit margin value. For example, a value that is 25% of the average value is set as the lower limit margin value Lth. If it is smaller than Lth, the lower limit value with margin Lth is set as the lower limit of the saturation S threshold in S1303. If it is larger, the lower limit determined in S1301 remains as it is. Figure 14 (D) shows an example in which the threshold of saturation S is limited to the lower limit margin value. There are no white areas in the background that are skin color. Note that in this example, the margin is calculated based on the average value, but it may be a fixed value, or it may be calculated using the average value and standard deviation.

<Embodiment 5>
In the second embodiment, when the standard deviation does not reach the target value, it is assumed that the subject may be wearing accessories such as glasses or a face mask, and the shape of the ellipse is changed. However, due to the influence of the camera type, auto white balance, lighting color, background color, etc., the standard deviation of the hue H may not reach the target even if the subject is not wearing an accessory. If the threshold is determined using the standard deviation when the target is not reached, the area that is determined to be skin color may become wider, and parts that are determined to be skin color may appear in the background.

In the third embodiment, a background mask is applied when skin-colored parts appear in the background. This background mask excludes all areas except the face, but does not accurately capture the shape of the face, and may include background parts or exclude face parts. It would be preferable if a threshold value that can distinguish between skin and background colors could be found.

Therefore, the skin color threshold determination unit 103 may determine a provisional threshold, check the skin color content inside and outside an ellipse in which a face is assumed to be, and modify the skin color threshold based on the skin color content inside and outside the ellipse. If the skin color content outside the ellipse is equal to or greater than a certain value, the standard deviation value that determines the width of the threshold is reduced to change the provisional threshold. If the skin color content inside the ellipse becomes too small, the provisional threshold is returned to the previous value.

Figure 15 is a flowchart for explaining the processing executed by the image processing device 1. The threshold correction processing may be performed as a post-processing step after the processing of S307 in the flowchart of Figure 4, after the threshold is determined, or may be performed as a processing step after S601 in Figure 7, when the standard deviation does not reach the target value. If placed after S601, the threshold correction processing is performed and then the process ends.

Figure 16 (A) to (F) show an example of the process.

In S1501, the skin color threshold determination unit 103 provisionally determines a skin color threshold. The provisional determination method is the same as the skin color threshold determination method in the flowchart of FIG. 4. FIG. 16(A) shows an example of thresholds determined using the image of FIG. 16(B). 1601 is the standard deviation of hue H, which is larger than the target value. 1602 is the standard deviation of saturation S, which is slightly larger. In this example, the target value of hue H is 12, and the target value of saturation S is 20.

In S1502, the skin color area detection unit 104 uses the determined threshold value to determine the skin color content inside and outside the ellipse that indicates the face. FIG. 16(C) is a diagram in which the parts determined to be skin color are shown in white, and the parts that are not skin color are shown in black. FIG. 16(D) is a diagram in which an ellipse 1608 is placed to separate the inside and outside. The ellipse in this case is the ellipse used initially to determine the threshold value, and is not the ellipse that has been narrowed down and made smaller in order to determine the threshold value.

In S1503, the skin color area detection unit 104 determines whether the skin color content rate outside the ellipse exceeds a certain value Oth. In this example, Oth is set to 10%. If it does not exceed the value, the process proceeds to S1504, where the provisional threshold value is set as the skin color threshold value.

The skin color content rate 1603 outside the ellipse in FIG. 16(A) is 40%, which exceeds the fixed value Oth. If it exceeds, the process proceeds to S1505, where the skin color threshold determination unit 103 determines whether the value of the standard deviation that determines the width of the threshold can be reduced. In this embodiment, the value of the standard deviation of the hue H is used to determine the width of the threshold. When this value is reduced, the statistical value becomes meaningless, but for convenience, the term standard deviation is used. In this example, the value that can be reduced is set to 1.2 times or more of the target value, and reduction is limited to one time and cannot be made smaller than 1.2 times the target value. If reduction is possible, the process proceeds to S1507, where it is determined whether the skin color content rate within the ellipse is greater than a fixed value Ith. In this example, Ith is 50%. The value of the standard deviation of the hue H in FIG. 16(A) can be reduced, and the skin color content rate 1604 within the ellipse is 64%, which is greater than the fixed value Ith. In S1509, the value of the standard deviation is reduced, and the process returns to S1501 and is repeated. In FIG. 16(A), 1605 and 1606 are the reduced values. Note that the reduction can be repeated multiple times starting from a larger multiple.

Figures 16 (E) and (F) show examples in which the skin color threshold is calculated using the reduced standard deviation, and the skin color content is checked again in S1502. Since the skin color content 1607 outside the ellipse 1609 is 6%, which is smaller than the constant value Oth, the skin color threshold determination unit 103 sets a threshold in S1504 and ends the process.

Figure 17 shows an example where the background and skin color cannot be distinguished within the range that can be reduced. Figure 17(A) shows an example of a threshold value determined using the image in Figure 17(B). 1701 and 1702 in Figure 17(A) are both large values. Figures 17(C) and (D) show the skin color parts in white and the non-skin color parts in black. When the skin color content rate inside and outside the ellipse in 1707 is calculated, the skin color content rate outside the ellipse 1703 is 50%. Figures 17(E) and (F) show the distribution of the skin parts as a result of reducing the standard deviation as in 1704 and 1705 and correcting the threshold value. When the skin color content rate inside and outside the ellipse 1708 in Figure 17(F) is calculated, the skin color content rate outside the ellipse 1706 is 32%, which exceeds the fixed value Oth. S1504 checks whether the image can be reduced, but the image has already been reduced to 1.2 times the target value, and further reduction is not possible. Proceed to S1506, set a background mask that excludes the area outside the ellipse from the range of pulse rate estimation, and proceed to S1504, where the provisional threshold is set as the skin color threshold.

1709 in Figure 17(G) is an example of a background mask.

Figures 18A to 18G show an example of preventing the skin color range from becoming too narrow due to the reduction in standard deviation, resulting in an increase in areas of the face that are not skin colored. The skin color threshold is calculated from the image in Figure 18B. The values of 1801 and 1802 in Figure 18A are somewhat large. Figures 18C and 18D show the areas that are skin colored in white and the areas that are not skin colored in black. As shown in Figure 18C, there are some areas in the background that are skin colored. The skin color content 1803 outside the ellipse 1809 in Figure 18D is 12%, which is somewhat larger than the constant value Oth. The standard deviation is reduced to 1807 and 1808 to calculate the skin color. Figures 18E and 18F show the distribution of areas that are skin colored as a result of correcting the threshold. As shown in Figure 18E, the areas that are skin colored in the background have decreased, but the areas that are not skin colored in the face have increased. The skin color content 1806 within the ellipse 1810 in Figure 18 (F) is 47%, which is smaller than the constant value Ith.

In S1507, if the skin color threshold determination unit 103 determines that the skin color content rate within the ellipse is smaller than a certain value Ith, the process proceeds to S1508, where the provisional threshold is restored to its original value. 1807 and 1808 in FIG. 18(A) have the same values as 1801 and 1802. Next, the process proceeds to S1506, where a background mask is set to exclude the area outside the ellipse from the range of pulse estimation. 1811 in FIG. 18(G) is this background mask. Next, in S1504, the provisional threshold is set as the skin color threshold. Note that if S1508 is reached without going through the process of reducing the standard deviation, the skin color threshold determination unit 103 does not change the provisional threshold.

Note that an ellipse is used to check the skin color content of the face and background, but a rectangular area corresponding to the neck position can also be added to the bottom of the ellipse and this area can also be measured within the ellipse.

Other Embodiments
The present invention can also be realized by executing the following process. That is, software (programs) that realize the functions of the above-described embodiments are supplied to a system or device via a data communication network or various storage media. Then, a computer (or a CPU, MPU, etc.) of the system or device reads and executes the program. The program may also be provided by recording it on a computer-readable recording medium.

In addition, among the above-mentioned processing units, the face detection unit 102 and the like may instead use a trained model that has been machine-learned for processing. In that case, for example, multiple combinations of input data and output data for the processing unit are prepared as training data, knowledge is acquired from them by machine learning, and a trained model is generated that outputs output data for the input data based on the acquired knowledge. The trained model can be configured, for example, as a neural network model. Then, the trained model performs the processing of the processing unit by operating in cooperation with a CPU or GPU as a program for performing processing equivalent to that of the processing unit. In addition, the trained model may be updated after a certain amount of processing as necessary.

REFERENCE SIGNS LIST 1 Image processing device 101 Image acquisition unit 102 Face detection unit 103 Skin color threshold determination unit 104 Skin color area detection unit 105 Hue calculation unit 106 Hue change management unit 107 Pulse estimation unit 108 Pulse recording unit

Claims (21)

An image processing device that estimates a pulse rate of a person detected from an image,
A setting means for setting an area including a face identified from an image;
an acquisition means for acquiring color information of pixels included in the set area;
a determination means for determining whether each component of the acquired color information satisfies a predetermined condition;
a determination means for determining a threshold value indicating a skin color range based on the color information in the region when the determination means determines that the acquired color information satisfies the predetermined condition ,
An image processing device characterized in that, when the judgment means judges that the color information does not satisfy the specified condition, the setting means reduces the size of the region depending on at least one of the components until the component satisfies the specified condition . The color information includes hue, saturation, and brightness.
2 . The image processing apparatus according to claim 1 , wherein the acquiring means acquires statistical values of hue and brightness for pixels included in the region. the determining means determines, as the predetermined condition, whether or not the statistical values of hue and brightness of the pixels included in the region satisfy a target value;
3. The image processing apparatus according to claim 2, wherein the determining means determines the threshold value from the color information in the region when the statistical values of the hue and brightness of the pixels included in the region satisfy the target value. The image processing device according to claim 3, characterized in that the determination means determines the threshold value based on the statistical values of hue and saturation or a preset fixed value. The statistical values are a standard deviation and an average value of each color information,
The image processing apparatus according to claim 4 , wherein the determining means determines the threshold value based on standard deviations of hue and saturation. The image processing device according to claim 5, characterized in that the determining means determines the saturation threshold value among the determined skin color threshold values to be a value obtained by adding or subtracting a predetermined value from the average saturation value. the determining means determines the average brightness value to be a preset fixed value if the average brightness value is within a range between a first predetermined value indicating a lower limit of the target value and a second predetermined value indicating an upper limit of the target value;
7. The image processing apparatus according to claim 5, wherein when the average brightness value is smaller than the first predetermined value or larger than the second predetermined value, the threshold value is determined based on the statistical value. 8. The image processing device according to claim 1, wherein the setting means sets a deformation area showing a predetermined shape in the area when the determination means determines that the color information does not satisfy the predetermined condition. 9. The image processing apparatus according to claim 8 , wherein said setting means sets said modification area in said area when the size of said area becomes equal to or smaller than a predetermined value. a detection means for detecting an object that is occluding the face detected from the image;
10. The image processing apparatus according to claim 8 , wherein the setting means sets the deformation area differently depending on the detected object. 11. The image processing apparatus according to claim 10 , wherein the detection means detects front hair, glasses, and a mask. an extraction means for extracting a skin color area from an input image including the face based on the determined skin color threshold;
12. The image processing apparatus according to claim 1 , further comprising: an estimation unit for estimating a pulse rate of a person based on the extracted skin-tone area. The face tracking device further includes a tracking means for tracking the face,
The image processing device according to claim 12 , wherein the estimation means estimates a pulse rate for the tracked face. A generating means for generating a background mask for the input image,
14. The image processing apparatus according to claim 12 , wherein the extraction means extracts the skin color area from an area not included in the background mask. 15. The image processing device according to claim 14, further comprising an update means for updating the determined skin color threshold value in accordance with a skin color content rate in the region or a skin color content rate outside the region , calculated using the determined skin color threshold value. 16. The image processing device according to claim 15, wherein the updating means reduces the determined skin color threshold value when it is determined that the skin color content rate outside the region is greater than a predetermined value and that the skin color content rate in the region is greater than a predetermined value. 17. The image processing device according to claim 15, wherein the updating means does not update the determined skin color threshold value when it determines that the skin color content rate outside the region is greater than a predetermined value and that the skin color content rate in the region is smaller than a predetermined value. 18. The image processing device according to claim 15 , wherein the generating means generates a background mask for the input image when the updating means determines that the content rate of skin color outside the region is greater than a predetermined value. 19. The image processing device according to claim 14, wherein the generating means generates a histogram indicating the number of pixels included in the determined threshold based on color information of pixels included in the region, and generates the background mask based on the histogram. 1. An image processing method for estimating a pulse rate of a person detected from an image, comprising:
A setting step of setting an area including a face identified from the image;
acquiring color information of pixels included in the set area;
a determination step of determining whether each component of the acquired color information satisfies a predetermined condition;
a determination step of determining a threshold value indicating a skin color range based on the color information in the region when the acquired color information is determined to satisfy the predetermined condition ,
An image processing method characterized in that, if it is determined in the judgment process that the color information does not satisfy the specified condition, in the setting process, a size of the region is reduced depending on at least one of the components until the component satisfies the specified condition . A program for causing a computer to function as each of the means included in the image processing apparatus according to any one of claims 1 to 19 .

Images