JP7675550B2 - Image analysis device, pulse wave detection device, and image analysis method - Google Patents

Description

The following disclosure relates to a video analysis device that detects the pulse waves of a living body by analyzing a video of the living body.

One example of a device for detecting pulse waves in a living body is a pulse wave detection device that detects pulse waves based on the output values (pixel values) of an imaging element (light receiving element). However, it is known that in such pulse wave detection devices, the accuracy of pulse wave detection can decrease due to saturation of pixel values. Therefore, Patent Document 1 proposes a technology aimed at improving the accuracy of pulse wave detection. Specifically, Patent Document 1 discloses a method of controlling a liquid crystal filter to suppress the amount of light received by the imaging element below the saturation level.

JP 2008-301934 A

Recently, a technology has been proposed that detects pulse waves by capturing images of a living body using an imaging device and analyzing the images. However, as described below, Patent Document 1 does not take into consideration any specific method for improving the accuracy of pulse wave detection based on such image analysis. An objective of one aspect of the present disclosure is to improve the accuracy of pulse wave detection using a method different from conventional methods.

To solve the above problem, a video analysis device according to one embodiment of the present disclosure includes a region of interest detection unit that detects a region of interest of a living body from a video image showing the living body based on a predetermined algorithm, and a pulse wave detection unit that detects the pulse wave of the living body based on the change over time in the pixel values of the plurality of pixels after reducing the influence of specific high-luminance pixels having pixel values equal to or greater than a pixel value threshold among the plurality of pixels included in the region of interest.

In addition, the video analysis method according to one aspect of the present disclosure includes a region of interest detection step of detecting a region of interest of a living body from a video image showing the living body based on a predetermined algorithm, and a pulse wave detection step of detecting a pulse wave of the living body based on a time change in the pixel values of the plurality of pixels after reducing the influence of a specific high-luminance pixel having a pixel value equal to or greater than a pixel value threshold among the plurality of pixels included in the region of interest.

According to one aspect of the present disclosure, it is possible to improve the accuracy of pulse wave detection using a method different from conventional methods.

1 is a block diagram showing a configuration of a main part of a pulse wave detecting device according to a first embodiment. 4A and 4B are diagrams illustrating an example of the arrangement of red, green, and blue color filters in a color filter portion. 4 is a diagram showing an example of a region of interest detected by a region of interest detection unit; FIG. 11A and 11B are diagrams illustrating an example of an interpolation process for missing pixel values in each color channel image. FIG. 13 is a diagram showing an example of a synthesis process of each color channel image after interpolation. FIG. 13 is a diagram illustrating an example of how a pixel value changes over time in a full-color region-of-interest image. FIG. 2 shows an example of classification of each pixel in a full-color region-of-interest image at a given time. FIG. 13 shows another example of classification of each pixel in a full-color region-of-interest image at a given time. 13A and 13B are diagrams for explaining pixel value thresholds in a modified example of the first embodiment. 11 is a block diagram showing the configuration of a main part of a pulse wave detecting device according to a second embodiment. FIG. 11A and 11B are diagrams illustrating an example of a process for setting each pixel value in each color low resolution channel image. FIG. 13 is a diagram showing an example of a synthesis process of low-resolution channel images of each color. FIG. 11 is a block diagram showing the configuration of a main part of a pulse wave detecting device according to a third embodiment.

[Embodiment 1]
A pulse wave detection device 1 according to the first embodiment will be described below. For convenience of explanation, the same reference numerals will be used in the following embodiments to designate components having the same functions as those described in the first embodiment, and the description thereof will not be repeated. For the sake of brevity, the description of matters similar to those in the publicly known technology will also be omitted as appropriate.

Please note that each configuration and each numerical value described in this specification is merely an example unless otherwise specified. Therefore, unless otherwise specified, the positional relationship of each member is not limited to the example in each figure. Also, please note that each drawing is intended to roughly explain the shape, structure, and positional relationship of each member, and is not necessarily drawn as it actually is. In this specification, the description "A to B" in relation to two numbers A and B means "greater than or equal to A and less than or equal to B" unless otherwise specified.

(Overview of pulse wave detection device 1)
1 is a block diagram showing the configuration of the main parts of a pulse wave detection device 1. The pulse wave detection device 1 detects the pulse waves of a living organism 900 (e.g., a human). The pulse wave detection device 1 is an example of a non-contact type pulse wave detection device. Hereinafter, the pulse waves of the living organism H will be simply referred to as the pulse wave. The pulse wave detection device 1 includes a video analysis device 10 and an imaging device 50 (e.g., a camera). The video analysis device 10 and the imaging device 50 are connected so as to be able to communicate with each other.

The imaging device 50 captures a video (moving image) showing the living body H. Specifically, the imaging device 50 captures images (still images) that constitute the video for each frame period. In the first embodiment, the imaging device 50 is an RGB (Red, Green, Blue) camera. Therefore, the video in the first embodiment is an RGB color video. The imaging device 50 includes a color filter unit 51 and an imaging unit 52.

The color filter unit 51 includes a red color filter 511R, a green color filter 511G, and a blue color filter 511B. The imaging unit 52 includes a plurality of imaging elements 520 arranged in an array. The imaging elements 520 are known image sensors. The imaging elements 520 receive light (e.g., external light) that arrives from the living body 900 and passes through the color filter unit 51.

The imaging element 520 is also referred to as a pixel of the imaging device 50. For this reason, in this specification, the imaging element 520 is also referred to as an imaging pixel. W x H imaging pixels are arranged in the imaging section 52. W represents the number of imaging pixels in the horizontal direction (imaging pixels per row), and H represents the number of imaging pixels in the vertical direction (imaging pixels per column). The imaging pixels output, as pixel values (tone values), the signal values (e.g., current values) of electrical signals indicating the intensity (e.g., luminance) of the light they receive. More specifically, the imaging pixels output pixel values as analog values.

In the imaging device 50, the pixel values as analog values are converted to digital values by an AD converter (not shown). The imaging device 50 then outputs a digital image (an image having digital pixel values). As an example, the pixel values in the first embodiment are sampled as 8-bit digital values. Therefore, the pixel values in the first embodiment are in the range of 0 to 255.

Figure 2 is a diagram showing an example of the arrangement of the red color filter 511R, green color filter 511G, and blue color filter 511B in the color filter section 51. In the imaging device 50, the red color filter 511R, green color filter 511G, and blue color filter 511B are arranged in one-to-one correspondence with the imaging elements 520. One red color filter 511R is arranged so as to cover one imaging element 520. One green color filter 511G is arranged so as to cover another imaging element 520. One blue color filter 511B is arranged so as to cover yet another imaging element 520.

As shown in FIG. 2, in the first embodiment, a red color filter 511R, a green color filter 511G, and a blue color filter 511B are arranged in a Bayer pattern. In the color filter section 51 in the example of FIG. 2, color filter units 510 corresponding to four (two horizontally by two vertically) imaging pixels are defined.

One color filter unit 510 includes (i) one red color filter 511R, (ii) two green color filters 511G (green color filters 511G1 and 511G2), and (iii) one green color filter 511B. In the color filter section 51, a plurality of color filter units 510 are repeatedly arranged in the horizontal and vertical directions. In this manner, in the color filter section 51, the red color filter 511R, the green color filter 511G, and the blue color filter 511B are arranged in a mosaic pattern (more specifically, in a checkered pattern) with a number ratio of 1:2:1.

In this specification, an imaging pixel corresponding to the red color filter 511R is referred to as a red imaging pixel. One red imaging pixel receives light (red light) that has passed through the red color filter 511R, and outputs a red pixel value (R pixel value) that indicates the intensity of the red light. Also, an imaging pixel corresponding to the green color filter 511G is referred to as a green imaging pixel. One green imaging pixel receives light (green light) that has passed through the green color filter 511G, and outputs a green pixel value (G pixel value) that indicates the intensity of the green light. Also, an imaging pixel corresponding to the blue color filter 511B is referred to as a blue imaging pixel. One blue imaging pixel receives light (blue light) that has passed through the blue color filter 511B, and outputs a blue pixel value (B pixel value) that indicates the intensity of the blue light.

The imaging unit 52 generates, for each frame period of the video, (i) an R channel image IMGR (an image showing the distribution of R pixel values output by a plurality of red imaging pixels), (ii) a G channel image IMGRG (an image showing the distribution of G pixel values output by a plurality of green imaging pixels), and (iii) a B channel image IMGB (an image showing the distribution of B pixel values output by a plurality of blue imaging pixels). IMGR, IMGG, and IMGB all have a resolution W×H. The imaging unit 52 then synthesizes IMGR, IMGG, and IMGB to generate an image IMG as a mosaic-like RGB color image. In other words, the imaging unit 52 generates IMG as a RAW image.

As described above, the imaging unit 52 generates IMG as RAW images for each frame that constitutes the video. The imaging unit 52 then generates video (RAW video) by arranging each generated frame in the order of frame number. The imaging unit 52 supplies the generated video to the video analysis device 10.

The video analysis device 10 includes a region of interest detection unit 11 and a pulse wave detection unit 12. The video analysis device 10 detects pulse waves by analyzing the video acquired from the imaging device 50. More specifically, the video analysis device 10 detects pulse waves by analyzing each frame of the video (each of the multiple IMGs).

The region of interest detection unit 11 detects a region of interest (ROI) from the IMG based on a predetermined algorithm (e.g., using a publicly known algorithm). The region of interest is a partial area of the IMG in which a predetermined part of the living body 900 is shown. FIG. 3 is a diagram showing an example of a region of interest detected by the region of interest detection unit 11. In the following description, the region of interest in the example of FIG. 3 is referred to as region of interest 910.

The IMG in the example of Figure 3 is an image showing the entire face of the living body 900. In the example of Figure 3, one cheek of the face is preset as a predetermined part. First, the region of interest detection unit 11 detects a face region (a region showing the face of the living body 900) from the IMG. Then, the region of interest detection unit 11 detects an area within the face region showing one cheek as the region of interest 910. The region of interest 910 in the example of Figure 3 is a rectangular region.

However, as will be clear to those skilled in the art, the predetermined area does not have to be limited to the cheek, but may be any area from which information regarding the blood vessels of the living body 900 can be extracted. Other examples of predetermined areas include the forehead, the root of the nose, the neck, the fingertips, and the palm. The region of interest may be any image area that depicts a change in skin color of the living body over time. Furthermore, the number of regions of interest is not necessarily limited to one. In this way, the IMG is not limited to the example of FIG. 3.

The pulse wave detection unit 12 detects pulse waves based on the change over time in pixel values of multiple pixels included in the region of interest 910. As an example, the pulse wave detection unit 12 calculates, for each frame, (i) the average value Rave of the R pixel values included in the region of interest 910, (ii) the average value Gave of the G pixel values included in the region of interest 910, and (iii) the average value Bave of the B pixel values included in the region of interest 910. In other words, the pulse wave detection unit 12 derives time series data for each of Rave, Gave, and Bave.

The pulse wave detection unit 12 may then detect (derive) the pulse wave by analyzing the time series data using a known algorithm. As an example, the pulse wave detection unit 12 may detect the pulse wave using the method disclosed in International Publication WO2020/090348.

However, if pixel value saturation occurs within the region of interest 910, the accuracy of pulse wave detection by the pulse wave detection unit 12 may decrease. Such pixel value saturation may occur, for example, when the brightness of external light is high. Therefore, in the first embodiment, as described below, the pulse wave detection unit 12 detects pulse waves after reducing the influence of pixels in the region of interest 910 where pixel value saturation occurs (hereinafter referred to as saturated pixels).

In this specification, "saturated pixel value" means that the pixel value is a predetermined maximum value that can be set in an image (more precisely, digital image data). As described above, in the first embodiment, pixel values are sampled as 8-bit digital values. Therefore, in the first embodiment, "saturated pixel value" means that the pixel value is 255. In contrast, in the first embodiment, "non-saturated pixel value" means that the pixel value is 0 to 254. For the above reasons, a saturated pixel in the first embodiment is a pixel that has a pixel value of 255. Therefore, in the first embodiment, the pixel value of 255 is also referred to as a saturated pixel value. Note that the pixel value in the above description refers to any one of the R pixel value, the G pixel value, and the B pixel value.

(Example of Processing of Pulse Wave Detector 12 in the First Embodiment)
4 and 5 are diagrams illustrating an example of processing by the pulse wave detection unit 12. As described below, Fig. 4 shows an example of interpolation processing for missing pixel values in each color channel image, while Fig. 5 shows an example of synthesis processing for each color channel image after interpolation.

The pulse wave detection unit 12 decomposes the image of the region of interest 910 as a RAW image (hereinafter referred to as the region of interest RAW image) into a region of interest R channel image IMGR1, a region of interest G channel image IMGG1, and a region of interest B channel image IMGB1. FIG. 4 shows examples of IMGR1, IMGG1, and IMGB1. In the following, for example, the pixel in the i-th row and j-th column in IMGR1 is represented as IMGR1(i,j). The pixel value of the pixel is also represented as IMGR1(i,j).

As shown in FIG. 4, IMGR1 contains pixels with missing pixel values. Hereinafter, pixels with missing pixel values are referred to as missing pixels. The missing pixels in IMGR1 are (i) pixels at positions corresponding to green imaging pixels, and (ii) pixels at positions corresponding to blue imaging pixels. On the other hand, pixels with no missing pixel values are referred to as non-missing pixels. The non-missing pixels in IMGR1 are pixels at positions corresponding to red imaging pixels.

The pulse wave detection unit 12 generates an interpolated R channel image of the region of interest IMGR2 by interpolating the missing pixel values (pixel values of missing pixels) in IMGR1 using the non-missing pixel values (pixel values of non-missing pixels) in IMGR1. That is, the pulse wave detection unit 12 generates IMGR2 by demosaicing IMGR1 (more specifically, color demosaicing).

As an example, consider IMGR1(2,2) as the pixel of interest in IMGR1. In the example of FIG. 4, IMGR1(2,2) is a missing pixel. The pulse wave detection unit 12 may interpolate the pixel value of a missing pixel (e.g., IMGR1(2,2)) using non-missing pixel values in a 3×3 rectangular area centered on the missing pixel. In other words, the pulse wave detection unit 12 may use the non-missing pixel values to set the pixel value of IMGR2 at the position corresponding to the missing pixel.

In the example of FIG. 4, IMGR1(1,1), IMGR1(1,3), IMGR1(3,1), and IMGR1(3,3) are non-missing pixel values in the rectangular region. Therefore, the pulse wave detection unit 12 sets IMGR2(2,2) using these four non-missing pixel values. As an example, the pulse wave detection unit 12 may set IMGR2(2,2) by linear interpolation using the four non-missing pixel values.

In the example of Fig. 4, the pulse wave detection unit 12 sets IMGR2(2,2) as the average value of the four non-defective pixel values.
IMGR2(2,2)
= {IMGR2(1,1)+IMGR2(1,3)
+IMGR2(3,1)+IMGR2(3,3)}/4...(1)
IMGR2(2,2) is set as follows. Other non-defective pixel values may be interpolated in the same manner as in the above example. For the interpolation when the pixel of interest is a non-defective pixel, see the example of IMGB1 described later.

Similarly, the pulse wave detection unit 12 generates an interpolated G channel image of the region of interest IMGG2 by demosaicing IMGG1. As an example, consider IMGG1(2,2) as the pixel of interest in IMGG1. IMGG1(2,2) in the example of Figure 4 is also a missing pixel, like IMGG1(2,2).

The pulse wave detection unit 12 sets IMGR2(2,2) using non-defective pixel values in a 3×3 rectangular region centered on IMGG1(2,2). In the example of Fig. 4, IMGG1(1,2), IMGG1(2,1), IMGG1(2,3), and IMGG1(3,2) are the non-defective pixel values in the rectangular region. In the example of Fig. 4, the pulse wave detection unit 12
IMGG2(2,2)
= {IMGG1(1,2)+IMGG1(2,1)
+IMGG1(2,3)+IMGG1(3,2)}/4...(2)
Then, IMGG2(2,2) is set as follows.

Similarly, the pulse wave detection unit 12 generates an interpolated region of interest B channel image IMGB2 by demosaicing IMGB1. As an example, consider IMGB1(2,2) as a pixel of interest in IMGB1. IMGB1(2,2) in the example of FIG. 4 is a non-defective pixel, unlike IMGR1(2,2) and IMGG1(2,2).

If the pixel of interest is a non-defective pixel, the pulse wave detection unit 12 sets the pixel value of the non-defective pixel as the pixel value of IMGB2 at the position corresponding to the non-defective pixel.
IMGB2 (2, 2) = IMGB1 (2, 2) ... (3)
Then, IMGB2(2,2) is set as follows.

As described above, the pulse wave detection unit 12 demosaices IMGR1, IMGG1, and IMGB1, respectively, to generate IMGR2, IMGG2, and IMGB2, which do not have non-defective pixels. IMGR2, IMGG2, and IMGB2 generated as described above each have a resolution of W×H.

The pulse wave detection unit 12 then generates a full-color image IMG2 of the region of interest having a resolution of W×H by combining IMGR2, IMGG2, and IMGB2. FIG. 5 shows an example of IMG2. As shown in FIG. 5, IMG2 is a full-color RGB image. A full-color RGB image is an image in which all pixels have three pixel values: an R pixel value, a G pixel value, and a B pixel value. Hereinafter, these three pixel values are collectively referred to as RGB pixel values.

Specifically, the pulse wave detection unit 12
IMG2(i,j)
={IMGR2(i,j),IMGG2(i,j),IMGB2(i,j)}...(4)
The R pixel value, G pixel value, and B pixel value of IMG2(i,j) are set as follows: In the example of Fig. 5, IMGR2(2,2) = 80, IMGG2(2,2) = 130, and IMGB2(2,2) = 200. Therefore, the pulse wave detection unit 12 sets IMG2(2,2) as IMG2(2,2) = (80, 130, 200).

Because IMG2 is a full-color RGB image, it contains more color information than the region of interest RAW image. In other words, IMG2 contains more information about the blood vessels of the living body 900. Therefore, by detecting the pulse wave based on IMG2, the accuracy of pulse wave detection can be improved.

Therefore, in the first embodiment, it is preferable that the pulse wave detection unit 12 calculates Rave, Gave, and Bave for each frame using the pixel values (RGB pixel values) of each pixel included in IMG2. In this way, it is preferable that the pulse wave detection unit 12 derives time series data of Rave, Gave, and Bave based on IMG2. Then, the pulse wave detection unit 12 detects the pulse wave by analyzing the time series data.

In addition, in the first embodiment, the pulse wave detection unit 12 detects a pulse wave after reducing the influence of saturated pixels included in IMG2. Saturated pixels are an example of specific high luminance pixels (described below) according to one aspect of the present disclosure. As an example, the pulse wave detection unit 12 determines (identifies) pixels in IMG2 that have a pixel value equal to or greater than a pixel value threshold value Lth as specific high luminance pixels.

In the first embodiment, Lth is set as a value indicating saturation of pixel values. That is, Lth is set as a value equal to the saturated pixel value. Specifically, Lth is set as 255. For this reason, Lth in the first embodiment may be referred to as a saturation threshold. Hereinafter, Lth as the saturation threshold will be represented as Lth1. In the first embodiment, the pulse wave detection unit 12 determines that, among the pixels in IMG2, pixels having a pixel value equal to Lth1 (i.e., pixels having a pixel value of 255) are saturated pixels.

Figure 6 is a graph illustrating the change over time in pixel value L of a pixel in IMG2 (e.g., IMG2(2,2)). L is one of the R pixel value, G pixel value, or B pixel value. In the example of Figure 6, L is the B pixel value. The horizontal axis in the graph of Figure 6 is time t. The horizontal axis may also be interpreted as the frame number.

In the example of FIG. 6, L = Lth1 from t1 to t2 (first time range) and from t3 to t4 (second time range). Therefore, the pulse wave detection unit 12 determines that IMG2(2,2) is a saturated pixel in the first time range and the second time range. In this case, as an example, the pulse wave detection unit 12 calculates Rave, Gave, and Bave without using the R pixel value, G pixel value, and B pixel value of IMG2(2,2) in the first time range and the second time range, respectively.

However, even if the B pixel value of IMG2(2,2) is saturated, the R pixel value and G pixel value of IMG2(2,2) may be unsaturated. Therefore, when the B pixel value is saturated and the R pixel value and G pixel value are unsaturated in IMG2(2,2), the pulse wave detection unit 12 may calculate Rave and Gave using the R pixel value and G pixel value of IMG2(2,2) in the first time range and the second time range, respectively. It is sufficient for the pulse wave detection unit 12 to be able to calculate the time series data by excluding pixel values where saturation has occurred.

Figure 7 is a diagram showing an example of classification of each pixel in IMG2 at a certain time (e.g., t1). As described above, the pulse wave detection unit 12 identifies saturated pixels 610 based on Lth1. In this case, the pulse wave detection unit 12 may further identify the pixels in IMG2 excluding the saturated pixels 610 as normal pixels 620. The pulse wave detection unit 12 may then detect a pulse wave based on the change over time in the pixel value of the normal pixels 620.

(effect)
As described above, in the technology of Patent Document 1 (an example of a conventional technology), the liquid crystal filter is controlled so as to suppress the amount of light received by the imaging element below a saturation level. However, the technology of Patent Document 1 is premised on a wristwatch-type pulse wave detection device (a contact-type pulse wave detection device). Therefore, in the technology of Patent Document 1, the direction in which light (e.g., external light) is incident on the region of interest is limited to a certain extent. For this reason, Patent Document 1 proposes a method for controlling the liquid crystal filter based on the premise that the direction of the external light is specified in advance.

However, in a pulse wave detection device that detects pulse waves by analyzing video captured by an imaging device (for convenience, referred to as an "image analysis-based pulse wave detection device"), external light typically enters the region of interest from many different directions. For this reason, in a video analysis-based pulse wave detection device, unlike the technology of Patent Document 1, it is difficult to specify in advance the direction in which external light will enter the region of interest.

Therefore, even if the technology of Patent Document 1 is adopted in an image analysis-based pulse wave detection device, it is not possible to appropriately control the amount of light received by the image sensor. In other words, even if the technology of Patent Document 1 is adopted in an image analysis-based pulse wave detection device, it is not possible to suppress the occurrence of saturated pixels in the region of interest. As described above, in the past, no consideration was given to methods for improving the detection accuracy of pulse waves in an image analysis-based pulse wave detection device.

On the other hand, as described above, the pulse wave detection device 1 (more specifically, the video analysis device 10) can detect pulse waves after reducing the influence of saturated pixels in the region of interest. Therefore, even if saturated pixels are present in the region of interest, it is possible to prevent a decrease in the accuracy of pulse wave detection. Therefore, unlike the technology of Patent Document 1, it is possible to improve the accuracy of pulse wave detection even if the direction of external light is not specified in advance. As described above, the pulse wave detection device 1 can improve the accuracy of pulse wave detection by a method different from conventional methods.

[Modifications]
(1) Fig. 8 is a diagram showing another example of classification of pixels in IMG2 at a certain time (e.g., t1). In the example of Fig. 8, similarly to the above-mentioned Fig. 6, a saturated pixel 610 is identified by the pulse wave detection unit 12. The pulse wave detection unit 12 may further identify pixels located around the saturated pixel 610 as surrounding pixels 615.

As an example, the pulse wave detection unit 12 may identify pixels within IMG2 that are within a predetermined distance range from the saturated pixel 610 as the surrounding pixels 615. In the example of FIG. 8, the pulse wave detection unit 12 identifies pixels adjacent to the saturated pixel 610 (pixels that are one pixel away) as the surrounding pixels 615. In the example of FIG. 8, the distance between pixels is defined as an 8-neighborhood distance (chessboard distance). However, it goes without saying that in one aspect of the present disclosure, the definition of the distance between pixels is not limited to the example of FIG. 8.

In the example of FIG. 8, the pulse wave detection unit 12 further identifies the pixels of IMG2, excluding the saturated pixels 610 and the surrounding pixels 615, as normal pixels 620A. The pulse wave detection unit 12 may then detect a pulse wave based on the change over time in the pixel value of the normal pixels 620A.

In many cases, the local distribution of pixel values within an image has a certain degree of continuity. For this reason, even if the surrounding pixels 615 are not saturated pixels at a certain time, they are expected to be pixels that are highly likely (probable) to change to saturated pixels as time progresses. Therefore, as shown in FIG. 8, the pulse wave detection unit 12 may exclude not only the saturated pixels 610 but also the surrounding pixels 615 to identify the normal pixels 620A. This makes it possible to detect the pulse wave while reducing the influence of not only the saturated pixels 610 but also the surrounding pixels 615. As a result, it becomes possible to further improve the accuracy of pulse wave detection.

(2) As described above, in one aspect of the present disclosure, the pulse wave detection unit 12 may detect a pulse wave based on the change over time in the pixel values of normal pixels. However, if the number of normal pixels is too small, there is a concern that the pulse wave may be detected with insufficient (low) accuracy.

The pulse wave detection unit 12 may therefore stop detecting pulse waves when the number of normal pixels is less than the pixel number threshold. In other words, the pulse wave detection unit 12 may detect pulse waves only when the number of normal pixels is equal to or greater than the pixel number threshold. Note that the pulse wave detection unit 12 may output notification information indicating that pulse wave detection will be stopped when the number of normal pixels is less than or equal to the pixel number threshold. As an example, when the number of normal pixels is less than or equal to the pixel number threshold, the pulse wave detection unit 12 may output an error message indicating that "pulse wave detection will be stopped because there is a concern that pulse waves will not be detected with sufficient accuracy" to a display unit (not shown) of the pulse wave detection device 1.

The above configuration can prevent a pulse wave from being detected with insufficient accuracy when the number of normal pixels is less than the pixel count threshold. The pixel count threshold is not particularly limited, but may be set based on the number of normal pixels that is expected to enable detection of a pulse wave with a relatively high degree of accuracy. As an example, the pixel count threshold may be set to 100.

(3) As is clear from the above explanations, Lth (pixel value threshold) is not limited to the saturation threshold (Lth1). As an example, Lth may be set as a value indicating a high probability of pixel value saturation. Thus, in this specification, pixels whose pixel values are equal to or greater than Lth are referred to as specific high luminance pixels. Note that, in this specification, saturated pixels are also included in the "pixels with a high probability of pixel value saturation."

Hereinafter, Lth in this modified example will be referred to as Lth2. As an example, Lth2 may be set to 230. However, Lth2 is of course not limited to the above example. Lth2 may be set to a value greater than or equal to 90% and less than or equal to 100% of the saturated pixel value. Therefore, when the pixel value is sampled as an 8-bit digital value (when the saturated pixel value is 255), Lth2 may be set to a value greater than or equal to 230 and less than or equal to 255.

Specific high-luminance pixels are thought to contribute to a decrease in the accuracy of pulse wave detection, even if the effect is not as significant as that of saturated pixels. For this reason, as an example, the pulse wave detection unit 12 may determine that pixels whose pixel value is equal to or greater than Lth2 are specific high-luminance pixels. The pulse wave detection unit 12 may then detect the pulse wave after reducing the influence of the specific high-luminance pixels. In this case, the accuracy of pulse wave detection can be further improved. As can be understood from the above explanation, the surrounding pixels in the example of Figure 8 described above can be said to be another example of specific high-luminance pixels.

Figure 9 is a diagram for explaining Lth2. As in Figure 6, Figure 9 also illustrates the change over time in pixel value L of one pixel in IMG2 (e.g., IMG2(2,2)). As in the example of Figure 6, L in the example of Figure 9 is also assumed to be a B pixel value.

In the example of FIG. 9, L≧Lth2 in (i) tm1 to tm2 (first time range), (ii) tm3 to tm4 (second time range), and (iii) tm5 to tm6 (third time range). Therefore, the pulse wave detection unit 12 determines that IMG2(2,2) is a specific high-luminance pixel in the first time range, second time range, and third time range.

Therefore, as an example, the pulse wave detection unit 12 calculates Rave, Gave, and Bave in the first time range, the second time range, and the third time range, respectively, without using the R pixel value, the G pixel value, and the B pixel value of IMG2(2,2). This allows the pulse wave detection unit 12 to detect the pulse wave while reducing the influence of IMG2(2,2), which is a specific high luminance pixel.

However, even if the B pixel value of IMG2(2,2) is equal to or greater than Lth2, the R pixel value and the G pixel value of IMG2(2,2) may be less than Lth2. Therefore, when the B pixel value of IMG2(2,2) is equal to or greater than Lth2 and the R pixel value and the G pixel value are less than Lth2, the pulse wave detection unit 12 may calculate Rave and Gave using the R pixel value and the G pixel value of IMG2(2,2) in the first time range, the second time range, and the third time range, respectively. In this case, the pulse wave detection unit 12 can detect the pulse wave after reducing the influence of IMG2(2,2), which is a specific high luminance pixel. As described above, the pulse wave detection unit 12 only needs to be able to calculate the time series data by excluding pixel values equal to or greater than Lth2.

(4) The method of detecting a pulse wave by reducing the influence of specific high-luminance pixels (e.g., saturated pixels) is not limited to the above-mentioned examples. For example, a pulse wave can be detected without necessarily eliminating the pixel values of saturated pixels.

As an example, consider the case where each pixel in IMG2 is classified into saturated pixels and normal pixels as shown in FIG. 7 above. In this case, the pulse wave detection unit 12 may set different weighting coefficients for saturated pixel values and normal pixel values (hereinafter referred to as normal pixel values). In the following description, the weighting coefficient for saturated pixel values is represented as w1, and the weighting coefficient for normal pixel values is represented as w2. Specifically, the pulse wave detection unit 12 sets w1 to a value smaller than w2. w1 and w2 may be referred to as the saturated weighting coefficient and the normal weighting coefficient, respectively.

The pulse wave detection unit 12 may then derive the pulse wave by substituting (i) the weighted saturated pixel value, which is the saturated pixel value multiplied by w1, and (ii) the weighted normal pixel value, which is the normal pixel value multiplied by w2, into a predetermined arithmetic expression for deriving the pulse wave. The arithmetic expression is set so that the smaller the weighting coefficient for a pixel value, the smaller the effect that pixel value has on the arithmetic result (pulse wave). As described above, the pulse wave detection unit 12 may detect the pulse wave in accordance with the arithmetic expression using the weighted saturated pixel value and the weighted normal pixel value. This allows the pulse wave to be detected while reducing the effect of the saturated pixel value on the pulse wave and taking into account the presence of the saturated pixel value.

As another example, consider a case where the pulse wave detection unit 12 classifies each pixel in IMG2 into a specific high luminance pixel and a normal pixel based on Lth2. In this case, similar to the above example for saturated pixels, the pulse wave detection unit 12 may set different weighting coefficients for the pixel values of the specific high luminance pixels (hereinafter referred to as specific high luminance pixel values) and the normal pixel values. In the following explanation, the weighting coefficient for the specific high luminance pixel values is denoted as w1a. The pulse wave detection unit 12 sets w1a to a value smaller than w2. w1a may be set to a value equal to w1 or to a value larger than w1. w1a may be referred to as a specific weighting coefficient.

The pulse wave detection unit 12 may then derive the pulse wave by substituting (i) the weighted specific high-luminance pixel value, which is the specific high-luminance pixel value multiplied by w1a, and (ii) the weighted normal pixel value into the above arithmetic formula. As described above, the pulse wave detection unit 12 may detect the pulse wave in accordance with the above arithmetic formula using the weighted specific high-luminance pixel value and the weighted normal pixel value. This makes it possible to detect the pulse wave while reducing the effect of the specific high-luminance pixel value on the pulse wave and taking into account the presence of the specific high-luminance pixel value.

As yet another example, consider a case in which the pixels in IMG2 are classified into saturated pixels, peripheral pixels, and normal pixels, as shown in FIG. 8 above. In this case, the pulse wave detection unit 12 may set different weighting coefficients for the saturated pixel values, the pixel values of the peripheral pixels (hereinafter referred to as peripheral pixel values), and the normal pixel values. In the following description, the weighting coefficient for the peripheral pixel values is represented as wm. The pulse wave detection unit 12 sets wm to a value smaller than w2 and larger than w1. wm may be referred to as the peripheral weighting coefficient.

The pulse wave detection unit 12 may then derive the pulse wave by substituting (i) the weighted saturated pixel value, (ii) the weighted surrounding pixel value, which is the surrounding pixel value multiplied by wm, and (iii) the weighted normal pixel value, into the above arithmetic formula. As described above, the pulse wave detection unit 12 may detect the pulse wave in accordance with the above arithmetic formula using the weighted saturated pixel value, the weighted surrounding pixel value, and the weighted normal pixel value. This allows the pulse wave to be detected while taking into account the difference in the influence that the saturated pixel value and the surrounding pixel value have on the pulse wave.

[Embodiment 2]
10 is a block diagram showing the configuration of the main parts of pulse wave detection device 2 of embodiment 2. The video analysis device of pulse wave detection device 2 is referred to as video analysis device 10A. Also, the pulse wave detection unit of video analysis device 10A is referred to as pulse wave detection unit 12A. Pulse wave detection unit 12A generates a full-color image of the region of interest by a method different from that of pulse wave detection unit 12.

Figures 11 and 12 are diagrams each explaining an example of processing by the pulse wave detection unit 12A. As described below, Figure 11 shows an example of processing for setting each pixel value in each color low-resolution channel image. Figure 11 shows examples of IMGR1, IMGG1, and IMGB1, similar to those in Figure 4 above. Figure 12 shows an example of processing for synthesizing each color low-resolution channel image.

The pulse wave detection unit 12A generates a low-resolution R channel image IMGR3, a low-resolution G channel image IMGG3, and a low-resolution B channel image IMGB3 by down-converting IMGR1, IMGG1, and IMGB1, respectively. In the example of FIG. 10, the pulse wave detection unit 12A generates IMGR3, IMGG3, and IMGB3, each having a resolution of W/2×H/2. That is, the pulse wave detection unit 12A performs down-conversion on IMGR1, IMGG1, and IMGB1, respectively, reducing the horizontal resolution and vertical resolution by half.

As an example, the pulse wave detection unit 12A sets one pixel value in IMGR3 using the pixel values of each pixel included in a 2x2 rectangular area in IMGR1 (hereinafter referred to as a red pixel unit). One pixel unit corresponds to one color filter unit 510 in FIG. 2 described above. Below, a pixel unit including four pixels, IMGR1(i,j), IMGR1(i,j+1), IMGR1(i+1,j), and IMGR1(i+1,j+1), is referred to as a red pixel unit Runit(i,j). The pixel units of other colors are denoted in a similar manner.

According to the arrangement of the color filters in FIG. 2, one Runit in IMGR1 contains only one non-defective pixel. For example, Runit (1,1) contains only IMGR1 (1,1) as a non-defective pixel. Therefore, the pulse wave detection unit 12A may set a single non-defective pixel value in one Runit as the pixel value of IMGR3 corresponding to that one Runit. As an example, the pulse wave detection unit 12A may set
IMGR3(1,1)=IMGR1(1,1)...(5)
10, IMGR1(1,1)=80. Therefore, the pulse wave detection unit 12A sets the pixel value of the pixel corresponding to Runit(1,1) as IMGR3(1,1)=80.

As described above, by setting the pixel value of IMGR3(i,j) so that it corresponds one-to-one with the single non-missing pixel value contained in Runit(i,j), IMGR3, an R channel image with a resolution of W/2 × H/2, is generated.

According to the arrangement of the color filters of each color in FIG. 3, one blue pixel unit Bunit in IMGB1 includes only one non-defective pixel. Therefore, IMGB3 may be generated based on IMGB1, similar to the above example for IMGR3. For example, Bunit(1,1) includes only IMGB1(2,2) as a non-defective pixel. Therefore, the pulse wave detection unit 12A
IMGB3(1,1)=IMGB1(2,2)...(6)
IMGB3(1,1) may be set as follows.

In the example of FIG. 11, IMGB1(2,2) = 180. Therefore, the pulse wave detection unit 12A sets the pixel value of the pixel corresponding to Bunit(1,1) as IMGB3(1,1) = 180. In this way, by setting the pixel value of IMGB3(i,j) so that there is a one-to-one correspondence with the single non-missing pixel value contained in Bunit(i,j), IMGR3, which is a B channel image having a resolution of W/2 × H/2, is generated.

According to the arrangement of the color filters of each color in FIG. 3, one green pixel unit Gunit in IMGG1 includes two non-defective pixels. For example, Gunit(1,1) includes two non-defective pixels IMGG1(1,2) and IMGG1(2,1).

Therefore, the pulse wave detection unit 12A may set the average value of the two non-defective pixel values in one bunit as the pixel value of IMGB3 corresponding to that pixel unit.
IMGG3(1,1)={IMGG1(1,2)+IMGG1(2,1)}/2
…(7)
11, IMGG1(1,2)=140, and IMGG1(2,1)=150. Therefore, the pulse wave detection unit 12A sets the pixel value of the pixel corresponding to Gunit(1,1) as IMGG3(1,1)=145.

In this way, by setting the pixel value of IMGG3(i,j) so that it corresponds one-to-one with the average value of the two non-missing pixel values contained in Gunit(i,j), IMGG3, a G channel image with a resolution of W/2 × H/2, is generated.

As described above, the pulse wave detection unit 12A may set the pixel value of a pixel corresponding to a pixel unit in a low-resolution channel image of a certain color based on a statistical value (e.g., average value) of pixel values of one or more non-missing pixels included in one pixel unit in a channel image of the certain color. This makes it possible to generate low-resolution channel images of each color without interpolating the non-missing pixel values.

Next, the pulse wave detection unit 12A generates a low-resolution full-color region-of-interest image IMG3 having a resolution of W/2×H/2 by combining IMGR3, IMGG3, and IMGB3. An example of IMG3 is shown in FIG. 12. IMG3 is a full-color RGB image, similar to IMG2 described above.

Specifically, the pulse wave detection unit 12A includes:
IMG3(i,j)
={IMGR3(i,j),IMGG3(i,j),IMGB3(i,j)}...(8)
The R pixel value, G pixel value, and B pixel value of IMG3(i,j) are set as follows: As described above, IMGR3(1,1) = 80, IMGG2(1,1) = 145, and IMGB3(1,1) = 180. Therefore, the pulse wave detection unit 12A sets IMG3(1,1) as IMG3(1,1) = (80, 145, 180).

The pulse wave detection unit 12A calculates Rave, Gave, and Bave for each frame using the pixel values (RGB pixel values) of each pixel included in IMG3. In this way, the pulse wave detection unit 12A derives time series data for Rave, Gave, and Bave based on IMG3. Then, the pulse wave detection unit 12 detects the pulse wave by analyzing the time series data, similar to embodiment 1.

As described above, pulse wave detection device 2 can detect pulse waves based on a full-color image (e.g., IMG3) of the region of interest that is generated without interpolating non-missing pixel values. This makes it possible to detect pulse waves after eliminating the effects of errors resulting from the interpolation of non-missing pixel values. Therefore, pulse wave detection device 2 can further improve the accuracy of pulse wave detection.

[Embodiment 3]
13 is a block diagram showing the configuration of the main parts of pulse wave detection device 3 of embodiment 3. The video analysis device of pulse wave detection device 3 is referred to as video analysis device 10B. Unlike video analysis device 10, video analysis device 10B further includes a video storage unit 13. In addition, the pulse wave detection unit of pulse wave detection device 3 is referred to as pulse wave detection unit 12B. Pulse wave detection unit 12B detects pulse waves using a method different from that of pulse wave detection unit 12.

The video storage unit 13 stores the video generated by the imaging device 50 (more specifically, the imaging unit 52). As an example, the video storage unit 13 stores the RAW video generated by the imaging unit 52. Note that the video storage unit 13 may record only the pixel values of pixels included in the region of interest 910 detected by the region of interest detection unit 11 from the video generated by the imaging unit 52.

The number of frames of the video stored in the video storage unit 13 may be a predetermined number of frames that has been set in advance. In other words, the duration of the video stored in the video storage unit 13 may be set in advance. As an example, consider a case where the predetermined number of frames is set to 300. In this case, the video storage unit 13 stores each frame of the video generated by the imaging unit 52 until the number of frames of the video reaches 300 frames.

The video storage unit 13 may store the video for the period during which the facial region of the living body 900 is detected from the video. For example, the video storage unit 13 may start storing the video when the facial region is detected from the video by the region of interest detection unit 11. Then, the video storage unit 13 may end storing the video when the facial region is no longer detected from the video by the region of interest detection unit 11 (detection of the facial region has ended).

In addition, the video storage unit 13 may terminate storage of the video if the position of the face area detected in the video changes significantly. Such a large change in the position of the face area may occur, for example, when the living body 900 moves significantly.

As an example, in the region of interest detection unit 11 of embodiment 3, an acceptable threshold for change in the position of the face region (in other words, an acceptable threshold for the movement of the living body 900) may be set in advance. In this case, the video storage unit 13 may terminate storage of the video when the region of interest detection unit 11 determines that the amount of change (amount of movement) in the position of the face region between any two frames of the video (e.g., between two adjacent frames) is equal to or greater than the acceptable threshold.

The tolerance threshold may be set as appropriate by the designer of the pulse wave detection device 3, based on the size of the living body 900 in real space (more specifically, the size of the face of the living body 900). For example, the tolerance threshold may be set to a length of 50 pixels in the image. In this example, the length of 50 pixels is set to a length equivalent to, for example, 5 centimeters in real space.

The pulse wave detection unit 12B identifies, among the multiple pixels contained in the region of interest 910 in each frame of the image stored in the image storage unit 13, pixels having a pixel value equal to or greater than Lth as specific high-luminance pixels. For example, the pulse wave detection unit 12B identifies, in each frame of the image stored in the image storage unit 13, pixels having a pixel value equal to or greater than Lth1 as saturated pixels.

In this case, for example, pulse wave detection unit 12B derives time series data for each of Rave, Gave, and Bave using only the pixel values of pixels identified as normal pixels across all frames of the video stored in video storage unit 13. For example, pulse wave detection unit 12B may derive time series data for each of Rave, Gave, and Bave using only the pixel values of pixels identified as non-saturated pixels across all frames of the video stored in video storage unit 13.

The pulse wave detection unit 12B may derive time series data for each of Rave, Gave, and Bave based on IMG2, as in the first embodiment. Alternatively, the pulse wave detection unit 12B may derive time series data for each of Rave, Gave, and Bave based on IMG3, as in the second embodiment.

In the video stored in the video storage unit 13, the number of pixels contained in the region of interest 910 must be the same across all frames. It is also desirable that each pixel in the video represents color information of the same location on the living body 900 across all frames.

For this reason, when the living body 900 moves, the video storage unit 13 may use a predetermined image processing technique to generate an image that tracks the same location of the living body 900. Alternatively, the imaging device 50 may be provided with a function for tracking the living body 900. Also, as described above, when the living body 900 moves significantly, the video storage unit 13 may end storage of the video.

The pulse wave detection device 3 can detect pulse waves using information on the same number of pixels (e.g., pixel value distribution of the same number of pixels) contained in the region of interest 910 across all frames of the video. In other words, the pulse wave detection device 3 can detect pulse waves while reducing the effect of errors caused by differences in the number of pixels contained in the region of interest 910 between frames of the video.

As described above, the pulse wave detection device 3 detects pulse waves based on the change over time in pixel values of multiple pixels included in the region of interest 910 after reducing the influence of specific high-luminance pixels across all frames of the image stored in the image storage unit 13 (in other words, across the entire duration of the image stored in the image storage unit 13).

However, as will be clear to those skilled in the art, the pulse wave detection device 3 does not necessarily need to reduce the influence of specific high-luminance pixels across all frames of the image. The pulse wave detection device 3 only needs to be able to detect pulse waves based on the temporal change in pixel values of multiple pixels included in the region of interest 910 after reducing the influence of specific high-luminance pixels in at least one frame of the image stored in the image storage unit 13.

[Software implementation example]
The functions of pulse wave detection devices 1 to 3 (hereinafter referred to as the "devices") can be realized by a program for causing a computer to function as the device, and a program for causing a computer to function as each control block of the device (in particular, each part included in video analysis devices 10 to 10B).

In this case, the device includes a computer having at least one control device (e.g., a processor) and at least one storage device (e.g., a memory) as hardware for executing the program. The control device and storage device execute the program, thereby realizing each of the functions described in each of the above embodiments.

The program may be recorded on one or more computer-readable recording media, not on a temporary basis. The recording media may or may not be included in the device. In the latter case, the program may be provided to the device via any wired or wireless transmission medium.

In addition, some or all of the functions of each of the above control blocks can be realized by a logic circuit. For example, an integrated circuit in which a logic circuit that functions as each of the above control blocks is formed is also included in the scope of the present invention. In addition, it is also possible to realize the functions of each of the above control blocks by, for example, a quantum computer.

The processes described in each of the above embodiments may be executed by AI (Artificial Intelligence). In this case, the AI may run on the control device, or on another device (such as an edge computer or a cloud server).

[Additional Notes]
One aspect of the present disclosure is not limited to the above-described embodiments, and various modifications are possible within the scope of the claims. An embodiment obtained by appropriately combining the technical means disclosed in each of the different embodiments is also included in the technical scope of one aspect of the present disclosure. Furthermore, by combining the technical means disclosed in each of the embodiments, new technical features can be formed.

REFERENCE SIGNS LIST 1, 2, 3 Pulse wave detection device 10, 10A, 10B Video analysis device 11 Region of interest detection section 12, 12A, 12B Pulse wave detection section 13 Video storage section 50 Imaging device 51 Color filter section 52 Imaging section 520 Imaging element 510 Color filter unit 511R Red color filter 511G, 511G1, 511G2 Green color filter 511B Blue color filter 610 Saturated pixel (specific high luminance pixel)
615 Surrounding pixels (specific high brightness pixels)
620, 620A Normal pixels (pixels excluding specific high luminance pixels)
900 Living body 910 Region of interest IMG Image (one frame of an image showing a living body)
IMG2 Full color image of the region of interest IMG3 Low-resolution full color image of the region of interest Lth1 Saturation threshold (pixel value threshold)
Lth2 pixel value threshold

Claims (8)

a region of interest detection unit that detects a region of interest of a living body from an image showing the living body based on a predetermined algorithm;
a pulse wave detection unit that detects a pulse wave of the living body based on a time change in pixel values of a plurality of pixels included in the region of interest after reducing an influence of a specific high luminance pixel having a pixel value equal to or greater than a pixel value threshold,
The pulse wave detection unit includes:
Identifying pixels other than the specific high luminance pixel from the plurality of pixels as normal pixels;
The image analyzing device detects the pulse wave when the number of the normal pixels is equal to or greater than a pixel number threshold . the pixel value threshold is set as a value indicating a high probability of pixel value saturation occurring,
The video analysis device according to claim 1 , wherein the pulse wave detection section identifies, among the plurality of pixels, pixels having a pixel value equal to or greater than the pixel value threshold as the specific high luminance pixel. the pixel value threshold is set to a value equal to a predetermined saturated pixel value;
3. The video analysis device according to claim 2, wherein the pulse wave detection section identifies, as saturated pixels, the specific high luminance pixels, which are pixels among the plurality of pixels and have a pixel value equal to the pixel value threshold value. The video analysis device according to claim 1 , wherein the pulse wave detection section detects the pulse wave based on a change over time in pixel values of the normal pixels. The pixel value of the specific high luminance pixel is referred to as a specific high luminance pixel value, and the pixel value of the normal pixel is referred to as a normal pixel value,
The pulse wave detection unit includes:
A specific weighting factor, which is a weighting factor for the specific high luminance pixel value, is set to be smaller than a normal weighting factor, which is a weighting factor for the normal pixel value; and
4. The video analysis device according to claim 1, wherein the pulse wave is detected in accordance with a predetermined arithmetic formula using (i) a weighted specific high-luminance pixel value obtained by multiplying the specific high-luminance pixel value by the specific weighting coefficient, and (ii) a weighted normal pixel value obtained by multiplying the normal pixel value by the normal weighting coefficient. Further comprising a video storage unit for storing the video,
6. The video analysis device according to claim 1, wherein the pulse wave detection unit detects the pulse wave based on changes in pixel values of the plurality of pixels over time after reducing the influence of the specific high-luminance pixels in at least one frame of the video stored in the video storage unit. A video analysis device according to any one of claims 1 to 6 ,
and an imaging device that captures the image. a region of interest detection step of detecting a region of interest of the living body from an image showing the living body based on a predetermined algorithm;
and a pulse wave detection step of detecting a pulse wave of the living body based on a time change in pixel values of a plurality of pixels included in the region of interest after reducing an influence of a specific high luminance pixel having a pixel value equal to or greater than a pixel value threshold ,
The pulse wave detection step includes:
identifying pixels, excluding the specific high luminance pixel, from the plurality of pixels as normal pixels;
detecting the pulse wave when the number of the normal pixels is equal to or greater than a pixel number threshold .

Images