JP6927322B2 - Pulse wave detector, pulse wave detection method, and program - Google Patents

Description

The present disclosure relates to a technique for acquiring pulse wave information from an image.

A method is known in which the face of a person to be measured is photographed and a pulse wave is detected from a moving image obtained by the photographing. For example, Non-Patent Document 1 and Patent Documents 1 to 3 describe a method for detecting a pulse wave from a moving image of a face. The "pulse wave" is a fluctuation in the volume of blood in a blood vessel. By analyzing the pulse wave, for example, the pulse rate (usually the same as the heart rate), which is one of the pulse wave information, can be found.

In Non-Patent Document 1, independent component analysis is performed on the time change of each average value of the light brightness of the R component (Red component), G component (Green component), and B component (Blue component) detected from the facial region. A method of estimating the pulse rate after processing with (ICA: Independent Component Analysis) is described. In this method, after converting the independent component into a frequency spectrum, the pulse rate is estimated from the frequency of the peak included in the frequency spectrum.

Patent Document 1 also describes a technique for detecting a pulse rate from a spectral distribution obtained by independent component analysis.

By the way, when the person to be measured moves during shooting, if the area where the luminance value is detected (also referred to as the measurement area or the area of interest) in the image is fixed, the part of the face included in the measurement area changes. , The detected brightness value changes. That is, changes in the position and orientation of the face cause noise for the signal indicating the pulse wave. Therefore, in order to acquire pulse wave information more accurately, it is important to take measures against noise caused by changes in the position and orientation of the face.

Patent Document 2 discloses a pulse wave detection device that performs processing so that the accuracy does not decrease even when the face moves laterally by observing the detection accuracy at the boundary of the measurement region.

Patent Document 3 is a measurement that tracks the position of a "sample" (a pair of small regions composed of a plurality of pixels) for acquiring a pulse wave signal, and calculates a pulse rate from the signal of a sample determined to be reliable. The device is disclosed.

Patent Document 4 describes a pulse wave detection device that detects the rotation angle of the face orientation of the person to be measured with respect to a predetermined direction and calculates the position of the pulse wave measurement region based on the detected rotation angle. ing.

Japanese Unexamined Patent Publication No. 2012-239661 Japanese Unexamined Patent Publication No. 2014-198200 Japanese Unexamined Patent Publication No. 2017-93760 Japanese Unexamined Patent Publication No. 2014-198201

Ming-Zher Poh, Daniel J. McDuff, and Rosalind W. Picard, “Non-contact, automated cardiac pulse measurements using video imaging and blind source separation”, OPTICS EXPRESS, Vol.18, No.10, May 2010, pp. 10762-10774.

Even if the measurement area is set to be linked to the range of a specific part of the face by the techniques described in Patent Documents 2 to 4, the detection is detected by a change in the direction of the specific part or a change in the way the light hits. The brightness value extracted from the region includes noise caused by a change in the orientation of the face.

One of the objects of the present invention is to provide a device and a method for suppressing a decrease in accuracy of pulse wave information due to the movement of the face of a person to be measured.

The pulse wave detection device according to an embodiment of the present invention includes a region of interest setting means for setting an region of interest of the subject's face with respect to a time-series image of the subject's face, and the region of interest. A brightness data generating means for generating brightness data, which is time-series data of representative values of brightness values of predetermined color components, and a displacement indicating the movement of the face of the person to be measured based on the time-series image. A displacement data generating means for generating data, a pulse wave component extracting means for extracting a signal component caused by a pulse wave in the brightness data using the displacement data, and a pulse wave based on the extracted signal component. It is provided with an output means for outputting the information of.

In the pulse wave detection method according to one embodiment of the present invention, the region of interest of the subject's face is set with respect to the time-series image of the subject's face, and the pixels in the region of interest of the pulse wave detection method. Brightness data, which is time-series data of representative values of brightness values of predetermined color components, is generated, displacement data indicating the movement of the face of the person to be measured is generated based on the time-series image, and the displacement data is used. Then, the signal component caused by the pulse wave in the brightness data is extracted, and the pulse wave information based on the extracted signal component is output.

The program according to one embodiment of the present invention includes a region of interest setting process for setting an region of interest of the subject's face on a time-series image in which the subject's face is captured, and the region of interest. A brightness data generation process for generating brightness data, which is time-series data of representative values of brightness values of predetermined color components, and displacement indicating the movement of the face of the person to be measured based on the time-series image. A displacement data generation process for generating data, a pulse wave component extraction process for extracting a signal component caused by a pulse wave in the brightness data using the displacement data, and a pulse wave based on the extracted signal component. The output process that outputs the information of is executed. The program is stored, for example, in a computer-readable non-temporary storage medium.

According to the present invention, the decrease in the accuracy of pulse wave information due to the movement of the subject's face is suppressed.

It is a block diagram which shows the structure of the pulse wave detection apparatus which concerns on 1st Embodiment of this invention. It is a figure which shows the example of the luminance data. It is a figure which shows the example of the displacement data. It is a figure which conceptually shows the example of the method of generating the correction data from the luminance data. It is a figure which conceptually shows the method of performing pulse estimation using independent component analysis. It is a flowchart which shows the operation flow of the pulse wave detection apparatus which concerns on 1st Embodiment. It is a block diagram which shows the structure of the modification of the pulse wave detection apparatus which concerns on 1st Embodiment. It is a block diagram which shows the structure of the pulse wave detection apparatus which concerns on 2nd Embodiment of this invention. It is a block diagram which shows the structure of the pulse wave detection apparatus which concerns on 3rd Embodiment of this invention. It is a figure for demonstrating an example of the heart rate estimation method which concerns on 3rd Embodiment. It is a block diagram which shows the structure of the pulse wave detection apparatus which concerns on one Embodiment of this invention. It is a flowchart which shows the operation flow of the pulse wave detection apparatus which concerns on one Embodiment. It is a block diagram which shows the example of the hardware which comprises each part of each embodiment of this invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

<< First Embodiment >>
First, the first embodiment of the present invention will be described. The pulse wave detection device 11 according to the first embodiment is a device that detects a pulse wave. Specifically, detecting a pulse wave means finding a signal that changes due to a pulse wave from data including a signal that changes due to the pulse wave, and deriving information on the pulse wave. Examples of pulse wave information include time series data of signals that change with the pulse wave, and pulse rate. Hereinafter, detecting a pulse wave is also referred to as pulse wave detection.

[composition]
FIG. 1 is a block diagram showing the configuration of the pulse wave detection device 11. In the first embodiment, the pulse wave detection device 11 and the image pickup device 30 are communicably connected. The method of communication between the pulse wave detection device 11 and the image pickup device 30 may be any communication method (for example, communication via a cable or wireless communication).

The pulse wave detection device 11 includes a storage unit 110, an area of interest setting unit 111, a luminance data generation unit 112, a displacement data generation unit 113, a pulse wave information derivation unit 114, and an output unit 119.

The region of interest setting unit 111, the luminance data generation unit 112, the displacement data generation unit 113, the pulse wave information derivation unit 114, and the output unit 119 are realized by, for example, one or a plurality of CPUs (Central Processing Units) that execute programs. Will be done.

<Image pickup device 30>
The image pickup device 30 is an image pickup device equipped with an image pickup device such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor) image sensor. For example, the image pickup apparatus 30 is equipped with a plurality of light receiving elements for measuring the intensity of light such as an R (Red) component, a G (Green) component, and a B (Blue) component. The image pickup device 30 may be a camera integrally mounted on the pulse wave detection device 11 such as an in-camera and an out-camera mounted on an existing mobile terminal device. Alternatively, the image pickup device 30 may be a digital camera, a webcam, or the like connected to the pulse wave detection device 11 via an external terminal. The image pickup device 30 is not limited to a visible light camera, but may be a near-infrared camera, an infrared camera, or the like. In the following description, a color image in which the color intensities of the R component, the G component, and the B component are described is mainly assumed, but the image may be a gray scale image. A grayscale image is considered to be an image that expresses the intensity of one color. Therefore, when the image is a grayscale image, the process described below as the process for any of the components R, G, and B may be interpreted as corresponding to the process for the brightness value of the grayscale image. ..

The image pickup device 30 takes a moving image of the face of a person whose pulse wave is to be detected. The imaging device 30 acquires a moving image by continuously imaging a target person at a predetermined frame rate for a certain period of time (for example, several seconds or several minutes). Hereinafter, the data of a plurality of time-continuous facial images (in other words, time-series images) obtained by imaging will be referred to as moving image data.

As an example, each frame of the moving image data is a rectangle having a width of 640 pixels and a height of 480 pixels. Each pixel included in the frame has color luminance information. Luminance information is given as a gradation value. For example, when the moving image is a grayscale moving image, the pixels have a gradation value representing the brightness. When the moving image is a color moving image, the pixel has, for example, the gradation values of the respective luminances of the three primary colors R component, G component, and B component. The gradation value is represented by, for example, 8-bit information. It should be noted that the gradation value of the pixel is not based on the RGB value based on the RGB color system, but based on another color system (HSV color system, YUV color system, etc.) that is in a mutually convertible relationship with the RGB value. It may be represented by a value.

In the following description, the gradation value of the brightness of each color given to the pixel is also referred to as a “pixel value” or a “luminance value”.

As a modification, the image pickup device 30 may be replaced with a device other than the image pickup device that outputs moving image data to the pulse wave detection device 11. Such a device is, for example, a reading device or a storage device that reads moving image data from a storage medium that stores moving image data. Further, as another modification, the pulse wave detection device 11 may read moving image data from a storage medium such as a flash memory. Alternatively, the pulse wave detection device 11 may read out the moving image data stored in the storage area in the pulse wave detection device 11.

<Structure of each part of pulse wave detection device 11>
The time range of the moving image data to be processed by each part of the pulse wave detection device 11 is hereinafter referred to as “target data range”. The target data range is the length of a part or all of the acquired moving image data. For example, the pulse wave detection device 11 has a target data range of 30 seconds. When the target data range is shorter than the moving image data, the pulse wave detection device 11 may execute a process related to pulse wave detection for each target data range, with the plurality of ranges as different target data ranges. .. The plurality of ranges in this case may be ranges in which some of them overlap each other.

<Memory unit 110>
The storage unit 110 stores the information used by the pulse wave detection device 11 and the information generated by the processing by the pulse wave detection device 11. The storage unit 110 is, for example, a memory. The storage unit 110 may be an auxiliary storage device such as a hard disk.

The storage unit 110 may store, for example, moving image data output from the image pickup apparatus 30.

<Output unit 119>
The output unit 119 outputs the information generated by the pulse wave detection device 11. Examples of output destinations for output by output unit 119 include display devices, storage devices, and communication networks. When the output unit 119 outputs the information to the display device, the output unit 119 may convert the information so that the display device can display the information. The display device and the storage device described above may be devices external to the pulse wave detection device 11 or may be components included in the pulse wave detection device 11.

<Interest area setting unit 111>
The region of interest setting unit 111 sets the region of interest (Region Of Interest). The region of interest in the present disclosure is a region in the image that is the target of image processing by the pulse wave detection device 11. The region of interest is also referred to as ROI (Region Of Interest). The pulse wave detection device 11 uses the pixel values of the pixels included in the region of interest to detect the pulse wave information.

The region of interest setting unit 111 may set the region of interest, for example, according to a predetermined algorithm (that is, automatically). For example, the region of interest setting unit 111 may detect feature points on the face in a typical frame and set the region of interest according to a predetermined algorithm based on the coordinates of the feature points.

For the detection of the facial feature points, an existing facial feature point detection method may be used. Examples of the face feature point detection method include a detection method that combines detection of Haar-like features and learning by AdaBoost (Adaptive Boosting).

The predetermined algorithm may be any algorithm as long as it is an algorithm for setting a region using feature points. As an example, the region of interest setting unit 111 is a triangular region having "a feature point corresponding to the right end of the mouth, a feature point corresponding to the right end of the nose, and a feature point corresponding to the lower center of the right eye" as vertices. The specified region may be set as the region of interest according to the algorithm of "setting the region of interest".

The above-mentioned typical frame selection method may be any method. For example, the first frame among the frames included in the target data range may be selected as a representative frame.

The area of interest setting unit 111 may receive an input for designating the area of interest and set the area of interest based on the input. In such a case, the region of interest setting unit 111 may have an input interface. The output unit 119 may display all or a part of the moving image data on the display device, and a person may input information for designating the region of interest via the input interface while referring to the displayed information. The area of interest setting unit 111 may receive information specifying the area of interest from the input interface and set the area of interest based on the received information. The information for designating the region of interest is represented by, for example, information representing a figure. As an example, by selecting two points in the image, the region of interest setting unit 111 has a rectangle whose two sides are parallel to the horizontal direction of the image, with the line segment connecting the two points as a diagonal line. Area may be set as an area of interest.

When the pulse wave detection device 11 can recognize parts such as the "forehead" and "cheek", the information for designating the region of interest may be information for designating the facial parts. In such a case, the region of interest setting unit 111 may specify the region occupied by the designated facial parts in a typical frame, and set that region as the region of interest.

The region of interest setting unit 111 may set a plurality of regions specified by different procedures as regions of interest.

<Brightness data generator 112>
The luminance data generation unit 112 receives the moving image data and the region of interest set by the region of interest setting unit 111 as inputs, and generates the luminance data. The luminance data is, in the present embodiment, time-series data of the luminance value.

The luminance data generation unit 112 generates variable data of the luminance value obtained from the region of interest for each frame. For example, the luminance data generation unit 112 calculates a representative value of the luminance value of each color of each pixel included in the region of interest of each frame. As a result, the luminance data generation unit 112 generates time-series data of representative values of the luminance values of each color. When the pixel value of the moving image data is represented by the respective brightness values of the R component, the G component, and the B component, the brightness data generation unit 112 determines the brightness values of the R component, the G component, and the B component, respectively. Generate time-series data of representative values. Hereinafter, the representative value of the luminance value is also referred to as a “luminance representative value”.

The representative value is a value that represents the pixel value of each pixel in the region of interest. The representative value is, for example, an average value, a median value, a mode value, or the like. The average value may be an arithmetic mean, a geometric mean, or a harmonic mean.

Let the t-th (t = 1, 2, 3, ...) Frame in the moving image data be the frame t. Let the luminance values of the R component, G component, and B component at the coordinates (i, j) of the frame t be R _t (i, j), G _t (i, j), and B _t (i, j), respectively. .. When the region of interest and R _OI, as an example, the luminance representative value G _t _Rep the G component of a pixel in a region of interest, are calculated by the following equations.

However, the symbol "S", the area of the region of interest R _OI, i.e., the number of pixels included in the region of interest R _OI. This calculation formula is a calculation formula when the arithmetic mean of each luminance value is adopted as the representative luminance value. In particular, when the region of interest is rectangular and is a set of points (i, j) satisfying _{i s} ≤ i ≤ i _e and j _s ≤ j ≤ j _{e, Gt_rep is calculated by the following equation.}

The calculation method of the luminance representative value R _t _rep of the R component and the luminance representative value B _t _rep of the B component is also the same.

The luminance data generation unit 112 calculates the _{luminance representative values R t} _rep, G _t _rep, and B _t _rep for each frame included in the target data range. The luminance data generation unit 112 generates time-series data R_rep, G_rep, and B_rep of the luminance representative values of each color by arranging the luminance representative values of each frame in time series.

FIG. 2 is a diagram showing an example of a graph of luminance data G_rep. The example of FIG. 2 is an example of luminance data G_rep generated for an image for 30 seconds (that is, 901 frames including endpoints) acquired at a frame rate of 30 frames / second. The value on the horizontal axis is the number of the frame included in the target data range, and the value on the vertical axis is the value (Intensity) of _{the luminance representative value G t _rep.} The value on the horizontal axis of the luminance data may be expressed in time (range is 0 to 30 seconds). Luminance data is time series data showing pulse waves. The luminance data itself generated by the luminance data generation unit 112 is one of the pulse wave information.

<Displacement data generator 113>
The displacement data generation unit 113 accepts moving image data as an input and generates displacement data in the target data range. The displacement data is data showing the movement of the face of the person to be measured. The displacement data is time-series data of signals that fluctuate according to changes in the position of the subject's face in the image and changes in the orientation with respect to the image.

The displacement data generation unit 113 generates displacement data based on, for example, the coordinate values in the image of at least one feature point. More specifically, for example, the displacement data generation unit 113 specifies the coordinate values in the lateral direction of a certain feature point for each frame. Then, the displacement data generation unit 113 generates displacement data by arranging the coordinate values in chronological order. The method of extracting the feature points from the face may be the same as the method already described.

FIG. 3 is an example of displacement data generated according to the above example. The vertical axis is the coordinate value "x" indicating the horizontal position of the target feature point in the image. If the image has a length of 640 pixels in the horizontal direction, the range of "x" is, for example, in the range 0 to 639.

The feature point used is, for example, one of the feature points on the contour of the nose. The nose is the part closest to the imaging device 30 when the subject faces the imaging device 30. In addition, the distance from the axis of rotation of the nose when the subject rotates his / her face is farther than that of other parts. Therefore, the coordinate values of the feature points of the nose are more likely to reflect changes in the orientation of the face than the coordinate values of other parts.

When the person to be measured is photographed so that the lateral direction in the image and the left-right direction for the person to be measured coincide with each other, the horizontal direction of the image coincides with the left-right direction for the person to be measured. According to the findings of the inventors, the fluctuation of the coordinate value in the horizontal direction generates a signal (that is, noise) that interferes with the information of the pulse wave rather than the fluctuation of the coordinate value in the vertical direction for the subject. It tends to be easy.

In the present disclosure, the "left-right direction" for the person to be measured is a direction perpendicular to the front direction and the vertical direction of the face when the person to be measured stands upright. The left-right direction corresponds to, for example, a direction parallel to a straight line passing through both eyes of the person to be measured. The left-right direction is, for example, perpendicular to the length direction of the neck of the subject.

If it is not assumed that the person to be measured is photographed so that the lateral direction in the image and the left-right direction for the person to be measured coincide with each other, the displacement data generation unit 113 will perform the left-right direction for the person to be measured in the image. After specifying the direction corresponding to the direction, the coordinate value indicating the position in that direction may be acquired. For example, the displacement data generation unit 113 detects the positions of both eyes of the person to be measured in a typical frame, and specifies the direction of the straight line passing through both eyes as the direction corresponding to the left-right direction for the person to be measured in the image. Just do it.

Since fluctuations in the coordinate values in the vertical direction can also cause noise, the displacement data generation unit 113 may generate displacement data using the fluctuations in the coordinate values in the vertical direction. For example, the displacement data generation unit 113 may generate time-series data of coordinate values in the vertical direction as displacement data.

The method of generating displacement data is not limited to the above example. For example, the displacement data generation unit 113 may generate time-series data indicating the transition of the two-dimensional position of a certain feature point as the displacement data. Further, the displacement data generation unit 113 may generate time-series data indicating the transition of the distance of a certain feature point from the reference position as the displacement data. In such a case, the reference position is, for example, a point specified by the mode of the coordinate values in the horizontal direction and the mode of the coordinate values in the vertical direction.

The displacement data generation unit 113 may generate displacement data based on changes in the positions of a plurality of feature points. In such a case, the displacement data generation unit 113 integrates the information on the positions of the plurality of feature points and generates time-series data indicating the transition of one variable based on the change in the positions of the plurality of feature points as the displacement data. You may.

For example, the displacement data generation unit 113 may extract a plurality of points from the feature points of the eyes, nose, and mouth, and acquire the transition of the coordinate values in the lateral direction of each of the plurality of points. Then, the displacement data generation unit 113 may generate time-series data indicating the transition of the average of the coordinate values of the plurality of points as the displacement data.

As described in Patent Document 3, the displacement data generation unit 113 obtains the rotation angle of the face in each frame based on the change in the position of a plurality of feature points, and displaces the time-series data indicating the transition of the rotation angle. It may be generated as data.

<Pulse wave information derivation unit 114>
The pulse wave information derivation unit 114 derives pulse wave information based on the luminance data and the displacement data.

The pulse wave information derivation unit 114 in the first embodiment includes a correction unit 1141 and a pulse rate estimation unit 1142. The correction unit 1141 corrects the luminance data based on the displacement data. Then, the pulse rate estimation unit 1142 estimates the pulse rate of the person to be measured based on the correction data which is the corrected luminance data.

Hereinafter, a specific example of a method in which the correction unit 1141 corrects the luminance data will be described.

The correction unit 1141 estimates, for example, the magnitude of the influence of the magnitude of the displacement on the luminance data by the following procedure.

The correction unit 1141 obtains the displacement difference data, which is the data representing the difference from the reference value of the displacement data, from the displacement data. The displacement difference data is time-series data of the deviation of the data value in the displacement data from the reference value of the displacement data. When the data values in the displacement data as x (t), the reference value of the displacement data and mu _x, data value dx in the displacement difference data (t) is dx (t) = x (t) - [mu] _x. The reference value μ _x is, for example, the average value of the data values x (t) in the measurement range. The correction unit 1141 calculates dx (t) at each time and obtains time-series data of dx (t).

Further, the correction unit 1141 obtains the luminance difference data, which is the data representing the difference from the reference value of the luminance data, from the luminance data. The luminance difference data is time-series data in which the data value deviates from the reference value of the luminance data. Assuming that the data value in the luminance data is I (t) and the reference value of the luminance data is μ _I , the data value d I (t) in the displacement difference data is dI (t) = I (t) −μ _I. The reference value μ _I is, for example, the average value of the data values I (t) in the measurement range. The correction unit 1141 calculates dI (t) at each time and obtains time-series data of dI (t).

The correction unit 1141 may obtain luminance difference data for each luminance data for each color component. The correction unit 1141 may obtain the luminance difference data for any one of the luminance data. In the following example, it is assumed that the correction unit 1141 obtains the luminance difference data dR, dG, and dB based on the luminance data of each of the R component, the G component, and the B component.

After obtaining the displacement difference data and the luminance difference data, the correction unit 1141 obtains a coefficient α representing the relationship between the displacement difference data and the luminance difference data.

A specific example is shown. Assuming that the relationship between the magnitude of the displacement difference and the magnitude of the effect on the luminance data (that is, the amount of change in the luminance value) is linear, the data representing the change in luminance due to the displacement is It is expressed by multiplying the data value of the displacement difference data by a constant. Therefore, the correction unit 1141 obtains a value of α such that the mean square error between the value obtained by multiplying each of the data values of the displacement difference data by α and the data value of the luminance difference data is minimized. .. Of the values that can be considered as values to be multiplied by each of the data values of the displacement difference data, α obtained in this way is the time-series data of the data values obtained by multiplying each of the data values of the displacement difference data. And the value most similar to the brightness data. α is, so to speak, a value indicating the degree of influence of displacement on the luminance data. α is a coefficient for converting a coordinate value into a luminance value.

The time-series data obtained by multiplying the data value of the displacement difference data by α is data representing the change in brightness due to the displacement. Therefore, the correction unit 1141 obtains correction data corrected for the change in brightness due to displacement by subtracting the product of the data value of the displacement difference data and α for each frame with respect to the data value of the brightness data. ..

FIG. 4 is a diagram conceptually showing an example of a method of generating correction data from luminance data. In this example, the correction unit 1141 generates the difference between the "luminance data" and the "α times the displacement difference data" as "correction data".

The correction data generated by the correction unit 1141 is data in which components caused by changes in the position and orientation of the face are suppressed from the luminance data. That is, the correction data is the pulse wave information from which the signal caused by the pulse wave is extracted (in other words, emphasized) from the luminance data. The pulse wave detection device 11 may output correction data by the output unit 119.

The method of correcting the luminance data based on the displacement data is not limited to the above example, but examples other than the above example will be described later in the description of [Modification Example 1-2].

The pulse wave detection device 11 of the present embodiment further estimates the pulse rate of the person to be measured by the pulse rate estimation unit 1142. Hereinafter, an example of a method for estimating the pulse rate by the pulse rate estimation unit 1142 will be described.

The pulse rate estimation unit 1142 executes a conversion process for converting the corrected data into a frequency spectrum. Examples of the conversion process to the frequency spectrum include a discrete Fourier transform (DFT) and a discrete cosine transform (DCT).

The pulse rate estimation unit 1142 generates a frequency spectrum by a conversion process. In the following description, a power spectrum is assumed as one aspect of the frequency spectrum.

When there are a plurality of correction data (for example, correction data for each of the R component, the G component, and the B component), the pulse rate estimation unit 1142 may generate a frequency spectrum for each of the plurality of correction data. .. Alternatively, the pulse rate estimation unit 1142 generates one time-series data by integrating a plurality of correction data (for example, summing the data values for each frame), and then obtains the frequency spectrum of the generated time-series data. It may be generated. Alternatively, the pulse rate estimation unit 1142 may select reliable correction data from the plurality of correction data and generate a frequency spectrum of the selected correction data. Examples of reliable correction data include correction data of G component, correction data having the smallest variance of data values, and the like. The reason why the corrected data with the smallest variance of the data value can be said to be reliable is that in the luminance data before correction, the fluctuation value of the signal derived from noise is usually larger than the fluctuation value of the signal derived from the pulse wave. be. That is, the smaller variance of the data values means that the noise can be reduced more appropriately by the correction.

The pulse rate estimation unit 1142 estimates the pulse based on the generated frequency spectrum. Pulse estimation is to derive an estimated value of pulse rate.

Specifically, the pulse rate estimation unit 1142 first receives a frequency domain (about 50 to 180 [bpm (beats per minute)]) corresponding to a range (for example, 50 to 180 [bpm (beats per minute)]) of the frequency spectrum in which the pulse rate of the person to be measured can be taken. In 0.83 to 3 [Hz]), the frequency at which the intensity has the maximum value is specified. Then, the pulse rate estimation unit 1142 determines the specified frequency as the pulse rate. The unit of the pulse rate may be [bpm]. The pulse rate estimation unit 1142 may convert a numerical value indicating the pulse rate according to the unit. When the unit is [bpm], the value of the pulse rate may be a real number obtained by multiplying the value when the unit is [Hz] by 60, or may be an integer closest to the real number.

When there are a plurality of frequency spectra such as frequency spectra generated for different color components, the pulse rate estimation unit 1142 may derive the pulse rate from each of the plurality of frequency spectra by the above-mentioned method. In such a case, since a plurality of pulse rates are derived, the pulse rate estimation unit 1142 determines a representative value (average value, median value, etc.) of the derived plurality of pulse rates as the estimated pulse rate. You may. Since the G component is absorbed into the blood better than the other color components, the pulse rate estimation unit 1142 calculates the weighted average value after giving a larger weight to the G component than the other color components. It may be calculated as a representative value.

The pulse wave information derivation unit 114 sends information indicating the pulse rate derived by pulse estimation to the output unit 119. The information indicating the pulse rate is, for example, an electrical signal indicating the numerical value of the pulse rate, text format data indicating the numerical value of the pulse rate, an image including a display of the numerical value of the pulse rate, and the like.

[Modification 1-1]
A modified example of the pulse estimation method will be described. The pulse rate estimation unit 1142 derives an independent component by performing an independent component analysis as described in Non-Patent Document 1 on the correction data related to each color component (that is, R component, G component, B component). You may. FIG. 5 is a diagram conceptually showing a method of performing pulse estimation using independent component analysis. As shown in FIG. 5, the component data (time series) showing the three independent components can be obtained from the three correction data by the independent component analysis. The pulse rate estimation unit 1142 obtains three frequency spectra (“component data (frequency)” in FIG. 5) by converting each of the obtained three component data into a frequency spectrum. The pulse rate estimation unit 1142 derives the pulse rate from the frequency spectrum thus obtained.

The method for deriving the pulse rate from the frequency spectrum of the independent component is as follows, for example. The pulse rate estimation unit 1142 identifies the maximum peak in the frequency region corresponding to the range in which the pulse rate of the subject can be taken in the three frequency spectra (“peak indicating the pulse rate” in FIG. 5). The pulse rate estimation unit 1142 determines the frequency of the identified peak as the estimated pulse rate.

[motion]
The operation of the pulse wave detection device 11 will be described. FIG. 6 is a flowchart showing the operation flow of the pulse wave detection device 11. When each process is executed by the device that executes the program, each process may be executed in the order of instructions in the program. When each process is executed by a separate device, the device that has completed the process may notify the device that executes the next process to execute the next process. It should be noted that each unit performing the processing may receive the data necessary for each processing from the unit that generated the data or read it from the storage unit 110.

First, the pulse wave detection device 11 acquires moving image data from the image pickup device 30 (step S11).

Then, the region of interest setting unit 111 sets the region of interest (step S12).

Then, the luminance data generation unit 112 generates the luminance data from the luminance values of the pixels included in the region of interest of each frame (step S13).

The displacement data generation unit 113 generates displacement data from the position of a specific feature point of each frame (step S14).

When the luminance data and the displacement data are generated, the correction unit 1141 of the pulse wave information derivation unit 114 corrects the luminance data based on the displacement data (step S15). Then, the pulse rate estimation unit 1142 of the pulse wave information derivation unit 114 estimates the pulse rate based on the corrected data (step S16).

Then, the output unit 119 outputs information indicating the estimated pulse rate (step S17).

The pulse wave detection device 11 may generate time-series data in real time when the moving image data is acquired. That is, the pulse wave detection device 11 may execute the processing for each frame in the processing of steps S12 to S14 at any time regardless of whether or not the moving image data has a sufficient length for analysis. In such a case, the storage unit 110 stores the luminance data and the displacement data. Then, when the data for a predetermined number of frames is accumulated, the pulse wave detection device 11 may execute the processes after step S15.

The order of the above processing is an example. The order of each process may be freely changed as long as it does not deviate from the technical idea disclosed in the present disclosure.

[effect]
According to the pulse wave detection device 11 according to the first embodiment, the decrease in the accuracy of the pulse wave information due to the movement of the face of the person to be measured is suppressed.

The reason is that the correction unit 1141 of the pulse wave information derivation unit 114 corrects the luminance data based on the displacement data, so that the luminance data in which the data representing the pulse wave is emphasized can be obtained.

According to the tests of the inventors, the pulse rate derived by using the correction data generated by the correction that the time series data obtained by multiplying the above-mentioned coefficient α by the displacement difference data is subtracted from the luminance data. Is known to have better accuracy than the case where the above correction is not performed.

[Modification 1-2]
An example other than the above-mentioned example of the method of correcting the luminance data based on the displacement data will be described.

The correction unit 1141 may obtain a β value such that the time series data obtained by multiplying each data value dx (t) of the displacement difference data to the β power is most similar to the luminance data. The method for determining similarity will be described later. Then, the correction unit 1141 corrects by subtracting the data value of the time series data obtained by multiplying each data value dx (t) of the displacement difference data by β-th power from the data value of the brightness data for each frame. Data may be generated.

More generalized, the correction unit 1141 prepares a plurality of functions f (dx (t)) for processing the data value dx (t) of the displacement difference data, and each of the functions is subjected to the data value dx (dx (t)) of the displacement difference data. By applying to t), a plurality of time series data may be generated, and among the generated time series data, a function that generates time series data most similar to the brightness data may be specified. Then, the correction unit 1141 generates correction data by subtracting the time series data generated by applying the specified function to the data value dx (t) of the displacement difference data from the data value of the luminance data. You may.

The plurality of functions f (dx (t)) described above may be a function of a predetermined type including parameters, and may have different parameter values. For example, the method already described in the description of the first embodiment corresponds to using a plurality of functions f (dx (t)) = dx (t) × α in which the values of the parameters α are different.

Another example of the function f (dx (t)) that processes the data value dx (t) is as follows.
・ F (dx (t)) = {dx (t)} ^ β (however, “^” represents a power)
・ F (dx (t)) = {dx (t)} ^ β / γ
F (dx (t)) = dx (t) / (dx (t) + β)
However, the above is just an example.

Incidentally, since it is dx (t) = x (t ) -μ x, in the above description, the function for processing the data value dx of the displacement difference data (t) f (dx (t )) , the data of the displacement data It may be read as a function f (x (t)) that processes the value x (t). In that case, an example of the function f (x (t)) is as follows.
F (x (t)) = (x (t) -μ _x ) x α
・ F (x (t)) = (x (t) -μ _x ) ^ β
F (x (t)) = (x (t) -μ _x ) / (x (t) -μ _x + β)
F (x (t)) = (x (t) -μ _x ) ^ β / γ
In the correction unit 1141, for example, among a plurality of predetermined functions determined by the type of function and the value of the parameter as described above, the time series data generated by processing the data value of the displacement data is most similar to the brightness data. , Identify a given function. The time-series data generated by processing using the specified predetermined function is time-series data showing the change in the luminance value due to the movement of the face of the person to be measured, and is also referred to as "noise time-series data" in the present disclosure. Refer to. Then, the correction unit 1141 generates correction data by subtracting the noise time series data from the data value of the luminance data.

Hereinafter, among the time series data obtained by applying each of the plurality of functions to the data value of the displacement data (hereinafter referred to as "correction candidate time series data"), "time series data most similar to the brightness data". An example of how to identify is described.

The correction unit 1141 may specify the correction candidate time series data having the smallest mean square error of the data values in the target data range as the "time series data most similar to the brightness data" with the brightness difference data. .. However, depending on the function, the correction unit 1141 may use the mean square error with the luminance data.

The correction unit 1141 selects the correction candidate time series data in which the difference between the data value at a specific time point in the correction candidate time series data and the data value at the specific time point in the brightness difference data is the smallest, "brightness data. It may be specified as "time series data most similar to". The specific time point is, for example, a time point corresponding to the data value having the largest absolute value in the correction candidate time series data. When the signal caused by noise is stronger than the signal caused by pulse wave, it is effective to refer to the data value having the largest absolute value in this way to identify the magnitude of the noise signal.

The correction unit 1141 has the correction candidate time series in which the square error of the plurality of data values corresponding to the plurality of time points in the correction candidate time series data with respect to the plurality of data values corresponding to the plurality of time points in the brightness displacement data is the smallest. The data may be specified as "time series data most similar to the brightness data". The plurality of time points may be extracted according to an arbitrary algorithm for extracting the plurality of time points, for example. For example, the correction unit 1141 may extract the time points corresponding to P data values (P is a set value) in descending order of the absolute value as the plurality of time points.

[Modification 1-3]
The pulse wave detection device 11 may set a plurality of target data ranges from the acquired moving image data and calculate the pulse rate for each of the plurality of target data ranges.

For example, the pulse wave detection device 11 sets the range from the first frame to the Nth frame (N is a set value) of the acquired moving image data as the first target data range, and sets k × M. The range from the first frame (M is a set value, k = 1, 2, ...) To the N + k × Mth frame may be set as the k-th target data range. Then, pulse wave information may be derived for each target data range, and a total of k estimated pulse rate values may be calculated.

Then, the pulse wave detection device 11 may output a representative value of the estimated values of k pulse rates as the estimated pulse rate. The type of the representative value does not matter, but as an example, it is the average value or the median value. In the pulse wave detection device 11, the representative value may be the mode value as long as the estimated value of k pulse rates is an integer value.

In such a configuration, the pulse wave detection device 11 may include an outlier removing unit 115 as shown in FIG. 7. FIG. 7 shows the configuration of the pulse wave detection device 11a including the outlier removing unit 115. The configuration of the pulse wave detection device 11a may be the same as the configuration of the pulse wave detection device 11 except that the outlier removing unit 115 is provided.

The outlier removing unit 115 removes the estimated value specified as the outlier from the plurality of calculated estimated values. The method of identifying outliers may be a method already known. The pulse wave detection device 11a obtains a representative value of the remaining estimated values obtained by removing outliers from the plurality of calculated estimated values, and outputs the representative values as the estimated pulse rate. By removing the outliers, the pulse rate can be estimated more accurately.

<< Second Embodiment >>
Hereinafter, as a second embodiment, an embodiment in which the pulse wave detection device derives pulse wave information using a plurality of regions of interest will be described.

FIG. 8 is a block diagram showing the configuration of the pulse wave detection device 12 according to the second embodiment. Similar to the pulse wave detection device 11a of the modified example of the first embodiment, the pulse wave detection device 12 has a storage unit 110, an area of interest setting unit 111, a brightness data generation unit 112, a displacement data generation unit 113, and an outlier removal unit. It includes 115 and an output unit 119. Then, the pulse wave detection device 12 includes a pulse wave information derivation unit 124 instead of the pulse wave information derivation unit 114.

The pulse wave information derivation unit 124 includes a correction unit 1241, an association unit 1242, a synthesis unit 1243, and a pulse rate estimation unit 1244.

The function of the correction unit 1241 may be the same as the function of the correction unit 1141 of the first embodiment.

However, in the second embodiment, the region of interest setting unit 111 sets a plurality of regions of interest. Then, the pulse wave detection device 12 generates luminance data and displacement data for each of the plurality of regions of interest. The correction unit 1241 of the pulse wave information derivation unit 124 generates correction data for each of the plurality of regions of interest.

The correction data of each of the plurality of regions of interest is integrated into one time-series data by the processing of the association unit 1242 and the synthesis unit 1243 described below.

<Association unit 1242>
The association unit 1242 associates each of the plurality of areas of interest with a reliability indicating a high degree of reliability. The reliability is expressed by a numerical value, for example, and the higher the reliability value, the higher the reliability.

The reliability indicated by the reliability is the reliability as a region where the pulse wave information can be accurately obtained. That is, the reliability indicates how reliable the fluctuation of the luminance value acquired from the region of interest to which the reliability is associated is as the data representing the pulse wave.

Specifically, the association unit 1242 associates the reliability with each region of interest so that the region of interest with a smaller variance of the data value of the correction data has a higher reliability. The size of the distribution of data values, for example, the evaluation formula Σ _{t = 1 → t_max} distribution value | only to be evaluated by the ^{_{2 / t_max | V E -V t}} . However, t_max is the number of frames included in the target data _{range, V t (t = 1,2,3,} ..., t_max) is t-th frame of data values, _{V E} is the reference value of the data values (e.g. _{V t} Mean, median or mode).

Further, the associating unit 1242 may uniformly associate the lowest value (for example, "0") as the reliability with respect to the region of interest in which the variance exceeds a predetermined reference (for example, the variance value exceeds the predetermined value). ..

The associating unit 1242 may use a method of associating the first numerical value D1 when the dispersion value exceeds a predetermined threshold value and associating the second numerical value D2 when the dispersion value does not exceed the predetermined threshold value. In such a case, D1 is a value smaller than D2.

After associating the reliability with all the regions of interest, the association unit 1242 may normalize each reliability so that the sum of the reliabilitys of each region of interest is 1.

<Synthesis unit 1243>
The synthesis unit 1243 synthesizes one time series data from the correction data of the plurality of interest regions based on the reliability of the plurality of interest regions.

When there is correction data for each color component, the compositing unit 1243 may synthesize the correction data for each color component. That is, the synthesis unit 1243 may generate, for example, one R component time series data, one G component time series data, and one B component time series data.

An example of a specific processing flow of synthesis using correction data of a plurality of regions of interest for one color component will be described below.

First, the synthesis unit 1243 weights the correction data of each region of interest based on the reliability. Specifically, for example, the synthesis unit 1243 obtains weighted correction data by multiplying each data value of the correction data of each region of interest by the reliability of the region of interest.

Then, the synthesis unit 1243 integrates each of the weighted correction data. For example, the synthesis unit 1243 obtains one time-series data by adding the data values of the weighted correction data for each frame. Alternatively, the synthesis unit 1243 may obtain one time-series data by calculating the average (regardless of the type of average) of each weighted correction data.

<Pulse rate estimation unit 1244>
The function of the pulse rate estimation unit 1244 may be the same as that of the pulse rate estimation unit 1142 of the first embodiment.

[effect]
According to the pulse wave detection device 12 according to the second embodiment, in addition to the effect in the first embodiment, pulse wave information can be acquired with higher accuracy. The reason is that the association unit 1242 associates the reliability with the region of interest so that the contribution of the data obtained from the highly reliable region of interest is large, and the synthesis unit 1243 associates the reliability with the time series data of the pulse wave based on the reliability. Is generated. This pulse wave time series data is one of the pulse wave information. Further, the estimated value of the pulse rate calculated based on the time series data of this pulse wave is expected to be more accurate.

<< Third Embodiment >>
As a third embodiment of the present invention, an embodiment in which the method for deriving the pulse rate is different from that of the first embodiment will be described.

FIG. 9 is a block diagram showing the configuration of the pulse wave detection device 13 of the third embodiment. Similar to the pulse wave detection device 11 of the first embodiment, the pulse wave detection device 13 includes a storage unit 110, an area of interest setting unit 111, a luminance data generation unit 112, a displacement data generation unit 113, and an output unit 119. Then, the pulse wave detection device 13 includes a pulse wave information derivation unit 134 instead of the pulse wave information derivation unit 114.

The pulse wave information derivation unit 134 derives pulse wave information from the luminance data and the displacement data. The pulse wave information derivation unit 134 includes a data analysis unit 1341 and a pulse rate estimation unit 1342.

The data analysis unit 1341 performs an independent component analysis on a plurality of luminance value data obtained in the same time range, and generates a plurality of component data.

The pulse rate estimation unit 1342 estimates the pulse rate based on the component data and the displacement data generated by the data analysis unit 1341.

The method of estimating the pulse rate using the component data obtained by the independent component analysis is usually in the range of the values that can be taken as the pulse rate in any of the component data, as disclosed in Non-Patent Document 1. The peak frequency is determined as the pulse rate. If a total of two or more peaks in the range of possible pulse rate values are found in multiple component data, one of the peak frequencies must be selected as the pulse rate estimate. be. However, in such a case, it may not be obvious which peak corresponds to the pulse rate. Any of the plurality of peaks is a peak derived from a noise component. If such a peak is determined as the pulse rate, an erroneous estimate will be output.

The pulse rate estimation unit 1342 of the present embodiment identifies a peak estimated to be a noise-derived peak based on the frequency spectrum of the displacement data, and estimates the pulse rate from peaks other than the peak. The data analysis unit 1341 may generate the frequency spectrum of the displacement data.

A specific example of the method of estimating the pulse rate will be described with reference to the example of FIG.

FIG. 10 shows an example of component data of three independent components. With respect to these three component data, the pulse rate estimation unit 1342 extracts, for example, a peak having a predetermined intensity or higher in a frequency region corresponding to a range in which the pulse rate of the person to be measured can be taken. As an example, it is assumed that one peak is extracted from each of the component data of the second component and the component data of the third component. These two peaks (denoted as "first candidate" and "second candidate", respectively) are candidates for peaks indicating the pulse rate.

The pulse rate estimation unit 1342 compares the peak in the frequency spectrum of the displacement data with the plurality of extracted peaks. Then, the pulse rate estimation unit 1342 estimates that the peak has the same frequency as the peak frequency existing in the frequency spectrum of the displacement data, and excludes it from the peak candidates indicating the pulse rate. For example, as shown in FIG. 10, it is assumed that there is a peak having the same frequency as the frequency of the second candidate in the frequency spectrum of the displacement data. In this case, the pulse rate estimation unit 1342 excludes the second candidate from the candidates for the peak indicating the pulse rate. Therefore, the pulse rate estimation unit 1342 selects the first candidate as the peak indicating the pulse rate.

Two peaks of the same frequency do not necessarily mean only two peaks with exactly the same frequency. The pulse rate estimation unit 1342 may consider two peaks whose frequency difference is within a predetermined threshold value as "peaks of the same frequency". The pulse rate estimation unit 1342 may consider the peak of the frequency closest to the peak frequency in the frequency spectrum of the displacement data among the plurality of candidates as the “peak of the same frequency”.

If there are three or more candidates for the peak indicating the pulse rate, and if two or more candidates still remain even after excluding one or more candidates from the candidates for the peak indicating the pulse rate, the pulse rate is estimated. Part 1342 may determine one peak from the remaining candidates as a peak indicating the pulse rate. For example, the pulse rate estimation unit 1342 may determine the peak having the highest intensity among the remaining candidates as the peak indicating the pulse rate.

The pulse rate estimation unit 1342 determines the frequency of the peak determined as the peak indicating the pulse rate as the estimated pulse rate.

The pulse wave information derivation unit 134 sends information indicating the determined pulse rate to the output unit 119. The format of the information indicating the pulse rate may be the same as the example shown in the first embodiment.

[effect]
According to the pulse wave detection device 13 according to the third embodiment, as in the first embodiment, the decrease in the accuracy of the pulse wave information due to the movement of the face of the person to be measured is suppressed. The reason is that the pulse rate estimation unit 1342 of the pulse wave information derivation unit 114 uses the displacement data to set a peak having a frequency different from the frequency of the peak derived from the noise component caused by the movement of the face, and a peak indicating the pulse rate. Because it is selected as. That is, it is possible to prevent the peak derived from the noise component from being determined as the peak indicating the pulse rate. As a result, the accuracy of the output pulse wave information is improved.

The process by the pulse rate estimation unit 1342 is a process of excluding a component derived from noise from a plurality of components, in other words, a process of selecting or extracting a signal component derived from a pulse wave from a plurality of components. be.

The pulse wave detection device 13 according to the third embodiment also calculates the pulse rate for a plurality of target data ranges as in the modification 1-3 of the pulse wave detection device according to the first embodiment. You may. The pulse wave detection device 13 according to the third embodiment may also have an outlier removal unit 115 like the pulse wave detection device 11a.

<< Fourth Embodiment >>
FIG. 11 is a block diagram showing a configuration of a pulse wave detection device 10 according to an embodiment of the present invention. The pulse wave detection device 10 includes a region of interest setting unit 101, a luminance data generation unit 102, a displacement data generation unit 103, a pulse wave component extraction unit 104, and an output unit 105.

The area of interest setting unit 101 sets the area of interest of the face of the person to be measured with respect to the time-series image of the face of the person to be measured. The region of interest setting unit 111 in each of the above embodiments is an example of the region of interest setting unit 101.

The luminance data generation unit 102 generates luminance data which is time-series data of representative values of the luminance values of predetermined color components of the pixels in the region of interest. The luminance data generation unit 112 in each of the above embodiments is an example of the luminance data generation unit 102.

The displacement data generation unit 103 generates displacement data indicating the movement of the face of the person to be measured based on the time-series image. The displacement data generation unit 113 in each of the above embodiments is an example of the displacement data generation unit 103.

The pulse wave component extraction unit 104 extracts the signal component caused by the pulse wave in the luminance data by using the displacement data. Correcting the luminance data based on the displacement data corresponds to extracting the signal component caused by the pulse wave. Selecting an estimated pulse rate by referring to the displacement data also corresponds to extracting the signal component due to the pulse wave.

The pulse wave information derivation units 114, 124, and 134 in each of the above embodiments are examples of the pulse wave component extraction unit 104.

The output unit 105 outputs pulse wave information based on the extracted signal component. The pulse wave information may be the extracted signal component itself, or may be an estimated value of the pulse rate calculated based on the extracted signal component. The output unit 119 in each of the above embodiments is an example of the output unit 105.

FIG. 12 is a flowchart showing the flow of operation by the pulse wave detection device 10.

First, the region of interest setting unit 101 sets the region of interest of the face of the person to be measured with respect to the time-series image (step S101).

Next, the luminance data generation unit 102 generates the luminance data which is the time-series data of the representative values of the luminance values of the predetermined color components of the pixels in the region of interest (step S102).

Further, the displacement data generation unit 103 generates displacement data indicating the movement of the face of the person to be measured based on the time-series image (step S103).

Then, the pulse wave component extraction unit 104 extracts the signal component caused by the pulse wave in the luminance data by using the displacement data (step S104).

Then, the output unit 105 outputs the pulse wave information based on the extracted signal component (step S105).

The processes of step S102 and step S103 may be performed in the reverse order or may be performed in parallel.

According to the pulse wave detection device 10 of the present embodiment, the pulse wave information can be acquired more accurately from the captured image of the person to be measured. The reason is that the pulse wave component extraction unit 104 extracts information due to the pulse wave from the luminance data based on the data indicating the movement of the face.

<Hardware configuration that realizes each part of the embodiment>
In each embodiment of the present invention described above, each component of each device indicates a block of functional units.

The processing of each component may be realized, for example, by the computer system reading and executing a program stored in a computer-readable storage medium that causes the computer system to execute the processing. The "computer-readable storage medium" includes, for example, portable media such as optical disks, magnetic disks, magneto-optical disks, and non-volatile semiconductor memories, and ROMs (Read Only Memory) and hard disks built in computer systems. It is a storage device. A "computer-readable storage medium" is a device that dynamically holds a program for a short period of time, such as a communication line when a program is transmitted via a network such as the Internet or a communication line such as a telephone line. In that case, it also includes the one that temporarily holds the program, such as the volatile memory inside the computer system that corresponds to the server or client. Further, the above-mentioned program may be a program for realizing a part of the above-mentioned functions, and may be a program for realizing the above-mentioned functions in combination with a program already stored in the computer system.

The "computer system" is, for example, a system including a computer 900 as shown in FIG. The computer 900 includes the following configurations.
-One or more CPUs (Central Processing Units) 901
-ROM902
-RAM (Random Access Memory) 903
-Program 904A and storage information 904B loaded into RAM903
A storage device 905 that stores the program 904A and the storage information 904B.
Drive device 907 that reads and writes the storage medium 906.
-Communication interface 908 that connects to the communication network 909
-I / O interface 910 for inputting / outputting data
-Bus 911 connecting each component

For example, each component of each device in each embodiment is realized by the CPU 901 loading the program 904A that realizes the function of the component into the RAM 903 and executing the program 904A. The program 904A that realizes the functions of each component of each device is stored in, for example, in the storage device 905 or ROM 902 in advance. Then, the CPU 901 reads out the program 904A as needed. The storage device 905 is, for example, a hard disk. The program 904A may be supplied to the CPU 901 via the communication network 909, or may be stored in the storage medium 906 in advance, read by the drive device 907, and supplied to the CPU 901. The storage medium 906 is a portable medium such as an optical disk, a magnetic disk, a magneto-optical disk, and a non-volatile semiconductor memory.

There are various modifications in the method of realizing each device. For example, each device may be implemented by a possible combination of a computer 900 and a program, each of which is separate for each component. Further, a plurality of components included in each device may be realized by a possible combination of one computer 900 and a program.

In addition, some or all of the components of each device may be realized by other general-purpose or dedicated circuits, computers, or a combination thereof. These may be composed of a single chip or may be composed of a plurality of chips connected via a bus.

When a part or all of each component of each device is realized by a plurality of computers, circuits, etc., the plurality of computers, circuits, etc. may be centrally arranged or distributed. For example, a computer, a circuit, or the like may be realized as a form in which each is connected via a communication network, such as a client-and-server system or a cloud computing system.

Some or all of the above embodiments may also be described, but not limited to:

<< Additional notes >>
[Appendix 1]
With respect to the time-series image in which the face of the person to be measured is photographed, the area of interest setting means for setting the area of interest of the face of the person to be measured and the means for setting the area of interest.
Luminance data generation means for generating luminance data, which is time-series data of representative values of luminance values of predetermined color components of pixels in the region of interest.
Displacement data generation means for generating displacement data indicating the movement of the face of the person to be measured based on the time-series image, and
Using the displacement data, a pulse wave component extraction means for extracting a signal component caused by a pulse wave in the luminance data, and a pulse wave component extraction means.
An output means for outputting pulse wave information based on the extracted signal component, and
A pulse wave detector equipped with.
[Appendix 2]
The displacement data generating means generates the displacement data based on a change in one reference direction set with respect to the time-series image of the position of a feature point detected from the face, according to Appendix 1. Pulse wave detector.
[Appendix 3]
The pulse wave detecting device according to Appendix 2, wherein the displacement data generating means sets the reference direction based on the orientation of the face.
[Appendix 4]
The pulse wave detecting device according to Appendix 2 or 3, wherein the displacement data generating means sets a direction corresponding to the left-right direction for the person to be measured as the reference direction.
[Appendix 5]
The pulse wave detection device according to any one of Supplementary note 1 to 4, wherein the displacement data generating means generates the displacement data by using the position of a feature point on the nose of the person to be measured.
[Appendix 6]
The pulse wave component extracting means uses the displacement data and the brightness data to generate noise time-series data indicating a change in the brightness value due to the movement of the face of the person to be measured, and the noise time-series data. The brightness data is corrected by subtracting the data value of the above from the data value of the brightness data.
The pulse wave detection device according to any one of Appendix 1 to 5.
[Appendix 7]
The pulse wave component extracting means identifies the predetermined function in which the time series data generated by the processing is most similar to the brightness data among the plurality of predetermined functions for processing the data value of the displacement data. The time-series data generated by processing using the specified predetermined function is generated as the noise time-series data.
The pulse wave detection device according to Appendix 6.
[Appendix 8]
The predetermined function is a function of subtracting a reference value of displacement from the data value of the displacement data and then multiplying by a coefficient.
The pulse wave component extracting means has a mean square error between the time series data generated by processing using the predetermined function and the brightness difference data indicating the difference between the brightness data and the reference value of the brightness. By calculating the value of the coefficient so as to be the minimum, the predetermined function whose time series data generated by processing is most similar to the brightness data is specified.
The pulse wave detection device according to Appendix 6.
[Appendix 9]
The luminance data generating means generates the luminance data from each of the plurality of regions of interest.
The pulse wave detection device is
An association means for associating the reliability based on the magnitude of the dispersion of the data values with each of the plurality of luminance data corrected by the pulse wave component extraction means.
Each of the corrected luminance data is weighted based on the reliability, and the data values of the weighted luminance data are added for each time point to generate the synthesized luminance data. Synthetic means and
The pulse wave detection device according to any one of Supplementary note 6 to 8, further comprising.
[Appendix 10]
The luminance data generation means generates the luminance data for each of the plurality of color components.
The pulse wave component extracting means generates a plurality of time-series data indicating different signal components by applying an independent component analysis to the brightness data of a plurality of color components, and a plurality of time-series data from the plurality of time-series data. Is specified as a candidate for the pulse rate, the same frequency as the frequency of the peak existing in the frequency spectrum of the displacement data is excluded from the candidates for the pulse rate, and the pulse rate is estimated from the candidates not excluded.
The pulse wave detection device according to any one of Appendix 1 to 5.
[Appendix 11]
For the time-series image in which the face of the person to be measured is taken, the area of interest of the face of the person to be measured is set.
Luminance data, which is time-series data of representative values of the luminance values of predetermined color components of the pixels in the region of interest, is generated.
Based on the time-series image, displacement data showing the movement of the face of the person to be measured is generated.
Using the displacement data, the signal component caused by the pulse wave in the luminance data is extracted.
Outputs pulse wave information based on the extracted signal component.
Pulse wave detection method.
[Appendix 12]
The pulse wave detection method according to Appendix 11, wherein the displacement data is generated based on a change in one reference direction set for the time-series image of the position of a feature point detected from the face.
[Appendix 13]
The pulse wave detection method according to Appendix 12, wherein the reference direction is set based on the orientation of the face.
[Appendix 14]
The pulse wave detection method according to Appendix 12 or 13, wherein a direction corresponding to the left-right direction for the person to be measured is set as the reference direction.
[Appendix 15]
The pulse wave detection method according to any one of Appendix 11 to 14, wherein the displacement data is generated using the position of the feature point on the nose of the person to be measured.
[Appendix 16]
Using the displacement data and the brightness data, noise time-series data indicating a change in the brightness value due to the movement of the face of the person to be measured is generated, and the data value of the noise time-series data is used as the brightness data. The brightness data is corrected by subtracting from the data value of.
The pulse wave detection method according to any one of Appendix 11 to 15.
[Appendix 17]
Among a plurality of predetermined functions for processing the data value of the displacement data, the predetermined function whose time series data generated by the processing is most similar to the brightness data is specified, and the specified predetermined function is used. The time-series data generated by the processing is generated as the noise time-series data.
The pulse wave detection method according to Appendix 16.
[Appendix 18]
The predetermined function is a function of subtracting a reference value of displacement from the data value of the displacement data and then multiplying by a coefficient.
The coefficient such that the mean square error between the time series data generated by processing using the predetermined function and the brightness difference data indicating the difference between the brightness data and the reference value of the brightness is minimized. By calculating the value of, the predetermined function whose time series data generated by processing is most similar to the brightness data is specified.
The pulse wave detection method according to Appendix 17.
[Appendix 19]
The luminance data is generated from each of the plurality of regions of interest, and the luminance data is generated.
For each of the plurality of luminance data corrected using the displacement data and the luminance data, the reliability based on the magnitude of the dispersion of the data values is associated.
Each of the corrected luminance data is weighted based on the reliability, and the data values of the weighted luminance data are added for each time point to generate the synthesized luminance data. ,
The pulse wave detection method according to any one of Appendix 16 to 18.
[Appendix 20]
The luminance data is generated for each of the plurality of color components, and the luminance data is generated.
By applying the independent component analysis to the brightness data of a plurality of color components, a plurality of time series data indicating different signal components are generated, and a plurality of frequencies are used as candidates for pulse rate from the plurality of time series data. Identify, exclude the same frequency as the frequency of the peak existing in the frequency spectrum of the displacement data from the candidates for the pulse rate, and estimate the pulse rate from the candidates that are not excluded.
The pulse wave detection method according to any one of Appendix 11 to 15.
[Appendix 21]
On the computer
With respect to the time-series image in which the face of the person to be measured is photographed, the area of interest setting process for setting the area of interest of the face of the person to be measured and
Luminance data generation processing that generates luminance data that is time-series data of representative values of luminance values of predetermined color components of pixels in the region of interest.
Displacement data generation processing that generates displacement data indicating the movement of the face of the person to be measured based on the time-series image, and
Using the displacement data, a pulse wave component extraction process for extracting a signal component caused by a pulse wave in the luminance data, and
Output processing that outputs pulse wave information based on the extracted signal components, and
A computer-readable, non-temporary storage medium that stores a program that runs a program.
[Appendix 22]
The displacement data generation process is described in Appendix 21, which generates the displacement data based on a change in the position of a feature point detected from the face in one reference direction set with respect to the time-series image. Storage medium.
[Appendix 23]
The storage medium according to Appendix 22, wherein the displacement data generation process sets the reference direction based on the orientation of the face.
[Appendix 24]
The storage medium according to Appendix 22 or 23, wherein the displacement data generation process sets a direction corresponding to the left-right direction for the person to be measured as the reference direction.
[Appendix 25]
The storage medium according to any one of Supplementary note 21 to 24, wherein the displacement data generation process generates the displacement data using the positions of feature points on the nose of the person to be measured.
[Appendix 26]
The pulse wave component extraction process uses the displacement data and the brightness data to generate noise time-series data indicating a change in the brightness value due to the movement of the face of the person to be measured, and the noise time-series data. The brightness data is corrected by subtracting the data value of the above from the data value of the brightness data.
The storage medium according to any one of Appendix 21 to 25.
[Appendix 27]
The pulse wave component extraction process identifies the predetermined function in which the time series data generated by the processing is most similar to the brightness data among the plurality of predetermined functions for processing the data value of the displacement data. The time-series data generated by processing using the specified predetermined function is generated as the noise time-series data.
The storage medium according to Appendix 26.
[Appendix 28]
The predetermined function is a function of subtracting a reference value of displacement from the data value of the displacement data and then multiplying by a coefficient.
In the pulse wave component extraction process, the mean square error between the time series data generated by processing using the predetermined function and the brightness difference data indicating the difference between the brightness data and the reference value of the brightness is obtained. By calculating the value of the coefficient so as to be the minimum, the predetermined function whose time series data generated by processing is most similar to the brightness data is specified.
The storage medium according to Appendix 27.
[Appendix 29]
The luminance data generation process generates the luminance data from each of the plurality of regions of interest.
The program is applied to the computer.
An association process for associating the reliability based on the magnitude of the variance of the data values with each of the plurality of luminance data corrected by the pulse wave component extraction process.
Each of the corrected luminance data is weighted based on the reliability, and the data values of the weighted luminance data are added for each time point to generate the synthesized luminance data. Synthetic processing and
The storage medium according to any one of Supplementary Provisions 26 to 28.
[Appendix 30]
The luminance data generation process generates the luminance data for each of the plurality of color components.
In the pulse wave component extraction process, a plurality of time-series data indicating different signal components are generated by applying an independent component analysis to the brightness data of a plurality of color components, and a plurality of time-series data are generated from the plurality of time-series data. Is specified as a candidate for the pulse rate, the same frequency as the frequency of the peak existing in the frequency spectrum of the displacement data is excluded from the candidates for the pulse rate, and the pulse rate is estimated from the candidates not excluded.
The storage medium according to any one of Appendix 21 to 25.

The present invention is not limited to the embodiments described above. Various changes that can be understood by those skilled in the art can be made within the scope of the present invention in the configuration and details of the embodiments described above.

This application claims priority on the basis of the international application PCT / JP2017 / 041932 filed on 22 November 2017 and incorporates all of its disclosures herein.

10-13, 11a Pulse wave detection device 30 Imaging device 101 Interest area setting unit 102 Brightness data generation unit 103 Displacement data generation unit 104 Pulse wave component extraction unit 105 Output unit 110 Storage unit 111 Interest area setting unit 112 Brightness data generation unit 113 Displacement data generation part 114 Pulse wave information derivation part 1141 Correction part 1142 Pulse number estimation part 115 Outlier removal part 119 Output part 124 Pulse wave information derivation part 1241 Correction part 1242 Association part 1243 Synthesis part 1244 Pulse number estimation part 134 Pulse wave information Derivation unit 1341 Data analysis unit 1342 Pulse rate estimation unit 900 Computer 901 CPU
902 ROM
903 RAM
904A Program 904B Storage information 905 Storage device 906 Storage medium 907 Drive device 908 Communication interface 909 Communication network 910 Input / output interface 911 Bus

Claims (10)

With respect to the time-series image in which the face of the person to be measured is photographed, the area of interest setting means for setting the area of interest of the face of the person to be measured and the means for setting the area of interest.
Luminance data generation means for generating luminance data, which is time-series data of representative values of luminance values of predetermined color components of pixels in the region of interest.
Displacement data generation that generates displacement data indicating the movement of the face of the person to be measured based on a change in one reference direction set for the time-series image of the position of a feature point detected from the face. Means and
Using the displacement data, a pulse wave component extraction means for extracting a signal component caused by a pulse wave in the luminance data, and a pulse wave component extraction means.
An output means for outputting pulse wave information based on the extracted signal component, and
The displacement data generation means is a pulse wave detection device that sets a direction corresponding to the left-right direction for the person to be measured as the reference direction. The pulse wave detecting device according to claim 1 , wherein the displacement data generating means sets the reference direction based on the orientation of the face. The pulse wave detection device according to claim 1 or 2 , wherein the displacement data generating means generates the displacement data by using the position of a feature point on the nose of the person to be measured. The pulse wave component extracting means uses the displacement data and the brightness data to generate noise time-series data indicating a change in the brightness value due to the movement of the face of the person to be measured, and the noise time-series data. The brightness data is corrected by subtracting the data value of the above from the data value of the brightness data.
The pulse wave detection device according to any one of claims 1 to 3. With respect to the time-series image in which the face of the person to be measured is photographed, the area of interest setting means for setting the area of interest of the face of the person to be measured and the means for setting the area of interest.
Luminance data generation means for generating luminance data, which is time-series data of representative values of luminance values of predetermined color components of pixels in the region of interest.
Displacement data generation means for generating displacement data indicating the movement of the face of the person to be measured based on the time-series image, and
Using the displacement data and the brightness data, noise time-series data indicating a change in the brightness value due to the movement of the face of the person to be measured is generated, and the data value of the noise time-series data is used as the brightness data. By correcting the brightness data by subtracting from the data value of, the pulse wave component extraction means for extracting the signal component caused by the pulse wave in the brightness data, and the pulse wave component extraction means.
An output means for outputting pulse wave information based on the extracted signal component, and
A pulse wave detector equipped with. The pulse wave component extracting means identifies and specifies the predetermined function in which the time series data generated by the processing is most similar to the brightness data among the plurality of predetermined functions for processing the data value of the displacement data. The time-series data generated by the processing using the predetermined function is generated as the noise time-series data.
The pulse wave detection device according to claim 4 or 5. The predetermined function is a function of subtracting a reference value of displacement from the data value of the displacement data and then multiplying by a coefficient.
The pulse wave component extracting means has a mean square error between the time series data generated by processing using the predetermined function and the brightness difference data indicating the difference between the brightness data and the reference value of the brightness. By calculating the value of the coefficient so as to be the minimum, the predetermined function whose time series data generated by processing is most similar to the brightness data is specified.
The pulse wave detection device according to claim 4 or 5. The luminance data generating means generates the luminance data from each of the plurality of regions of interest.
The pulse wave detection device is
An association means for associating the reliability based on the magnitude of the dispersion of the data values with each of the plurality of luminance data corrected by the pulse wave component extraction means.
Each of the corrected luminance data is weighted based on the reliability, and the data values of the weighted luminance data are added for each time point to generate the synthesized luminance data. Synthetic means and
The pulse wave detection device according to any one of claims 4 to 7. For the time-series image in which the face of the person to be measured is taken, the area of interest of the face of the person to be measured is set.
Luminance data, which is time-series data of representative values of the luminance values of predetermined color components of the pixels in the region of interest, is generated.
Based on the change in one reference direction set for the time-series image of the position of the feature point detected from the face, displacement data indicating the movement of the face of the person to be measured is generated.
Using the displacement data, the signal component caused by the pulse wave in the luminance data is extracted.
The pulse wave information based on the extracted signal component is output, and the information is output.
The displacement data is generated by setting a direction corresponding to the left-right direction for the person to be measured as the reference direction.
Pulse wave detection method. On the computer
With respect to the time-series image in which the face of the person to be measured is photographed, the area of interest setting process for setting the area of interest of the face of the person to be measured and
Luminance data generation processing that generates luminance data that is time-series data of representative values of luminance values of predetermined color components of pixels in the region of interest.
Displacement data generation that generates displacement data indicating the movement of the face of the person to be measured based on a change in one reference direction set for the time-series image of the position of a feature point detected from the face. Processing and
Using the displacement data, a pulse wave component extraction process for extracting a signal component caused by a pulse wave in the luminance data, and
Output processing that outputs pulse wave information based on the extracted signal components, and
To run ,
The displacement data generation process, the direction corresponding to the lateral direction for the subject, that generates said displacement data is set as the reference direction program.

Images