JP7802335B2 - Pulse wave detection system, pulse wave detection method and program - Google Patents

Description

The present invention relates to a pulse wave detection system, pulse wave detection method, and program that can detect a subject's pulse wave in a non-contact manner and determine the subject's condition.

In recent years, with various changes in the environment, such as the COVID-19 pandemic, it has become increasingly important to assess the stress of employees while they are at work, in order to understand their condition. There are various methods for stress assessment, but one method is to detect pulse waves linked to the heartbeat.

Pulse waves can be measured, for example, using a sensor in a smartwatch worn on the subject's wrist. A method for evaluating stress using the measured pulse waves is to determine whether the subject is in a stressed or relaxed state from the power spectral density of the pulse wave peak interval, for example.
Another method for assessing stress using pulse waves is to use the amplitude of the pulse wave. When the amplitude of the pulse wave increases, it can be evaluated as a state of stress, and when the amplitude of the pulse wave decreases, it can be evaluated as a state of relaxation.

Patent Document 1 describes a technology that captures an image of a person being measured, acquires biological information such as pulse rate from the captured image, and determines their condition, such as their stress level.

Japanese Patent Application Laid-Open No. 2021-132726

As described in Patent Document 1, it has been proposed to perform non-contact stress assessment from pulse waves, without the need to wear a device such as a smartwatch. However, pulse waves detected from images are not highly accurate, which has led to the problem of stress assessments not always being accurate. Furthermore, while various research efforts have been conducted to improve the accuracy of pulse wave detection, detecting pulse waves with high accuracy requires specialized, expensive detection equipment with a high sampling frequency. Therefore, there has been a demand for highly accurate pulse wave detection using general-purpose equipment with a lower sampling frequency.

The object of the present invention is to provide a pulse wave detection system, pulse wave detection method, and program that are simple and inexpensive, and capable of non-contact, highly accurate pulse wave detection.

The pulse wave detection system of the present invention comprises an image input unit that acquires image data of an image of a subject's face; a first color space conversion unit that converts the image data acquired by the image input unit into image data in an HSV color space consisting of components of hue, saturation, and lightness, and that limits the hue during conversion to a hue range corresponding to skin color; a second color space conversion unit that converts the image data in the HSV color space converted by the first color space conversion unit into a primary color color space; an independent component analysis unit that independently analyzes each primary color component from the image data in the primary color space converted by the second color space conversion unit; a Fourier transform unit that performs frequency analysis on the results analyzed by the independent component analysis unit; and a signal selection and averaging unit that selects, from the results of Fourier transform performed by the Fourier transform unit for each of a plurality of sections included in a certain time period, amplitude values that result in a first peak in a frequency band that includes a component corresponding to a pulse wave, and averages the selected amplitude values to obtain a detection result of the amplitude value of the subject's pulse wave.

Furthermore, the pulse wave detection method of the present invention is a pulse wave detection method in which an information processing device performs arithmetic processing based on image data obtained by photographing the face of the subject, thereby detecting the pulse wave of the subject. The arithmetic processing performed by the information processing device includes: an image acquisition process for acquiring image data obtained by photographing the face of the subject; a first color space conversion process for converting the image data obtained by the image acquisition process into image data in an HSV color space consisting of components of hue, saturation, and luminance, and limiting the hue during conversion to a hue band corresponding to skin color; a second color space conversion process for converting the image data in the HSV color space converted by the first color space conversion process into a color space of primary colors; an independent component analysis process for independently analyzing each primary color component from the image data in the color space of primary colors converted by the second color space conversion process; a Fourier transform process for frequency-analyzing the results analyzed by the independent component analysis process; and a signal selection and averaging process for selecting, from the results of Fourier transforming a plurality of sections included in a certain time period by the Fourier transform process, amplitude values of results that include a first peak in a frequency band that includes a component corresponding to the pulse wave, and averaging the selected amplitude values, thereby obtaining a detection result of the amplitude value of the pulse wave of the subject.
The program of the present invention causes a computer to execute the steps of the pulse wave detection method described above.

According to the present invention, pulse waves can be detected with high accuracy from images of a face. This makes it possible to perform highly accurate stress assessments, for example, based on the detected pulse waves. In particular, pulse waves can be detected with high accuracy from images taken using an inexpensive camera, such as a general-purpose web camera, to acquire image data, enabling highly accurate stress assessments with an inexpensive configuration.

1 is a configuration diagram showing an example of a pulse wave detection system according to an embodiment of the present invention; 10A and 10B are diagrams illustrating an example of a detection state according to an embodiment of the present invention. 1 is a block diagram showing an example of a hardware configuration when a pulse wave detection device according to an embodiment of the present invention is configured using a computer. 4 is a flowchart showing the flow of pulse wave detection processing according to an embodiment of the present invention. 10A and 10B are diagrams illustrating how the peaks of a waveform for detecting a pulse wave are crushed depending on the sampling rate. FIG. 10 is a diagram showing an example of information contained in each hue band. 10A and 10B are diagrams showing examples of pulse wave detection processing waveforms according to an embodiment of the present invention. 10A and 10B are diagrams showing pulse wave detection processing results for each hue detection range according to an embodiment of the present invention.

One embodiment of the present invention (hereinafter referred to as "this embodiment") will be described below with reference to the accompanying drawings.

[System configuration]
FIG. 1 is a functional block diagram showing the configuration of the processing performed by a pulse wave detection system 100 of this embodiment.
The pulse wave detection system 100 of this example captures image data captured by a camera 1. The image data is a photograph of the face of a subject whose pulse wave is to be detected. The image data captured by the camera 1 is video data at a constant frame rate, with each frame of image data being composed of pixel data of the primary colors red (R), green (G), and blue (B). The frame rate of the image data is, for example, 30 frames per second, and the pixels in each frame represent the brightness values of red (R), green (G), or blue (B) in a predetermined gradation.

Image data captured by pulse wave detection system 100 is supplied to first color space conversion unit 101. First color space conversion unit 101 converts the primary color image data into image data in the HSV color space, which is made up of three components: hue, saturation (chroma), and value (brightness). Value in the HSV color space is also called luminance.
Here, the first color space conversion unit 101 performs processing to extract a specific hue band by applying restrictions to the hue (Hue). A specific example of processing to extract a specific hue band will be described later.

The image data in the HSV color space converted by the first color space conversion unit 101 is supplied to the second color space conversion unit 102. The second color space conversion unit 102 reconverts the image data in the HSV color space into image data in the primary colors of red (R), green (G), and blue (B).
The image data converted by the second color space conversion unit 102 is supplied to the independent component analysis unit 103 for each component. The independent component analysis unit 103 performs an analysis process to obtain the amplitude of the change in each color component for each frame. The independent component analysis unit 103 performs a process to obtain the amplitude value of the minute vibration state corresponding to the heartbeat for the skin part included in the facial image.

That is, the independent component analysis unit 103 performs an analysis process to extract amplitude components containing pulse waves from the changes in each color component for each frame.
The amplitude data obtained by analyzing each color component by the independent component analysis unit 103 is supplied to a Fourier transform unit 104 .
The Fourier transform unit 104 performs frequency analysis on the amplitude data. The result of the frequency analysis by the Fourier transform unit 104 is supplied to the signal selection and averaging unit 105.

The signal selection and averaging unit 105 selects signals for detecting pulse waves based on the results of the frequency analysis and averages the selected signals. The signals selected and averaged by the signal selection and averaging unit 105 are output from the output unit 106 as pulse wave detection values.

[Example of pulse wave detection state]
FIG. 2 shows an example of a state in which the pulse wave detection system 100 of this embodiment detects the pulse wave of a subject.
As shown in Fig. 2, a camera 1 attached to a pulse wave detection system 100 configured, for example, as a personal computer, captures an image of the face of a subject operating the personal computer, and detects pulse waves. The results of the detected pulse waves can be used, for example, for evaluating the stress of the subject. Note that the personal computer serving as pulse wave detection system 100 can be configured to include a program for pulse wave detection installed on the computer on which the subject works, allowing the subject to evaluate their own stress while at work.

[Example of hardware configuration of pulse wave detection system]
FIG. 3 shows an example of the hardware configuration of pulse wave detection system 100 when configured using a computer.
The computer (pulse wave detection system 100) shown in FIG. 3 includes a CPU (Central Processing Unit) 100a, a main memory unit 100b, a non-volatile storage 100c, a network interface 100d, an image input unit 100e, an output unit 100f, and an operation unit 100g, which are all connected to a bus.

The CPU 100a is an arithmetic processing unit that reads out and executes program code of software that realizes the functions performed by the pulse wave detection system 100 from the main storage unit 100b or the non-volatile storage unit 100c.
The CPU 100a reads program code from the main memory unit 100b or the non-volatile storage 100c and executes arithmetic processing in the work area of the main memory unit 100b, thereby configuring various processing function units in the main memory unit 100b. For example, the main memory unit 100b is configured with a first color space conversion unit 101, a second color space conversion unit 102, an independent component analysis unit 103, a Fourier transform unit 104, and a signal selection and averaging unit 105 shown in FIG. 1 .

Non-volatile storage 100c may be a large-capacity information storage medium such as a hard disk drive (HDD), solid state drive (SSD), or memory card. Non-volatile storage 100c stores software that realizes the functions of pulse wave detection system 100, as well as data obtained by executing that program.

The network interface 100d is, for example, a network interface card (NIC), and transmits and receives data to and from other devices.
Image data from the camera 1 is input to the image input unit 100e, and image acquisition processing is performed.
The output unit 100f is connected to a display 2, and data such as images showing the pulse wave detection results and evaluation results such as stress evaluation based on the pulse wave detection are output from the output unit 100f, and these detection results and evaluation results are displayed on the display 2. The detection results and evaluation results may be transmitted to an external device via the network interface 100d.
The operation unit 100g accepts operations on devices such as a keyboard and a mouse operated by the operator of this computer.

[Pulse wave detection process flow]
FIG. 4 is a flowchart showing the flow of pulse wave detection processing performed by pulse wave detection system 100 of this embodiment with the configuration shown in FIG.
First, the image input unit 100e (FIG. 3) of the pulse wave detection system 100 captures the facial image data of the subject photographed by the camera 1 (step S11).
Then, the first color space conversion unit 101 performs a first color space conversion process to convert the primary color image data into image data in the HSV color space consisting of three components: hue, saturation (chroma), and value (brightness) (step S12). When converting into image data in the HSV color space, a process is performed in which hue is restricted and a specific hue band is extracted.

Then, the image data in the HSV color space converted by the first color space conversion unit 101 is reconverted by the second color space conversion unit 102 into image data in the primary colors of red (R), green (G), and blue (B) (second color space conversion process) (step S13).
Furthermore, the image data converted by the second color space conversion unit 102 is subjected to a process (independent component analysis process) in which each component is analyzed individually for each color component by the independent component analysis unit 103, and amplitude data is obtained from the changes for each pixel for each frame (step S14).

The amplitude data obtained in the independence component analysis process of step S14 is then subjected to frequency analysis using a Fourier transform by the Fourier transform unit 104 (step S15). From the results of the frequency analysis by the Fourier transform unit 104, signal selection and averaging is performed by the signal selection and averaging unit 105, a pulse wave is detected from the averaged results, and the detected pulse wave value is output (step S16). In this step S16, a stress assessment of the subject is also performed based on the state of the detected pulse wave, and the determined stress assessment result is output.

[Description of each process for detecting pulse waves]
FIG. 5 compares (a) an example of an original waveform simulating a pulse wave, which is a heartbeat component; (b) a waveform obtained by sampling the original waveform using a photoplethysmograph, a conventionally known pulse wave detection device; and (c) a waveform obtained by sampling the original waveform at the sampling rate of video data.
In the case of the photoplethysmograph shown in FIG. 5(b), the sampling frequency is, for example, 1000 Hz, and the peak shape of the original waveform shown in FIG. 5(a) is directly captured as data.

On the other hand, when the movement of a subject is detected from the video data shown in Fig. 5(c), the sampling frequency is 30 Hz, which corresponds to the frame frequency, and the peaks of the original waveform shown in Fig. 5(a) are flattened. That is, the video data handled by pulse wave detection system 100 of this example is video data acquired by a camera such as a web camera normally used in personal computer devices, and is image data with a relatively low frequency such as 30 Hz, and the sampling frequency is correspondingly low.
The movement components shown in Figure 5(c) correspond to the pulse wave components captured by pulse wave detection system 100 of this example, and it is difficult to correctly detect the pulse wave, which is a heartbeat component, without any modification. For this reason, pulse wave detection system 100 of this example is able to properly detect pulse waves by using the configuration shown in Figure 1 and the processing flow shown in Figure 4. Each process in pulse wave detection system 100 of this example will be described below in order.

The upper part of Fig. 6 shows the hue in the HSV color space converted by the first color space conversion unit 101, in the range of 0° to 360°. Note that the graph in the upper part of Fig. 6 is actually a graph showing the change in color from 0° to 360°, but since the drawing is expressed in black and white, the color is not shown. Therefore, in Fig. 6, the positions of the hues corresponding to the three primary colors R, G, and B are indicated by arrows in the drawing.
It is known that human skin color falls within the range of 0 to 15 degrees within the hues from 0 to 360 degrees. However, the skin color here is assumed to be, for example, Caucasian or Mongoloid, and the range of skin color may differ slightly depending on the race.

Therefore, the first color space conversion unit 101 in this example performs processing to extract human skin color components from the image data converted to the HSV color space as the hue, such as from 0° to 15°. Note that the band for extracting skin color components from the hue is merely an example, and is not limited to this band.
In this way, first color space conversion unit 101 converts image data from the camera into the HSV color space, extracts components corresponding to human skin color, and supplies the extracted signals to second color space conversion unit 102. Second color space conversion unit 102 reconverts the input HSV color space signal into the RGB color space, thereby obtaining reconverted RGB color space image data that makes it easier to capture changes in blood flow.

As shown in the lower part of Figure 6, skin information is included in the hue in the HSV color space. On the other hand, AC light source components are included in the luminance, which can adversely affect pulse wave detection. Therefore, the band corresponding to the AC light source components included in the luminance may be removed.
As shown in the lower part of FIG. 6, the red (R) component in the RGB color space contains arterial information, and the green (G) component contains capillary information.

The left side of Figure 7 shows the change over time in the waveform obtained by the independent component analysis unit 103. The vertical axis of the graph shown on the left side of Figure 7 is amplitude, and the horizontal axis is time. The waveform shown on the left side of Figure 7 represents the minute movements in the image obtained by the independent component analysis unit 103 in amplitude. As already explained, the image data handled by the independent component analysis unit 103 here is the component that corresponds to the color of human skin extracted, and corresponds to the minute movements of the skin part of the face shown in amplitude.

The amplitude waveform shown on the left side of Fig. 7 is Fourier transformed by the Fourier transform unit 104, and frequency analysis is performed. In a normal Fourier transform, the data is divided into sections d1, d2, d3, ... at regular time intervals as data sections for performing Fourier transform processing (so-called FFT processing), as shown on the left side of Fig. 7, and data for performing FFT processing is obtained in order.
In contrast, in the Fourier transform unit 104 of this example, FFT processing is performed by setting partially overlapping intervals D11, D12, D13, ... as shown on the left side of Fig. 7, and frequency analysis result data for each interval is obtained in sequence. Here, the FFT processing intervals D11, D12, D13 are set so that they overlap by 50%, but 50% is just an example.

Then, the signal selection and averaging unit 105 selects signals from which the heartbeat component has been properly acquired from the results of frequency analysis for each section D11, D12, D13, ..., and performs processing to average the selected frequency analysis for each fixed section.
When the signal selection and averaging unit 105 selects a signal from which the heartbeat component has been properly acquired, the frequency analysis result in which the first peak has been detected in the frequency band from 1 Hz to 2 Hz, which is the frequency band that contains the heartbeat component, is selected, and the amplitude value of the first peak of the selected frequency analysis result is obtained.

That is, as shown in the upper right-hand panel of Figure 7, the signal selection and averaging unit 105 selects, as the frequency analysis result for a certain section, a signal in which the first peak P1 is located within the band Da from 1 Hz to 2 Hz, and obtains the amplitude value (vertical axis) of this first peak P1. In contrast, as shown in the lower right-hand panel of Figure 7, the signal selection and averaging unit 105 does not select, as the frequency analysis result for a certain section, a signal in which the first peak P2 is located at a frequency position outside the band Db from 1 Hz to 2 Hz. Note that in the pulse wave detection process of this example, the amplitude value of the first peak P1 is used, and information on the frequency position of the first peak P2 is not used in pulse wave detection.

The signal selection and averaging unit 105 then collects the amplitude values of the frequency analysis results selected in this manner and performs averaging. For example, as shown on the left side of Fig. 7, the signal selection and averaging unit 105 collects the frequency analysis results selected every certain time period A1 and performs averaging. The certain time period A1 is set to be sufficiently longer than the duration of each of the sections D11, D12, D13, ...
In this way, the average of the amplitude values obtained by signal selection and averaging unit 105 is output from output unit 106 and displayed on display 2 or the like connected to pulse wave detection system 100. The average of the displayed amplitude values is a value that correlates with the pulse wave, and the detected pulse wave is displayed on display 2.

[Example of correlation between hue restriction band and pulse wave (heart rate)]
As already explained, the image data in the HSV color space converted by the first color space conversion unit 101 is subjected to processing to extract the color components of human skin as the hue.
In the above explanation, the process for extracting the color components of human skin was performed by extracting the hue from 0° to 15°. However, Figure 8 shows an example in which the bandwidth is narrower. The images in the upper rows of Figures 8(a), (b), and (c) show the area where the skin color components were extracted from the face, and the table in the lower row shows the similarity between the detected pulse wave and the actual heart rate. In the images in the upper rows of Figures 8(a), (b), and (c), the relevant areas to be extracted are those where the face is visible, i.e., the "non-black areas" in the image; black areas are outside the bandwidth and are not extracted. A similarity value close to 1 shown in the table in the lower row indicates a high similarity to the heart rate.

Figure 8(a) shows an example of detecting heartbeats, which correspond to pulse waves, by extracting the 4° to 14° hue band. In this example, as can be seen from the top image in Figure 8(a), a large area of the face is used for detection. When the 4° to 14° hue band is used, the similarity between all six subjects is high, and it can be seen that pulse waves can be detected well.

Figure 8(b) shows an example in which the 5° to 9° hue band was extracted to detect the heartbeat, which corresponds to the pulse wave. In this example, as can be seen by comparing the top image in Figure 8(a) with the top image in Figure 8(b), the area used is slightly narrower, with the focus being on extracting the earlobe area. When the 5° to 9° hue band was used, the similarity was slightly reduced for only one subject (subject 4), but the similarity was high for the other five subjects, indicating that the pulse wave can be detected even in this band.

Figure 8(c) shows an example in which the heart rate corresponding to the pulse wave was detected by extracting the 2° to 5° hue band. In this example, as can be seen from the upper image in Figure 8(c), detection is performed using a relatively narrow region centered on the cheek of the face. When the 5° to 9° hue band was used, it was found that the similarity was high for all six subjects, and pulse waves could be detected satisfactorily.
For reference, when a pulse wave is detected from an image using a conventional method, the similarity indicating the correlation between the detected pulse wave and the actually measured heart rate is about 0.70. The average similarity value of 0.87 to 0.88 shown in FIG. 8 indicates a much higher correlation than conventional methods.

In this way, pulse wave detection system 100 of this example can acquire and process a facial image of the subject, thereby enabling highly accurate pulse wave detection. In particular, a web camera with a general frame rate that can be connected to a personal computer or the like can be used as the camera for capturing the facial image of the subject, enabling highly accurate pulse wave detection with a simple and inexpensive configuration.
Then, it becomes possible to appropriately evaluate the stress of the subject based on the change in the detected pulse wave. For example, when the pulse wave remains high, it can be evaluated as a state of stress. Alternatively, when a state of the pulse wave decreasing is detected, it can be evaluated as a state of less stress. As a specific example of evaluating stress based on the detected pulse wave, various known methods can be applied.

Note that while the pulse wave detection system 100 shown in Figure 1 is an example configured as a system for detecting pulse waves, the pulse wave detection system 100 of this example can also be configured with a computer as shown in Figure 3, for example. Therefore, by incorporating a program for operating the pulse wave detection system 100 of this example into an information processing device such as a computer or smartphone, pulse wave detection processing can be performed in parallel with various processes being executed by the computer or smartphone. As a result, it becomes possible to detect the subject's pulse waves and perform stress assessments at any time by evaluating the pulse wave detection results. The program for operating the pulse wave detection system 100 of this example can be created by executing the processes described in the flowchart of Figure 4 as procedures. This program can also be stored on various types of memory, IC cards, SD cards, optical discs, and other recording media.

1...Camera, 2...Display, 4...Subject, 100...Pulse wave detection system, 100a...CPU, 100b...Main memory unit, 100c...Non-volatile storage, 100d...Network interface, 100e...Image input unit, 100f...Output unit, 100g...Operation unit, 101...First color space conversion unit, 102...Second color space conversion unit, 103...Independent component analysis unit, 104...Fourier transform unit, 105...Averaging unit, 106...Output unit

Claims (5)

a first color space conversion unit that acquires image data of a face of a subject, converts the acquired image data into image data in an HSV color space consisting of components of hue, saturation, and brightness, and limits the hue during conversion to a hue range corresponding to skin color;
a second color space conversion unit that converts the image data in the HSV color space converted by the first color space conversion unit into a primary color space;
an independent component analysis unit that analyzes components of each primary color independently from the image data in the color space of the primary colors converted by the second color space conversion unit;
a Fourier transform unit that performs frequency analysis on the results of the analysis by the independent component analysis unit;
a signal selection and averaging unit that selects, from the results of Fourier transform performed by the Fourier transform unit for each of a plurality of sections included in a certain time period , amplitude values that result in a first peak being included in a frequency band that includes a component corresponding to a pulse wave, and averages the selected amplitude values to obtain a detection result of the amplitude value of the pulse wave of the subject. The pulse wave detection system according to claim 1 , wherein the signal selection and averaging unit detects an amplitude value of the arterial pulse wave of the subject by selecting red color analyzed by the independent component analysis unit. The pulse wave detection system according to claim 1 or 2, wherein the unit section subjected to frequency analysis by the Fourier transform section is a section that partially overlaps with preceding and succeeding sections. A pulse wave detection method in which an information processing device performs arithmetic processing based on image data of a face of a subject, and detects an amplitude value of a pulse wave of the subject,
The arithmetic processing performed by the information processing device includes:
a first color space conversion process for converting image data of the subject's face into image data in an HSV color space consisting of components of hue, saturation, and luminance, and limiting the hue during conversion to a hue range corresponding to skin color;
a second color space conversion process for converting the image data in the HSV color space converted by the first color space conversion process into a primary color space;
an independent component analysis process for independently analyzing components of each primary color from the image data in the color space of the primary colors converted by the second color space conversion process;
a Fourier transform process for frequency-analyzing the results of the analysis performed by the independent component analysis process;
and a signal selection and averaging process for selecting, from the results of Fourier transform performed by the Fourier transform process for each of a plurality of sections included in a certain time period , amplitude values resulting in a first peak being included in a frequency band that includes a component corresponding to a pulse wave, and averaging the selected amplitude values to obtain a detection result of the amplitude value of the pulse wave of the subject. A program for detecting a pulse wave of a subject based on image data of the subject's face,
a first color space conversion step of converting image data of the subject's face into image data in an HSV color space consisting of components of hue, saturation, and luminance, and limiting the hue during conversion to a hue range corresponding to skin color;
a second color space conversion step of converting the image data in the HSV color space converted by the first color space conversion step into a primary color space;
an independent component analysis step of independently analyzing components of each primary color from the image data in the primary color color space converted by the second color space conversion step;
a Fourier transform step for frequency-analyzing the results of the analysis performed in the independent component analysis step;
a signal selection and averaging step of selecting, from the results of performing Fourier transform on each of a plurality of sections included in a certain time period using the Fourier transform step, amplitude values resulting in a first peak being included in a frequency band including a component corresponding to a pulse wave, and averaging the selected amplitude values to obtain a detection result of the amplitude value of the pulse wave of the subject;
A program that is implemented and executed on a computer.