JP7564632B2 - Information processing device and pulse wave calculation method - Google Patents

Description

This disclosure relates to an information processing device and a pulse wave calculation method.

For example, Patent Document 1 discloses a technique for detecting a subject's pulse wave using luminance signals from multiple pixels in an image of the subject.

JP 2014-073159 A

In the technology disclosed in Patent Document 1, when pulse waves of different blood vessels are detected in multiple pixels, the shapes of the pulse waves corresponding to each pixel are different, and the pulse wave cannot be calculated accurately. One aspect of the present disclosure aims to provide an information processing device and a pulse wave calculation method that contribute to accurate calculation of vascular pulse waves from an image of the body surface.

An information processing device according to one aspect of the present disclosure includes a pulse wave calculation unit that calculates multiple pulse waves corresponding to different positions on a body surface from a moving image of the body surface, a similarity calculation unit that calculates the similarity between at least two of the multiple pulse waves, and a pulse wave synthesis unit that synthesizes the pulse waves based on the similarity.

A pulse wave calculation method according to one aspect of the present disclosure includes the steps of calculating multiple pulse waves corresponding to different positions on a body surface from a moving image of the body surface, calculating a similarity between at least two of the multiple pulse waves, and synthesizing the pulse waves based on the similarity.

1 is a diagram showing an example of a relationship between an information processing device 100 and a living body A. 2 is a block diagram showing an example of an electrical configuration of the information processing device; 2 is a block diagram showing an example of a functional configuration of a control unit according to the first embodiment; FIG. FIG. 2 is a diagram showing an example of a region of interest in a frame image. FIG. 11 is a diagram showing an example of pixels corresponding to a synthetic candidate pulse wave. 5 is a flowchart illustrating an example of processing executed by the information processing apparatus of the first embodiment. FIG. 13 is a diagram showing an example of a relationship between blood vessels and pixels corresponding to a pulse wave of interest. FIG. 2 is a diagram showing an example of a pulse wave. FIG. 2 is a diagram showing an example of a pulse wave. FIG. 11 is a block diagram showing an example of a functional configuration of a control unit according to a second embodiment. FIG. 13 is a block diagram showing an example of a functional configuration of a control unit according to a third embodiment. FIG. 2 is a diagram illustrating an example of a region of interest. FIG. 13 is a block diagram showing an example of a functional configuration of a control unit according to a fourth embodiment. FIG. 11 is a diagram showing an example of a pixel corresponding to a pulse wave and surrounding pixels. FIG. 13 is a block diagram showing an example of a functional configuration of a control unit in a fifth embodiment. FIG. 4 is a diagram illustrating an example of a pulse wave feature amount. 13 is a flowchart illustrating an example of a process flow executed by an information processing apparatus according to a fifth embodiment. FIG. 13 is a block diagram showing an example of a functional configuration of a control unit in a sixth embodiment. 13 is a flowchart illustrating an example of a process flow executed by an information processing apparatus according to a sixth embodiment. FIG. 11 is a diagram showing an example of a relationship between blood vessels and pixels corresponding to a first pulse wave and a second pulse wave.

First Embodiment
A first embodiment will be described. In the drawings, the same or equivalent elements are denoted by the same reference numerals, and duplicated explanations will be omitted.

FIG. 1 is a diagram showing an example of the relationship between an information processing device 100 and a living body A. The information processing device 100 captures an image of the body surface of the living body A using an imaging unit 101, and calculates the pulse wave of the living body A. The pulse wave is a signal indicating a change in the volume of blood vessels. Specifically, the color of the captured body surface changes as the volume of blood vessels changes with the beating of the heart. Therefore, the information processing device 100 calculates the pulse wave based on the change in color of the body surface captured by the imaging unit 101. For example, the information processing device 100 is a PC (Personal Computer), a smartphone, a tablet, a notebook computer, or a dedicated terminal for measuring pulse waves.

The imaging unit 101 captures the body surface of the living body A and acquires a moving image of the body surface. Specifically, the imaging unit 101 captures the body surface of the living body A at a predetermined frame rate (for example, 10 to 300 fps (frames per second)) for a predetermined time (for example, 10 to 60 seconds). The imaging unit 101 stores the acquired moving image of the body surface in the storage unit 201 (see FIG. 2). In the following description, for convenience of explanation, the moving image of the body surface is simply referred to as a "moving image". The body surface includes the face of the living body A. The imaging unit 101 is only required to be able to acquire a moving image of the body surface, and the position and type of the imaging unit 101 are not limited. For example, the imaging unit 101 may be an RGB camera or an infrared camera. Note that in the following description, for convenience of explanation, a human is used as an example of the living body A, but the living body A may be a living body other than a human.

Figure 2 is a block diagram showing an example of the electrical configuration of the information processing device 100. The information processing device 100 includes an imaging unit 101, a storage unit 201, a control unit 202, an output unit 203, etc.

The storage unit 201 is a storage device capable of storing any information, and is configured, for example, with a hard disk, a solid state drive (SSD), a semiconductor memory, etc. The storage unit 201 stores moving images of the body surface acquired by the imaging unit 101. The storage unit 201 also stores, for example, a program that causes the control unit 202 to function, data required for the control unit 202 to function, etc.

The control unit 202 executes various processes according to the programs and data stored in the memory unit 201. The control unit 202 is realized by a processor such as a CPU (Central Processing Unit), for example.

The output unit 203 outputs the pulse wave of the living body A. For example, the output unit 203 is a display device.

FIG. 3 is a block diagram showing an example of the functional configuration of the control unit 202. The control unit 202 includes an image acquisition unit 301, a region of interest extraction unit 302, a pulse wave calculation unit 303, a similarity calculation unit 304, a pulse wave synthesis unit 305, an output control unit 306, etc.

The image acquisition unit 301 acquires moving images from the storage unit 201. The moving images stored in the storage unit 201 include multiple frame images. A frame image is an image of each frame of the moving image.

The region of interest extraction unit 302 extracts one or more regions of interest from frame images included in the acquired video. A region of interest is a part of the body surface region in a frame image, and is a region that includes multiple pixels. For example, if the video includes the face region of living body A, the regions of interest are the cheek and forehead.

The pulse wave calculation unit 303 calculates multiple pulse waves corresponding to different positions on the body surface from the moving image. In this specification, a pulse wave is a time-series signal indicating a change in the volume of blood vessels calculated from a time-series signal indicating pixel values (e.g., luminance values) of each frame image for the same position on the body surface. In this specification, a pixel value is information indicating the brightness of a pixel in a moving image, and is, for example, the pixel value or luminance value of each pixel for each of R (Red), G (Green), and B (Blue). In this embodiment, the same position on the body surface is the same pixel in the frame image as long as the positional relationship between the imaging unit 101 and the living body A is constant. Also, in this embodiment, different positions on the body surface are each pixel in the region of interest. Therefore, in this embodiment, the pulse wave calculation unit 303 calculates a time-series signal indicating a change in the volume of blood vessels from the temporal change in pixel value for each pixel in the region of interest.

The similarity calculation unit 304 calculates the similarity between at least two of the multiple pulse waves. Specifically, the similarity calculation unit 304 calculates the similarity based on at least one of the correlation value and the difference value between at least two of the multiple pulse waves. In the following explanation, for convenience of explanation, this similarity is referred to as "pulse wave similarity." Also, in the following explanation, for convenience of explanation, at least two pulse waves related to one pulse wave similarity are referred to as "pulse waves of interest."

The pulse wave synthesis unit 305 synthesizes pulse waves based on the pulse wave similarity. For example, the pulse wave synthesis unit 305 synthesizes at least two pulse waves calculated by the pulse wave calculation unit 303 based on the pulse wave similarity. In the following explanation, for convenience of explanation, the at least two pulse waves to be synthesized are referred to as "synthetic candidate pulse waves." Also, in the following explanation, for convenience of explanation, the pulse wave synthesized by the pulse wave synthesis unit 305 is referred to as the "synthetic pulse wave."

The output control unit 306 causes the output unit 203 to output the composite pulse wave. Alternatively, the output control unit 306 may transmit the composite pulse wave to a device other than the information processing device 100 via a communication means.

Next, the pulse waves corresponding to pixels in a moving image of the body surface will be described in detail. FIG. 4 shows an example of a region of interest 402 in one frame image 401. When the image acquisition unit 301 extracts a region of interest 402 of W vertical x H horizontal pixels from the frame image 401, the pulse wave calculation unit 303 calculates a pulse wave corresponding to each pixel in the region of interest 402. In other words, when the image acquisition unit 301 extracts a region of interest 402 of W vertical x H horizontal pixels, the pulse wave calculation unit 303 calculates W x H pulse waves. For example, the region of interest 402 illustrated in FIG. 4 includes 10 vertical x 10 horizontal pixels. Therefore, the pulse wave calculation unit 303 calculates 100 pulse waves corresponding to each pixel in the region of interest 402.

The similarity calculation unit 304 calculates at least two pulse wave similarities among the pulse waves corresponding to each pixel in the region of interest 402. For example, the similarity calculation unit 304 selects the pulse wave corresponding to pixel 403a in the region of interest 402 and the pulse wave corresponding to pixel 403b in the region of interest 402 as pulse waves of interest. The similarity calculation unit 304 then calculates the pulse wave similarity between the pulse wave corresponding to pixel 403a and the pulse wave corresponding to pixel 403b. In this manner, the similarity calculation unit 304 calculates the pulse wave similarity for the pulse waves corresponding to all pixels in the region of interest 402.

Figure 5 is a diagram showing an example of pixels corresponding to a composite candidate pulse wave. If the pulse wave similarity of the pulse waves corresponding to each pixel of the multiple pixels 501 satisfies the above-mentioned predetermined condition, the pulse wave synthesis unit 305 adds the pulse waves corresponding to the multiple pixels 501 to a composite pulse wave candidate list. Then, the pulse wave synthesis unit 305 synthesizes the pulse waves included in the composite pulse wave candidate list (i.e., the pulse waves corresponding to the multiple pixels 501). In this way, the pulse wave synthesis unit 305 synthesizes the pulse waves corresponding to the multiple positions that satisfy the above-mentioned predetermined condition to calculate a composite pulse wave.

Next, an example of a processing flow executed by the information processing device 100 of this embodiment will be described. FIG. 6 is a flowchart showing an example of processing executed by the information processing device 100 of this embodiment. In this example, after the imaging unit 101 starts capturing images, if the number of frames of moving images stored in the memory unit 201 exceeds a predetermined number of frames, the control unit 202 starts the processing of S601 shown in FIG. 6. It is also assumed that a composite candidate pulse wave has not been determined at the time of starting the processing of S601.

In S601, the image acquisition unit 301 acquires a moving image of the body surface from the storage unit 201. For example, the image acquisition unit 301 acquires the moving image from the storage unit 201 after a predetermined first time (e.g., 30 seconds) has elapsed since the imaging unit 101 started capturing images. The first time is the capturing time required for the information processing device 100 to calculate the pulse wave from the moving image. Alternatively, the image acquisition unit 301 may acquire the moving image from the storage unit 201 every second time (e.g., 1 second) that is shorter than the first time.

In S602, the region of interest extraction unit 302 extracts a region of interest 402 from each frame image 401 included in the moving image acquired in S601. Then, the region of interest extraction unit 302 outputs the extracted region of interest 402 to the pulse wave calculation unit 303. Alternatively, the region of interest extraction unit 302 may output the extracted region of interest 402 to the pulse wave calculation unit 303 in association with the name of a part of the face (cheek, forehead, etc.).

The region of interest extraction unit 302 may also determine whether or not a face region is included in the frame image 401. For example, the region of interest extraction unit 302 authenticates the face of living organism A from the acquired moving image. Here, the region of interest extraction unit 302 may authenticate the face of living organism A from the moving image using pattern matching, a support vector machine (SVM), a neural network, or the like, and the details of the authentication method are not important. For example, the region of interest extraction unit 302 extracts the region of interest 402 from a plurality of frame images 401 that contain face regions consecutively in time series, out of the frame images 401 that contain face regions.

In S603, the pulse wave calculation unit 303 calculates a plurality of pulse waves from the region of interest 402 extracted in S602. Specifically, the pulse wave calculation unit 303 extracts pixel values from each pixel in the region of interest 402. Then, when the number of frames of the moving image acquired in S601 is equal to or greater than the number of frames corresponding to a predetermined first time, the pulse wave calculation unit 303 extracts a signal indicating a temporal change in pixel value for each pixel in the region of interest 402 over the predetermined first time. The pulse wave calculation unit 303 processes the extracted signal using a predetermined signal processing method, and calculates the processed result as a pulse wave. For example, the signal processing method may be independent component analysis, principal component analysis, pigment component separation, bandpass filter, lowpass filter, or the like. Furthermore, the pulse wave calculation unit 303 may normalize the calculated pulse wave. For example, the pulse wave calculation unit 303 normalizes the pulse wave so that the maximum value of the pulse wave is 1 and the minimum value of the pulse wave is -1.

The pulse wave calculation unit 303 associates the calculated pulse wave with the pulse wave identification information and outputs it to the similarity calculation unit 304. The pulse wave identification information is information that identifies the pulse wave, and is, for example, information that identifies a position corresponding to the pulse wave (for example, the coordinate value of a pixel corresponding to the pulse wave in the frame image 401). Alternatively, the pulse wave identification information may be a number that is assigned to each position corresponding to the pulse wave and is different from the coordinate value in the frame image 401.

In S604, the similarity calculation unit 304 selects at least two pulse waves of interest from the multiple pulse waves calculated in S603. In other words, the similarity calculation unit 304 selects at least two pulse waves of interest that correspond to different pixels from among the multiple pixels in the region of interest 402.

In S605, the similarity calculation unit 304 calculates the pulse wave similarity of the pulse wave of interest. Specifically, the similarity calculation unit 304 calculates the pulse wave similarity for all combinations of pulse waves included in the pulse wave of interest. In this case, the similarity calculation unit 304 calculates _N C ₂ (=(N×(N−1)/2)) similarities. Here, N is the number of pulse waves included in the pulse wave of interest and is a natural number. Alternatively, the similarity calculation unit 304 may determine one reference pulse wave from at least two pulse waves included in the pulse wave of interest. Then, the similarity calculation unit 304 may calculate the pulse wave similarity between the determined reference pulse wave and a pulse wave other than the reference pulse wave included in the pulse wave of interest. In this case, the similarity calculation unit 304 calculates N−1 similarities.

For example, the similarity calculation unit 304 calculates the correlation coefficient r1 calculated using the following formula (1) as the pulse wave similarity. In the following description, the value of the pulse wave calculated from the pixel value (e.g., luminance value) of the position p on the body surface at time point i is denoted as x(p, i). In addition, in the following description, the pulse wave signal at time t at the position p on the body surface is denoted as X(p, t). That is, X(p, t) is a vector of x(p, i) (i is an integer), and X(p, t) = {x(p, 0), x(p, 1), ..., x(p, t), ...}. In addition, X(p1, t) and X(p2, t) shown in formula (1) are pulse wave signals at time t corresponding to the positions p1 and p2 on the body surface, respectively. For example, x(p1, i) and x(p2, i) indicate pixel values at time point i corresponding to different positions p1 and p2, respectively. For example, time point i is a frame number of a video sequence.

ここで、Ｔは動画像の時間長、つまり動画像に含まれるフレーム画像の枚数である。 For example, the expected value E(X(p1, t)) and the expected value E(X(p2, t)) shown in formula (1) are determined using the following formulas (2) and (3).

Here, T is the time length of the moving image, that is, the number of frame images contained in the moving image.

Alternatively, the similarity calculation unit 304 may calculate the pulse wave similarity using a correlation function R(τ) calculated using the following equation (4).

The similarity calculation unit 304 may determine the time difference τ _max when the correlation function R(τ) shown in formula (4) is maximized. The similarity calculation unit 304 may then calculate the value R(τ) of the cross-correlation function at the time difference τ _max as the pulse wave similarity.

Alternatively, the similarity calculation unit 304 may calculate the pulse wave similarity based on a difference value between the pulse waves of interest. For example, the similarity calculation unit 304 may use the following formula (5) to calculate the reciprocal r2 of the mean absolute deviation between the pulse waves of interest corresponding to the body surface position p1 and the body surface position p2 as the pulse wave similarity.

The above formulas (1), (4), and (5) are examples of formulas for calculating pulse wave similarity. The pulse wave similarity may be any index that indicates the similarity between pulse waves corresponding to different positions on the body surface, and is not limited to the correlation value, cross-correlation value, and difference value calculated by the above formulas.

In S606, the similarity calculation unit 304 associates the pulse wave similarity calculated in S605 with the pulse wave of interest corresponding to the pulse wave similarity and stores them in the memory unit 201.

In S607, the similarity calculation unit 304 determines whether or not pulse wave similarity has been calculated for all pulse waves. If pulse wave similarity has not been calculated for all pulse waves (S607: No), at least two new pulse waves of interest are selected from the multiple pulse waves (S608). The control unit 202 then returns the process to S605 to continue the process. On the other hand, if pulse wave similarity has been calculated for all pulse waves (S607: Yes), the control unit 202 transitions the process to S609.

In S609, the pulse wave synthesis unit 305 adds the pulse wave whose pulse wave similarity satisfies a predetermined condition to a list of pulse waves to be synthesized. For example, if the pulse wave similarity is equal to or greater than a predetermined threshold (e.g., threshold = 0.9), the pulse wave synthesis unit 305 determines that the pulse wave of interest corresponding to the pulse wave similarity satisfies the above-mentioned predetermined condition. Alternatively, the pulse wave synthesis unit 305 may determine that a predetermined number (e.g., 10) of pulse waves of interest, in descending order of pulse wave similarity, from among the pulse wave similarities stored in the memory unit 201, satisfy the above-mentioned predetermined condition.

In S610, the pulse wave synthesis unit 305 synthesizes the pulse waves included in the list of synthesis candidate pulse waves determined in S609. For example, the pulse wave synthesis unit 305 calculates the average value of pixel values (e.g., luminance values) at each time point in the synthesis candidate pulse waves determined in S609. The pulse wave synthesis unit 305 may calculate the average value of pixel values at each time point using an arithmetic mean or a geometric mean, for example. Then, the pulse wave synthesis unit 305 calculates a synthetic pulse wave based on a time series signal of the calculated average value.

Alternatively, the pulse wave synthesis unit 305 may weight the pixel values of the synthetic candidate pulse waves according to the pulse wave similarity corresponding to the synthetic candidate pulse wave. In this case, the pulse wave synthesis unit 305 synthesizes the weighted synthetic candidate pulse waves to calculate a synthetic pulse wave.

Alternatively, the pulse wave synthesis unit 305 may perform a process to remove the phase difference between the synthetic candidate pulse waves and calculate the mode or median of the synthetic candidate pulse waves. The pulse wave synthesis unit 305 then calculates the time series signal of the calculated mode or median as the synthetic pulse wave.

Alternatively, the pulse wave synthesis unit 305 may perform a process to remove the phase difference between the composite candidate pulse waves, and perform multivariate analysis (principal component analysis, independent component analysis, etc.) on the composite candidate pulse waves. The pulse wave synthesis unit 305 may then calculate the components obtained by the multivariate analysis as a composite pulse wave.

Alternatively, when the similarity calculation unit 304 calculates the value R(τ) of the cross-correlation function at the time difference τ _max using equation (4) as the pulse wave similarity, the pulse wave synthesis unit 305 may weight the pixel values of the synthesis candidate pulse waves in accordance with the time difference τ _max . In this case, the pulse wave synthesis unit 305 synthesizes the weighted synthesis candidate pulse waves.

In S611, the output control unit 306 causes the output unit 203 to output the composite pulse wave synthesized in S610. Alternatively, the output control unit 306 may transmit the composite pulse wave to a device other than the information processing device 100 via a communication means.

When the pulse wave calculation unit 303 calculates a pulse wave corresponding to one pixel, there is a risk that the calculated pulse wave will contain noise due to the influence of lighting during shooting, etc. Therefore, by combining pulse waves corresponding to multiple pixels, the noise contained in the pulse wave is smoothed. However, since the pulse waves of different blood vessels have different changes in blood vessel volume, when pulse waves of different blood vessels are combined, the pulse wave cannot be calculated accurately. On the other hand, for at least two pulse waves corresponding to different positions, the pulse wave similarity corresponding to the pulse wave of the same blood vessel is higher than the pulse wave similarity corresponding to the pulse wave of a different blood vessel. Therefore, by having the pulse wave combination unit 305 combine at least two pulse waves with a relatively high pulse wave similarity, the information processing device 100 can improve the signal-to-noise ratio of the calculated pulse wave.

Figure 7 shows an example of the relationship between the pixel corresponding to the pulse wave of interest and the blood vessel. For example, assume that the pulse wave calculation unit 303 calculates the pulse waves corresponding to pixels 701a and 701b in the region of interest 701. The two pulse waves corresponding to pixels 701a and 701b are pulse waves corresponding to blood vessel 711. The pulse wave similarity between the two pulse waves of interest corresponding to pixels 701a and 701b exceeds a predetermined threshold.

Similarly, it is assumed that the pulse wave calculation unit 303 has calculated pulse waves corresponding to pixels 702a and 702b in the region of interest 702. Here, the two pulse waves corresponding to 702a and pixel 702b are pulse waves corresponding to blood vessel 711. It is also assumed that the pulse wave similarity between the two pulse waves of interest corresponding to pixels 702a and 702b exceeds a predetermined threshold. In this case, the pulse wave synthesis unit 305 synthesizes the pulse waves corresponding to pixels 701a, 701b, 702a, and 702b to calculate a synthesized pulse wave.

On the other hand, suppose that the pulse wave calculation unit 303 has calculated pulse waves corresponding to pixels 703a and 703b within the region of interest 703. Here, the two pulse waves corresponding to pixels 703a and 703b are pulse waves corresponding to blood vessel 712. However, for example, the two pulse waves corresponding to pixels 701a and 703a are pulse waves of different blood vessels. Furthermore, suppose that the two pulse waves corresponding to pixels 701a and 703a are below a predetermined threshold. In that case, the composite candidate pulse wave does not include a combination of pulse waves corresponding to pixels 701a and 703a.

It is also assumed that the pulse wave calculation unit 303 has calculated pulse waves corresponding to pixels 704a and 704b in the region of interest 704. Here, the two pulse waves corresponding to pixels 704a and 704b are pulse waves corresponding to blood vessel 712. It is also assumed that the pulse wave similarity between the two pulse waves of interest corresponding to pixels 704a and 704b exceeds a predetermined threshold. In this case, the pulse wave synthesis unit 305 synthesizes the pulse waves corresponding to pixels 703a, 703b, 704a, and 704b to calculate a synthetic pulse wave.

Figures 8A and 8B are diagrams showing an example of a pulse wave. Pulse wave X (701a, t) shown in Figure 8A shows a pulse wave corresponding to pixel 701a shown in Figure 7. Pulse wave X (702a, t) shown in Figure 8A shows a pulse wave X corresponding to pixel 702a shown in Figure 7. In other words, pulse wave X (701a, t) and pulse wave X (702a, t) are pulse waves corresponding to blood vessel 711 shown in Figure 7.

Here, it is assumed that the pulse wave similarity between the pulse wave X (701a, t) and the pulse wave X (702a, t) satisfies a predetermined condition. Furthermore, as shown in FIG. 8A, it is assumed that the pulse wave X (702a, t) is delayed by a time difference τ1 from the pulse wave X (701a, t). In this case, the pulse wave synthesis unit 305 synthesizes the pulse wave X (701a, t) with the pulse wave X (701a, t) by shifting the pulse wave X (702a, t) in the time axis direction by the time difference τ1. Here, the pulse wave X (702a, t) = {x (702a, τ1), x (702a, τ1 + 1), ..., x (702a, τ1 + t), ...}. The time difference τ1 is, for example, the above-mentioned time difference τ _max . That is, the pulse wave synthesis unit 305 performs a process of removing the phase difference between the synthesis candidate pulse waves, and synthesizes the pulse waves.

On the other hand, pulse wave X (703a, t) shown in FIG. 8B indicates a pulse wave corresponding to pixel 703a shown in FIG. 7. Pulse wave X (704a, t) shown in FIG. 8B indicates a pulse wave corresponding to pixel 704a shown in FIG. 7. In other words, pulse wave X (703a, t) and pulse wave X (704a, t) are pulse waves corresponding to blood vessel 712 illustrated in FIG. 7.

Similarly, it is assumed that the pulse wave similarity between the pulse wave X (703a, t) and the pulse wave X (704a, t) satisfies a predetermined condition. Furthermore, as shown in FIG. 8B, it is assumed that the pulse wave X (704a, t) is delayed from the pulse wave X (703a, t) by a time difference τ2. In this case, the pulse wave synthesis unit 305 synthesizes the pulse wave X (703a, t) with the pulse wave X (703a, t), which is obtained by shifting the pulse wave X (704a, t) in the time axis direction by the time difference τ2. Here, the pulse wave X3 (704a, t) = {x (704a, τ2), x (704a, τ2 + 1), ..., x (704a, τ2 + t), ...}. The time difference τ2 is, for example, the above-mentioned time difference τ _max .

As described above, the pulse wave synthesis unit 305 synthesizes the pulse waves of the same blood vessels to calculate a synthetic pulse wave. As a result, the information processing device 100 can calculate the pulse wave more accurately than when outputting a pulse wave corresponding to a single pixel.

(Variation 1) The image acquisition unit 301 may determine the entire frame image 401 as the region of interest 402. In other words, the pulse wave calculation unit 303 may calculate a pulse wave for each pixel in the frame image 401. That is, the pulse wave synthesis unit 305 synthesizes a pulse wave from a moving image. Specifically, the pulse wave synthesis unit 305 synthesizes a pulse wave based on the pulse wave similarity corresponding to the pulse wave calculated from each pixel of the frame image 401.

(Variation 2) A device other than the information processing device 100 may include the imaging unit 101. In this case, the information processing device 100 acquires a moving image generated by the imaging unit 101 from the device other than the information processing device 100 via a network or the like.

(Variation 3) The information processing device 100 may have a function for tracking living organism A. In this case, the information processing device 100 can track living organism A and acquire moving images of the body surface of the same living organism A. As a result, the information processing device 100 can calculate the pulse wave from the same position on the body surface even when living organism A is moving. For example, the imaging unit 101 may move in accordance with the movement of living organism A, and each pixel of the imaging unit 101 may continue to capture the same position on the body surface of living organism A. Also, for example, the same position of living organism A may be tracked between image frames by image processing. In this case, the same position on the body surface is not necessarily the same pixel in the frame image.

Second Embodiment
A second embodiment will be described. In each of the following embodiments, configurations and processes having substantially the same functions as those of the first embodiment will be referred to by the same reference numerals, and descriptions thereof will be omitted, and only differences from the first embodiment will be described.

FIG. 9 is a block diagram showing an example of the functional configuration of the control unit 202 of this embodiment. The control unit 202 illustrated in FIG. 9 differs from the control unit 202 illustrated in FIG. 3 in that the control unit 202 further includes a biometric information extraction unit 901.

The biometric information extraction unit 901 extracts biometric information based on the composite pulse wave. The biometric information includes at least one of pulse rate, stress level, and blood pressure. In this embodiment, calculating and estimating the biometric information is referred to as "extracting biometric information."

For example, the biometric information extraction unit 901 counts the number of peaks in the composite pulse wave and calculates the counted number of peaks as the pulse rate.

For example, the biometric information extraction unit 901 performs a fast Fourier transform on a predetermined time interval in the composite pulse wave to calculate a power spectrum. The biometric information extraction unit 901 then analyzes the power spectrum to calculate the stress level.

For example, the biometric information extraction unit 901 also performs two time derivatives on the composite pulse wave to calculate an acceleration pulse wave, and estimates blood pressure from the acceleration pulse wave.For example, the biometric information extraction unit 901 also estimates blood pressure from the shape of the composite pulse wave.

As described above, the composite pulse wave has a higher signal-to-noise ratio than a pulse wave corresponding to a single position on the body surface (e.g., a single pixel). Therefore, the biometric information extraction unit 901 can extract more reliable biometric information than when biometric information is extracted based on a pulse wave corresponding to a single position on the body surface.

Third Embodiment
A third embodiment will now be described. Fig. 10 is a block diagram showing an example of a functional configuration of a control unit 202 according to this embodiment. The control unit 202 shown in Fig. 10 differs from the control unit 202 shown in Fig. 3 in that it further includes a region determination unit 1001. Furthermore, the control unit 202 according to this embodiment includes a pulse wave calculation unit 1002 instead of the pulse wave calculation unit 303.

The region determination unit 1001 determines multiple regions each containing multiple pixels from a frame image included in the moving image acquired by the image acquisition unit 301.

The pulse wave calculation unit 1002 calculates one pulse wave from each region. Specifically, the pulse wave calculation unit 1002 calculates each of the multiple pulse waves based on a statistical value (e.g., a total value or an average value) of the pixel values of the multiple regions. For example, the pulse wave calculation unit 1002 calculates a statistical value of the pixel values of the regions for each frame image, and calculates a signal indicating the change over time in the calculated statistical value. The pulse wave calculation unit 1002 then processes the calculated signal using a predetermined signal processing method, and calculates the processed result as a pulse wave.

The similarity calculation unit 304 calculates the similarity between at least two of the pulse waves corresponding to the multiple regions calculated by the pulse wave calculation unit 1002. For example, the pulse wave similarity is calculated for combinations of pulse waves in all regions. In this case, the similarity calculation unit 304 calculates _N C ₂ (=(N×(N−1)/2)) similarities. Here, N is the number of regions and is a natural number. Alternatively, the similarity calculation unit 304 may determine one reference pulse wave from the pulse waves in at least two regions. Then, the similarity calculation unit 304 may calculate the pulse wave similarity between the determined reference pulse wave and a pulse wave other than the reference pulse wave. In this case, the similarity calculation unit 304 calculates N−1 similarities.

The pulse wave synthesis unit 305 synthesizes a synthetic pulse wave based on the pulse wave similarity calculated by the similarity calculation unit 304. For example, the pulse wave synthesis unit 305 synthesizes pulse waves in at least two regions set by the region determination unit 1001 based on the pulse wave similarity. In the following explanation, for convenience of explanation, the at least two regions to be synthesized are referred to as "synthesis candidate regions."

Next, the pixels corresponding to the pulse wave in the moving image of the body surface will be described in detail. FIG. 11 is a diagram showing an example of a region of interest. The region determination unit 1001 determines a region from the region of interest 402. Regions 1101, 1102, 1103, and 1104 shown in FIG. 11 are examples of regions.

Here, the memory unit 201 stores the pulse wave corresponding to each region within the region of interest 402 in association with the pulse wave similarity corresponding to the pulse wave.

For example, the pulse wave synthesis unit 305 determines a synthesis candidate region including multiple regions within the region of interest 402. The pulse wave synthesis unit 305 determines multiple synthesis candidate regions from regions 1101, 1102, 1103, and 1104 based on the pulse wave similarity corresponding to each region. For example, the pulse wave synthesis unit 305 determines regions 1101 and 1102 as synthesis candidate regions. In this case, the pulse wave synthesis unit 305 calculates a synthetic pulse wave as a single pulse wave based on a signal indicating the temporal change in the statistical values of the pixel values of the pixels included in regions 1101 and 1102.

This allows the pulse wave synthesis unit 305 to synthesize pulse waves of interest that have a relatively high degree of pulse wave similarity to calculate a synthetic pulse wave. Therefore, for the same reason as above, the signal-to-noise ratio of the calculated synthetic pulse wave is improved. Furthermore, for the same reason as above, the output unit 203 can output the pulse wave of one blood vessel more accurately than when outputting a pulse wave corresponding to a single position on the body surface (e.g., a single pixel).

(Fourth embodiment)
12 is a block diagram showing an example of the functional configuration of the control unit 202 of the fourth embodiment. The control unit 202 of the present embodiment includes a pulse wave calculation unit 1201 instead of the pulse wave calculation unit 303.

The pulse wave calculation unit 1201 calculates one of the multiple pulse waves corresponding to a pixel included in a moving image based on a statistical value (e.g., a sum or average value) of the pixel value of the pixel and the pixel value of at least one pixel adjacent to the pixel. In the following explanation, for convenience of explanation, at least one pixel adjacent to the pixel corresponding to the pulse wave is referred to as a "surrounding pixel."

Figure 13 is a diagram showing an example of a pixel corresponding to a pulse wave and surrounding pixels. For example, for a pixel 1301 in the region of interest 402, the pulse wave calculation unit 1201 calculates a statistical value of the pixel value of pixel 1301 and the pixel values of eight surrounding pixels 1302 adjacent to pixel 1301. The pulse wave calculation unit 1201 calculates the pulse wave corresponding to pixel 1301 based on a signal indicating the change over time in the statistical value of pixel 1301.

As shown in FIG. 13, the peripheral pixels 1302 of pixel 1301 are eight pixels surrounding pixel 1301. When image acquisition unit 301 extracts region of interest 402 of W pixels vertical by H pixels horizontal, pulse wave calculation unit 303 calculates (W-2) x (H-2) pulse waves. For example, region of interest 402 shown in FIG. 13 includes 10 pixels vertical by 10 pixels horizontal. In this case, pulse wave calculation unit 303 calculates 64 (= (10-2) x (10-2)) pulse waves corresponding to each pixel in region 1303 shown by dashed lines in FIG. 13 based on the statistical values of each pixel and the peripheral pixels. Note that peripheral pixels 1302 shown in FIG. 13 are eight pixels surrounding pixel 1301, but peripheral pixels 1302 may be fewer than eight pixels. Furthermore, the number of peripheral pixels 1302 may be more than eight pixels. For example, the surrounding pixels 1302 may include pixels surrounding pixel 1301 as well as adjacent pixels.

This allows the pulse wave calculation unit 1201 to smooth out noise contained in the calculated pulse wave more easily than when the pulse wave is calculated from the pixel value of a single pixel. As a result, the pulse wave calculation unit 1201 can improve the signal-to-noise ratio of the calculated pulse wave more easily than when the pulse wave is calculated from the pixel value of a single pixel, and can calculate the pulse wave more accurately.

Fifth Embodiment
A fifth embodiment will be described. Fig. 14 is a block diagram showing an example of a functional configuration of the control unit 202 of this embodiment. The control unit 202 shown in Fig. 14 is different from the control unit 202 shown in Fig. 3 in that it further includes a feature extraction unit 1401. Furthermore, the control unit 202 of this embodiment includes a similarity calculation unit 1402 and a pulse wave synthesis unit 1403 instead of the similarity calculation unit 304 and the pulse wave synthesis unit 305.

The feature extraction unit 1401 extracts features that indicate the characteristics of the shape of the pulse wave. In the following explanation, for convenience of explanation, the feature is referred to as a "pulse wave feature." The pulse wave feature is a value related to a feature point in the pulse wave, and is, for example, at least one of the period, amplitude, frequency, onset angle, and spectrum. In the following explanation, for convenience of explanation, the feature point is referred to as a "pulse wave feature point." The pulse wave feature point is a maximum point, minimum point, inflection point, etc. of the pulse wave. Alternatively, the pulse wave feature point may be a signal obtained by first or multiple differentiation of the pulse wave, or a maximum point, minimum point, inflection point, etc. of a spectrum obtained by frequency analysis.

The similarity calculation unit 1402 calculates the pulse wave similarity based on the comparison results of at least two pulse wave features.

The pulse wave synthesis unit 1403 synthesizes at least two pulse waves based on the pulse wave similarity calculated by the similarity calculation unit 1402.

FIG. 15 is a diagram showing an example of a pulse wave feature. In the following description, FIG. 15 shows an example of a pulse wave X1501. t11 shown in FIG. 15 indicates a time point corresponding to the minimum value a15 in the pulse wave X1501. Also, t12 shown in FIG. 15 indicates a time point corresponding to the maximum value a16 in the pulse wave X1501. h shown in FIG. 15 indicates a difference value between the minimum value a15 at time t11 and the maximum value a16 at time t12 on the pulse wave X1501. For example, the feature extraction unit 1401 calculates h=(a16-a15) shown in FIG. 15 as one of the pulse wave feature values. In the following description, for convenience of explanation, the difference value between adjacent minimum and maximum points in one pulse wave is referred to as "wave height."

15 indicates the rising angle of the pulse wave X1501 between time t11 and time t12. For example, the feature amount extraction unit 1401 calculates the rising angle θ=tan ⁻¹ ((a16−a15)/(t12−t11)) (rad) shown in FIG. 15 as one of the pulse wave feature amounts.

Next, an example of a processing flow executed by the information processing device 100 of this embodiment will be described. FIG. 16 is a flowchart showing an example of a processing flow executed by the information processing device 100 of this embodiment. In this example, after the imaging unit 101 starts capturing images, if the number of frames of moving images stored in the storage unit 201 exceeds a predetermined number of frames, the control unit 202 of this embodiment executes the processing of S601 to S604 shown in FIG. 6. Then, when the control unit 202 executes the processing of S604 shown in FIG. 6, it starts the processing of S1601 shown in FIG. 16.

In S1601, the feature extraction unit 1401 extracts pulse wave features from the pulse waves of interest selected in S604. For example, the feature extraction unit 1401 extracts pulse wave feature points from each of the selected pulse waves of interest. Then, the feature extraction unit 1401 extracts pulse wave features from the pulse waves of interest based on the extracted pulse wave feature points.

When the feature extraction unit 1401 extracts multiple maximum points and multiple minimum points from one pulse wave, it extracts multiple wave heights h based on the maximum points and minimum points. For example, when the feature extraction unit 1401 extracts 10 combinations of maximum points and minimum points from one pulse wave, it calculates 10 wave heights h.

Furthermore, when the feature extraction unit 1401 extracts multiple pulse wave feature amounts of the same type, it outputs a statistical value (e.g., an average value or a median value) of the multiple extracted pulse wave feature amounts as a pulse wave feature amount to the similarity calculation unit 1402. Alternatively, the feature extraction unit 1401 outputs the multiple extracted pulse wave feature amounts to the similarity calculation unit 1402.

Alternatively, the feature extraction unit 1401 may associate the pulse wave feature with the feature identification information and output the result to the similarity calculation unit 1402. The feature identification information is information for identifying the pulse wave feature, and is, for example, information indicating a time point corresponding to the pulse wave feature, and information for identifying a pulse wave feature point corresponding to the pulse wave feature. The information for identifying the pulse wave feature point is, for example, a number assigned to each pulse wave feature point in one pulse wave.

In S1602, the similarity calculation unit 1402 compares the pulse wave feature amounts. Specifically, the similarity calculation unit 1402 compares the pulse wave feature amounts of each of the at least two pulse waves of interest. For example, when the feature extraction unit 1401 extracts multiple pulse wave feature amounts, the pulse wave feature amounts are compared.

In S1603, the similarity calculation unit 1402 calculates pulse wave similarity based on the comparison result of each pulse wave feature corresponding to at least two pulse waves of interest. Specifically, the similarity calculation unit 1402 calculates pulse wave similarity based on the comparison result of each pulse wave feature corresponding to at least two pulse waves of interest. Then, the control unit 202 shifts the process to S606. Then, the control unit 202 executes the processes of S606 to S608. Then, when the pulse wave similarity has been calculated for all pulse waves (S607: Yes), the control unit 202 shifts the process to S1604.

For example, the similarity calculation unit 1402 calculates the pulse wave similarity based on the difference value of pulse wave feature amounts of the same type corresponding to at least two pulse waves of interest. The difference value of the pulse wave feature amount is the difference between the feature amounts of the at least two pulse waves of interest. Here, the similarity calculation unit 1402 may calculate the mean square error of the feature amounts of the at least two pulse waves of interest as the difference value of the pulse wave feature amounts. Then, the similarity calculation unit 1402 outputs the calculated pulse wave similarity to the pulse wave synthesis unit 1403.

Furthermore, when the feature extraction unit 1401 extracts pulse wave feature values of multiple pulse waves of interest of the same type for at least two pulse waves of interest, the similarity calculation unit 1402 calculates difference values between the multiple pulse wave feature values of one pulse wave of interest and the multiple pulse wave feature values of a pulse wave of interest different from the one pulse wave of interest. That is, the similarity calculation unit 1402 calculates multiple difference values for pulse wave feature values of the same type. In this case, the similarity calculation unit 1402 outputs a statistical value (e.g., an average value or a median value) of the multiple calculated difference values to the pulse wave synthesis unit 1403 as pulse wave similarity. Alternatively, when the similarity calculation unit 1402 calculates multiple difference values for pulse wave feature values of the same type, it may output the multiple difference values to the pulse wave synthesis unit 1403 as pulse wave similarity.

For example, suppose that the feature extraction unit 1401 extracts multiple (e.g., 10) wave height waveform feature values for each of the two pulse waves, such as h1, h2, .... In this case, the similarity calculation unit 1402 calculates the difference value between the wave height h1 corresponding to one pulse wave and the wave height h1 corresponding to the other pulse wave, the difference value between the wave height h2 corresponding to one pulse wave and the wave height h2 corresponding to the other pulse wave, .... Here, the wave height h1 corresponding to one pulse wave and the wave height h1 corresponding to the other pulse wave may be feature values related to the pulse wave at the same time. Then, the similarity calculation unit 1402 outputs the difference value of the extracted wave heights h1, h2, ... to the pulse wave synthesis unit 1403 as the pulse wave similarity. Here, the average value of the difference values of the extracted wave heights h1, h2, ... may be output to the pulse wave synthesis unit 1403.

Furthermore, when the feature extraction unit 1401 extracts pulse wave feature values of multiple types of pulse waves of interest for at least two pulse waves of interest, the similarity calculation unit 1402 calculates a difference value between the pulse wave feature value of one of the multiple types of pulse waves of interest and the pulse wave feature value of a pulse wave of interest different from the one of the multiple types of pulse waves of interest. In other words, the similarity calculation unit 1402 calculates each difference value for the multiple types of pulse wave feature values.

For example, suppose that the feature extraction unit 1401 extracts pulse wave features including a wave height h and a rising angle θ for each of two pulse waves. In this case, the similarity calculation unit 1402 calculates a difference value between the wave height h corresponding to one pulse wave and the wave height h corresponding to the other pulse wave. Similarly, the similarity calculation unit 1402 calculates a difference value between the rising angle θ corresponding to one pulse wave and the rising angle θ corresponding to the other pulse wave. The similarity calculation unit 1402 then outputs the extracted difference value of the wave height h and the extracted difference value of the rising angle θ to the pulse wave synthesis unit 1403 as pulse wave similarity.

In S1604, the pulse wave synthesis unit 1403 adds to the synthesis candidate pulse wave list the pulse wave whose pulse wave similarity stored in the memory unit 201 in S606 satisfies a predetermined condition. Then, the control unit 202 transitions the process to S610.

For example, the similarity calculation unit 1402 calculates the difference value between pulse wave feature quantities of the same type as the pulse wave similarity. In this case, the smaller the difference value of the pulse wave feature quantities, the more similar the characteristics of the at least two pulse waves of interest corresponding to that difference value. Therefore, when the difference value of the pulse wave feature quantities corresponding to at least two pulse waves of interest is within a predetermined range, the pulse wave synthesis unit 1403 adds the at least two pulse waves of interest to a list of pulse waves to be synthesized.

Alternatively, the similarity calculation unit 1402 calculates multiple difference values between pulse wave feature quantities of the same type for multiple types of pulse wave feature quantities, and calculates the multiple calculated difference values as pulse wave similarities. When the similarity calculation unit 1402 determines that the difference value of one pulse wave feature quantity among the multiple types of pulse wave feature quantities is within a predetermined range, the pulse wave synthesis unit 1403 adds at least two pulse waves of interest that correspond to the difference value to a list of pulse waves to be synthesized.

Alternatively, if the difference value of more than a predetermined number of pulse wave feature quantities among multiple types of pulse wave feature quantities is within a predetermined range, the similarity calculation unit 1402 causes the pulse wave synthesis unit 1403 to add at least two pulse waves of interest corresponding to the difference value to the list of pulse waves to be synthesized.

Alternatively, the pulse wave synthesis unit 1403 may determine the priority of the pulse wave of interest for each type of pulse wave feature among the multiple types of pulse wave feature, in ascending order of the difference value of the pulse wave feature. Alternatively, the pulse wave synthesis unit 1403 may determine the priority of the pulse wave of interest for multiple types of pulse wave feature, in ascending order of the difference value of the pulse wave feature. In this case, the pulse wave synthesis unit 1403 determines the pulse wave of interest to add to the synthesis candidate pulse wave list based on the determined priority.

As described above, the information processing device 100 of this embodiment can reduce the effect of noise contained in the pulse waves by comparing pulse wave feature values and calculating pulse wave similarity, compared to when the entire pulse waves are compared to calculate pulse wave similarity. Furthermore, the information processing device 100 of this embodiment can reduce the amount of calculation required to calculate pulse wave similarity by comparing pulse wave feature values, compared to when the entire pulse waves are compared.

Sixth Embodiment
A sixth embodiment will be described. Fig. 17 is a block diagram showing an example of a functional configuration of a control unit 202 of this embodiment. The control unit 202 shown in Fig. 17 is different from the control unit 202 shown in Fig. 3 in that a part detection unit 1701 is further provided. Furthermore, the control unit 202 of this embodiment includes a pulse wave calculation unit 1702, a similarity calculation unit 1703, and a pulse wave synthesis unit 1704 instead of the pulse wave calculation unit 303, the similarity calculation unit 304, and the pulse wave synthesis unit 305.

The body part detection unit 1701 detects a first body part of the living body A and a second body part different from the first body part of the living body A from the moving image acquired by the image acquisition unit 301. For example, the first body part is the neck and the second body part is the face.

The pulse wave calculation unit 1702 calculates a first pulse wave corresponding to a first part of a living body from the moving image, and at least two second pulse waves corresponding to a second part different from the first part.

The similarity calculation unit 1703 calculates the pulse wave similarity between the first pulse wave and at least two second pulse waves.

The pulse wave synthesis unit 1704 synthesizes at least two second pulse waves when the pulse wave similarity between the first pulse wave and at least two second pulse waves exceeds a predetermined threshold.

Next, an example of a process flow executed by the information processing device 100 of this embodiment will be described. FIG. 18 is a flowchart showing an example of a process flow executed by the information processing device 100 of this embodiment. In this example, the first part is the neck of living body A, and the second part is the face of living body A. Also, in this example, after the imaging unit 101 starts capturing images of the neck and face of living body A, if the number of frames of the moving image stored in the memory unit 201 exceeds a predetermined number of frames, the control unit 202 starts the process of S1801 shown in FIG. 18.

In S1801, the image acquisition unit 301 acquires a moving image of the body surface from the storage unit 201. Each frame image 401 of the moving image includes pixel values of the neck region and face region of living body A.

In S1802, the body part detection unit 1701 identifies the neck position and face position from the video image acquired in S1801.

For example, the part detection unit 1701 detects facial feature points (e.g., both ends of the eyes, the nose, and both ends of the mouth) from each frame image 401, and identifies the position of the face (second part) contained in each frame image 401. Here, the position of the face is an area including multiple pixels. The position of the face of living body A may be identified from the moving image using pattern matching, a support vector machine (SVM), a neural network, etc., and the details of the method for identifying the position of the face are not important.

For example, the body part detection unit 1701 identifies the position of the neck (first body part) from a region of the body surface different from the identified position of the face in each frame image 401. Here, the position of the neck is an area including multiple pixels.

In S1803, the pulse wave calculation unit 1702 calculates a first pulse wave from the neck position identified in S1802. Here, the pulse wave calculation unit 1702 may calculate two first pulse waves corresponding to the left and right carotid arteries.

For example, the pulse wave calculation unit 1702 calculates multiple pulse waves (hereinafter referred to as "first pulse wave candidates") based on the temporal change in pixel value of each pixel included in the neck position. Then, the pulse wave calculation unit 1702 determines the first pulse wave from the multiple calculated first pulse wave candidates. For example, the pulse wave calculation unit 1702 determines the pulse wave with the largest statistical value (e.g., average value) of the pulse wave amplitude as the first pulse wave from the multiple calculated first pulse wave candidates.

Alternatively, the pulse wave calculation unit 1702 determines, as the first pulse wave, the pulse wave with the maximum signal-to-noise ratio among the calculated first pulse wave candidates. For example, the pulse wave calculation unit 1702 performs a fast Fourier transform on the calculated first pulse wave candidates for a predetermined time interval to calculate a power spectrum. Then, the pulse wave with the maximum statistical value (e.g., maximum value) of the power spectrum in a predetermined frequency band (e.g., 0.5 Hz to 4.0 Hz) may be determined as the first pulse wave. Alternatively, for example, the pulse wave with the maximum ratio between the integral value (signal) of the power spectrum in a predetermined frequency band (e.g., 0.5 Hz to 4.0 Hz) and the integral value (noise) of the power spectrum outside the predetermined frequency band may be determined as the first pulse wave.

In S1804, the region of interest extraction unit 302 extracts a region of interest 402 from the face position identified in S1802. If the pulse wave calculation unit 1702 calculates two first pulse waves corresponding to the left and right carotid arteries, the region of interest extraction unit 302 extracts two regions of interest 402 corresponding to the left and right cheeks from the identified face position.

In S1805, the pulse wave calculation unit 1702 calculates a plurality of second pulse waves from the region of interest 402. When the region of interest extraction unit 302 extracts two regions of interest 402 corresponding to the left and right cheeks, the pulse wave calculation unit 1805 calculates a second pulse wave from each of the regions of interest 402.

In S1806, the similarity calculation unit 1703 selects at least two pulse waves of interest from the multiple second pulse waves.

In S1807, the similarity calculation unit 1703 calculates the pulse wave similarity between the first pulse wave and the pulse wave of interest selected in S1806. That is, the similarity calculation unit 1703 calculates the pulse wave similarity between the pulse wave at the neck position and the pulse wave at the face position. Specifically, when the pulse wave calculation unit 1702 calculates two first pulse waves corresponding to the left and right carotid arteries, the similarity calculation unit 1703 calculates the pulse wave similarity between the first pulse wave corresponding to the left carotid artery and multiple second pulse waves corresponding to the left cheek.

Similarly, when the pulse wave calculation unit 1702 calculates two first pulse waves corresponding to the left and right carotid arteries, the similarity calculation unit 1703 calculates the pulse wave similarity between the first pulse wave corresponding to the right carotid artery and multiple second pulse waves corresponding to the right cheek.

Here, since the carotid artery is thicker than the blood vessels in the face, the change in the volume of the carotid artery accompanying the pulsation of the heart is greater than the change in the volume of the blood vessels in the face. Therefore, the change in color of the body surface of the carotid artery is more noticeable than the change in color of the body surface of the face. In other words, the pulse wave corresponding to the pixels of the carotid artery calculated by the pulse wave calculation unit 1702 has a higher signal-to-noise ratio than the pulse wave corresponding to the pixels of the face. Therefore, by calculating the pulse wave similarity between the pulse wave of the carotid artery and the pulse wave of the face, the similarity calculation unit 1703 can calculate the pulse wave similarity of pulse waves connected to the same blood vessel more accurately than when calculating the pulse wave similarity between pulse waves in the face.

In S1808, the similarity calculation unit 1703 associates the pulse wave similarity calculated in S1807 with the pulse wave of interest corresponding to the pulse wave similarity and stores them in the memory unit 201.

In S1809, the similarity calculation unit 1703 determines whether or not pulse wave similarities have been calculated for all second pulse waves. If pulse wave similarities have not been calculated for all second pulse waves (S1809: No), the similarity calculation unit 1703 selects at least two new pulse waves of interest from the multiple second pulse waves (S1810). The control unit 202 then returns the process to S1807 to continue the process. On the other hand, if pulse wave similarities have been calculated for all second pulse waves (S1809: Yes), the control unit 202 transitions the process to S1811.

In S1811, the pulse wave synthesis unit 1704 adds the second pulse wave whose pulse wave similarity satisfies a predetermined condition to the synthesis candidate pulse wave list. This allows the pulse wave synthesis unit 1704 to add pulse waves corresponding to blood vessels connected to the same carotid artery, among the pulse waves corresponding to blood vessels in the face, to the synthesis candidate pulse wave list. The details of the processing of S1811 are the same as those of 609 above, so a detailed description will be omitted.

In S1812, the pulse wave synthesis unit 1704 synthesizes the second pulse wave included in the list of synthesis candidate pulse waves.

In S1813, the pulse wave synthesis unit 1704 outputs the synthesized second pulse wave.

Figure 19 shows an example of the relationship between the pixels corresponding to the first and second pulse waves and the blood vessels. Figure 19 shows a frame image 1901 including the face and neck area of living body A. Blood vessel 1903 is a blood vessel in the face that connects to the carotid artery 1902. Blood vessel 1905 is a blood vessel in the face that connects to the carotid artery 1904. Pixel 1906 is a pixel located corresponding to the carotid artery 1902. Pixel 1907 is a pixel located corresponding to the carotid artery 1904.

The pulse wave calculation unit 1702 calculates a first pulse wave corresponding to pixel 1907. Furthermore, when the region of interest extraction unit 302 extracts a region of interest 1909, the pulse wave calculation unit 1702 calculates a second pulse wave corresponding to each pixel in the region of interest 1909. For example, the similarity calculation unit 1703 calculates the pulse wave similarity between the first pulse wave corresponding to pixel 1907 and the second pulse wave corresponding to pixel 1914. Furthermore, the similarity calculation unit 1703 calculates the pulse wave similarity between the first pulse wave corresponding to pixel 1907 and the second pulse wave corresponding to pixel 1915.

Here, the first pulse wave corresponding to pixel 1907 is a pulse wave of the carotid artery 1904, and therefore has a higher signal-to-noise ratio than the second pulse wave corresponding to the facial blood vessel 1905. Therefore, by the similarity calculation unit 1703 calculating the pulse wave similarity between the first pulse wave corresponding to pixel 1907 and the second pulse wave corresponding to pixel 1915, it is possible to calculate the pulse wave similarity more accurately than when calculating the pulse wave similarity between the second pulse wave corresponding to pixel 1914 and the second pulse wave corresponding to pixel 1915.

The pulse wave synthesis unit 1704 determines, from at least two pulse waves corresponding to pixels in the region of interest 1909, a pulse wave whose pulse wave similarity to the first pulse wave corresponding to pixel 1907 satisfies a predetermined condition as a synthetic candidate pulse wave. This results in a higher signal-to-noise ratio for the synthetic pulse wave than when the pulse wave synthesis unit 1704 determines a synthetic candidate pulse wave based on the pulse wave similarity of pulse waves corresponding to at least two pixels in the region of interest 1909 and calculates a synthetic pulse wave.

Similarly, when the region of interest extraction unit 302 extracts the region of interest 1908, the pulse wave calculation unit 1702 calculates the second pulse wave corresponding to each pixel in the region of interest 1908. Here, the first pulse wave corresponding to pixel 1906 is the pulse wave of the carotid artery 1902, and therefore has a higher signal-to-noise ratio than the pulse wave corresponding to pixel 1912 at a position corresponding to the blood vessel 1903 in the face. Therefore, similarly to the above, the similarity calculation unit 1703 calculates the pulse wave similarity between the first pulse wave corresponding to pixel 1906 and the second pulse wave corresponding to pixel 1913, and thereby the pulse wave similarity can be calculated more accurately than when the pulse wave similarity between the second pulse wave corresponding to pixel 1912 and the second pulse wave corresponding to pixel 1913 is calculated.

The output control unit 306 calculates composite pulse waves corresponding to the left and right sides of the face. If there is no abnormality in the blood vessels from the heart to the face, the pulse waves on the left and right sides of the face will be similar to each other. However, if there is an abnormality in the blood vessels from the heart to the face, the shape of one of the pulse waves on the left and right sides of the face may change from normal. As a result, if there is an abnormality in the blood vessels from the heart to the face, the pulse waves on the left and right sides of the face will be different. Therefore, the difference between the pulse waves on the left and right sides of the face can be an indicator for determining whether or not there is an abnormality in the blood vessels from the heart to the face. Therefore, the output control unit 306 may output the difference between the pulse waves on the left and right sides of the face. This allows the information processing device 100 to output information indicating that there is an abnormality in the blood vessels from the heart to the face.

The subject of the information processing device or pulse wave calculation method of the present disclosure is a computer equipped with a control unit 202. This computer executes a program stored in the storage unit 201 to realize the functions of the subject of the information processing device or pulse wave calculation method of the present disclosure. The computer has a processor that operates according to the above program as a main hardware configuration. The type of processor is not important as long as it can realize the functions by executing the above program. The processor is composed of one or more electronic circuits including a semiconductor integrated circuit (IC) or an LSI (Large Scale Integration). The multiple electronic circuits may be integrated into one chip or may be provided on multiple chips. The multiple chips may be integrated into one device or may be provided on multiple devices. The above program is recorded on a non-transitory recording medium such as a computer-readable ROM, optical disk, or hard disk drive. The above program may be stored in the recording medium in advance, or may be supplied to the recording medium via a wide area communication network including the Internet.

One aspect of the present disclosure is not limited to the above-described embodiment and modified examples, and the above configurations can be replaced with substantially the same configurations, configurations that achieve the same effects, or configurations that can achieve the same purpose.

100 Information processing device, 101 Imaging unit, 201 Memory unit, 202 Control unit, 203 Output unit, 301 Image acquisition unit, 302 Region of interest extraction unit, 303 Pulse wave calculation unit, 304 Similarity calculation unit, 305 Pulse wave synthesis unit, 306 Output control unit, 401 Frame image, 402 Region of interest, 403a Pixel, 403b Pixel, 501 Pixel, 701 Region of interest, 701a Pixel, 701b Pixel, 702 Region of interest, 702a Pixel, 702b Pixel, 703 Region of interest, 703a Pixel, 703b Pixel, 704 Region of interest, 704a Pixel, 704b Pixel, 711 Blood vessel, 712 Blood vessel, 901 Biometric information extraction unit, 1001 Region determination unit, 1002 Pulse wave calculation unit, 1101 region, 1102 region, 1103 region, 1104 region, 1201 pulse wave calculation unit, 1301 pixel, 1302 surrounding pixel, 1303 region, 1401 feature amount extraction unit, 1402 similarity calculation unit, 1403 pulse wave synthesis unit, 1701 part detection unit, 1702 pulse wave calculation unit, 1703 similarity calculation unit, 1704 pulse wave synthesis unit, 1901 frame image, 1902 carotid artery, 1903 blood vessel, 1904 carotid artery, 1905 blood vessel, 1906 pixel, 1907 pixel, 1908 region of interest, 1909 region of interest, 1912 pixel, 1913 pixel, 1914 pixel, 1915 pixel

Claims (9)

a pulse wave calculation unit that calculates a plurality of pulse waves corresponding to different positions on the body surface from a moving image of the body surface;
a similarity calculation unit that calculates a similarity between two pulse waves included in the plurality of pulse waves;
a pulse wave synthesis unit that adds the two pulse waves to a synthesis candidate pulse wave list when the similarity satisfies a predetermined condition, and synthesizes the pulse waves included in the synthesis candidate pulse wave list;
An information processing device comprising: The information processing device according to claim 1, characterized in that the pulse wave synthesis unit synthesizes a pulse wave from the moving image. The information processing apparatus according to claim 1 , further comprising an imaging unit configured to capture an image of the body surface and acquire the moving image. The information processing device according to claim 1 , wherein the similarity calculation section calculates the similarity based on at least one of a correlation value and a difference value of the two pulse waves. A feature extraction unit is further provided for extracting a feature indicating a feature of the shape of the pulse wave,
The information processing device according to claim 1 , wherein the similarity calculation section calculates the similarity based on a comparison result of at least two of the feature amounts. A region determining unit determines a plurality of regions including a plurality of pixels from the moving image,
The pulse wave calculation unit calculates each of the plurality of pulse waves based on a statistical value of pixel values of each of the plurality of regions.
The information processing device according to any one of claims 1 to 5 . The information processing device according to any one of claims 1 to 5, wherein the pulse wave calculation unit calculates one pulse wave corresponding to one pixel among the plurality of pulse waves based on a statistical value of a pixel value of the pixel included in the moving image and a pixel value of at least one pixel adjacent to the one pixel. The information processing device according to claim 1 , further comprising a biological information extraction unit that extracts biological information based on the pulse wave synthesized by the pulse wave synthesis unit. calculating a plurality of pulse waves corresponding to different positions on the body surface from a moving image of the body surface;
calculating a similarity between two pulse waves included in the plurality of pulse waves;
adding the two pulse waves to a synthesis candidate pulse wave list if the similarity satisfies a predetermined condition, and synthesizing the pulse waves included in the synthesis candidate pulse wave list;
A pulse wave calculation method comprising:

Images