JP7768950B2 - Arithmetic device, arithmetic method, and program - Google Patents

Description

The present invention relates to a computing device, a computing method, and a program.

Modern society is often called a stressful society, with mental illness caused by stress being a major problem. Mental illness can be caused by a variety of factors, including interpersonal relationships at work, the workplace environment, and excessive workloads, as well as lack of exercise, sleep, and an unbalanced diet. In order to prevent stress-related mental illness, there is a demand for stress assessment tools that allow for easy measurement of stress levels at home and that allow for lifelogging.

In recent years, psychological assessment questionnaires, which are subjective assessment methods, have become a common method for assessing stress. However, stress assessments using psychological assessment questionnaires have the problem that the results are easily influenced by the subject's physical condition and mood at the time. Therefore, research has been conducted into objective assessment methods that acquire physiological information from the subject, such as breathing, heart rate, saliva, and pulse wave, and then perform stress assessments based on this physiological information. Of these physiological information types, pulse waves in particular can be measured inexpensively using inexpensive equipment, and the measurement device can be implemented in wearable devices such as smartwatches. Therefore, even though pulse waves are relatively difficult to acquire, they can be easily measured. Known devices that incorporate this technology include biometric information detection devices (see, for example, Patent Document 1).

JP 2011-147746 A

The biometric information detection devices described in the above-mentioned documents can detect biometric information such as pulse waves based on scattered light obtained by flashing a sensor. For example, such biometric information detection devices could be used to detect changes in a user's biometric information while they are viewing content such as e-books on a smartphone. However, the penetration rate of wearable devices such as smartwatches equipped with such biometric information detection devices is not necessarily higher than that of smartphones. Therefore, to detect changes in a user's biometric information using such technology, the user must wear a wearable device such as a smartwatch equipped with a biometric information detection device in addition to a smartphone for viewing content such as e-books. If biometric information could be detected non-contact using a smartphone or other device, the user would not need to wear the wearable device, and it would also be possible to obtain biometric information from users who are not wearing the wearable device.

The present invention aims to provide a computing device, computing method, and program that can detect biometric information without contact.

[1] In order to solve the above problem, one aspect of the present invention includes a video information acquisition unit that acquires video information including a plurality of consecutive frame images of a person to be measured, the frame images being frame images of the person; a region identification unit that identifies the earlobe of the person from among the regions of the person captured in the frame images as a region to be measured; a statistical calculation unit that performs statistical calculations on at least two of the RGB pixel values of the plurality of consecutive frame images for the identified region; a difference calculation unit that calculates the difference between at least two of the pixel values for which statistical calculations have been performed; a calculation unit that converts information on the difference values for each time calculated by the difference calculation unit into a frequency axis; and a signal obtained by converting the information on the difference values converted by the calculation unit. and an output unit that outputs a pulse wave amplitude value and a heart rate of the person being measured based on the results of the measurement , wherein the area identification unit identifies a plurality of areas in addition to the earlobes of the person photographed in the frame image as areas to be measured, the statistical calculation unit performs statistical calculations for each of the plurality of areas identified by the area identification unit, the difference calculation unit calculates a difference for each of the plurality of areas identified by the area identification unit, the calculation unit converts each of the plurality of areas identified by the area identification unit into a frequency axis, and the output unit outputs a pulse wave amplitude value and a heart rate of the person being measured based on at least one of the plurality of areas identified by the area identification unit .

[ 2 ] In another aspect of the present invention, in the calculation device described in [ 1 ] above, a priority is assigned to each of the multiple regions identified by the region identification unit in advance, the calculation unit calculates a pulse wave amplitude value and a heart rate for each of the multiple regions identified by the region identification unit, and if the calculation result is equal to or less than a threshold, determines that the calculation result is incorrect, and the output unit outputs the result measured at the location with the highest priority among the calculation results greater than the threshold.

[ 3 ] In another aspect of the present invention, in the calculation device described in [1] or [ 2] above, the statistical calculation unit performs statistical calculations on at least R and G pixel values among the RGB pixel values of the plurality of consecutive frame images, and the difference calculation unit calculates the difference between the R and G pixel values for which the statistical calculations have been performed.

[ 4 ] In another aspect of the present invention, in the calculation device described in any one of [1] to [ 3 ] above, the statistical calculation unit calculates the average value of at least any two pixel values of RGB of a plurality of consecutive frame images for a specified region.

[ 5 ] In addition, according to one aspect of the present invention, the calculation device according to any one of [1] to [ 4 ] above further comprises an inverse Fourier calculation unit that calculates a pulse wave by performing an inverse Fourier transform based on the result of the conversion by the calculation unit, and the output unit further outputs the pulse wave calculated by the inverse Fourier calculation unit.

[ 6 ] Also, one aspect of the present invention includes a video information acquisition step of acquiring video information including a plurality of consecutive frame images in which a person to be measured is captured; a region identification step of identifying an earlobe of the person from among the regions of the person captured in the frame images as a region to be measured; a statistical calculation step of performing statistical calculation on at least two of the RGB pixel values of the plurality of consecutive frame images for the identified region; a difference calculation step of calculating the difference between at least two of the pixel values for which statistical calculation has been performed; a calculation step of converting information about the difference values for each time calculated by the difference calculation step into a frequency axis; and, based on the results converted by the calculation step, and an output step of outputting a pulse wave amplitude value and a heart rate of the person to be measured, wherein the area specifying step specifies a plurality of areas in addition to the earlobe of the person photographed in the frame image as areas to be measured, the statistical calculation step performs a statistical calculation for each of the plurality of areas specified by the area specifying step, the difference calculation step calculates a difference for each of the plurality of areas specified by the area specifying step, the calculation step converts each of the plurality of areas specified by the area specifying step into a frequency axis, and the output step outputs a pulse wave amplitude value and a heart rate of the person to be measured based on at least one of the plurality of areas specified by the area specifying step .

[ 7 ] Also, one aspect of the present invention is a method for measuring a human body by a computer, the method comprising: a video information acquisition step of acquiring video information including a plurality of consecutive frame images in which a person to be measured is captured; an area specification step of specifying an earlobe of the person from among the areas of the person captured in the frame images as an area to be measured; a statistical calculation step of performing statistical calculation on at least two pixel values from among the RGB pixel values of the plurality of consecutive frame images for the specified area; a difference calculation step of calculating the difference between at least two pixel values for which statistical calculation has been performed; a calculation step of converting information about the difference values for each time calculated by the difference calculation step into a frequency axis; and, based on the results converted by the calculation step, and an output step of outputting a pulse wave amplitude value and a heart rate value for the person to be measured, wherein the area specifying step specifies a plurality of areas in addition to the earlobes of the person photographed in the frame image as areas to be measured, the statistical calculation step performs a statistical calculation for each of the plurality of areas specified by the area specifying step, the difference calculation step calculates a difference for each of the plurality of areas specified by the area specifying step, the calculation step converts each of the plurality of areas specified by the area specifying step into a frequency axis, and the output step outputs a pulse wave amplitude value and a heart rate value for the person to be measured based on at least one of the plurality of areas specified by the area specifying step .

The present invention provides a computing device, computing method, and program that can detect biometric information without contact.

FIG. 2 is a functional configuration diagram showing an example of the functional configuration of the content information providing device according to the present embodiment. FIG. 2 is a functional configuration diagram showing an example of the functional configuration of a calculation device according to the present embodiment. 10A and 10B are diagrams for explaining an example of region specification performed by the calculation device according to the present embodiment. 10A and 10B are diagrams for explaining a method for acquiring pulse wave components by a calculation device according to the present embodiment. FIG. 10 is a diagram illustrating an example of RGB difference values according to the embodiment. 4A and 4B are diagrams showing an example of pulse wave amplitude values and heart rates obtained by the calculation device according to the present embodiment. FIG. 10 is a diagram showing a comparison result between the change over time of the pulse wave amplitude value obtained based on the RG difference value according to the present embodiment and the change over time of the pulse wave amplitude value obtained by a photoplethysmograph. FIG. 10 is a diagram showing the results of a comparison between the heart rate obtained by the calculation device according to the present embodiment and the heart rate obtained by a photoplethysmography system. 1 is a flowchart showing the flow of a series of processes performed by a calculation method according to the present embodiment. FIG. 10 is a functional configuration diagram showing a modified example of the functional configuration of the arithmetic device according to the present embodiment. FIG. 2 is a block diagram showing an example of the internal configuration of a calculation device according to the present embodiment.

[Embodiment]
Below, preferred embodiments of a computing device, a computing method, and a program according to the present invention will be described in detail with reference to the accompanying drawings. Note that the present invention is not limited to these embodiments and includes various modifications and improvements. In other words, the components described below include those that would be easily conceivable to a person skilled in the art or that are substantially identical, and the components described below can be combined as appropriate. Furthermore, various omissions, substitutions, or modifications of the components can be made without departing from the spirit of the present invention. Furthermore, in the drawings, the scale and number of components may differ from the scale and number of the actual components to make each configuration easier to understand.

The arithmetic device according to this embodiment detects biometric information of user U in a non-contact manner. The biometric information detected from user U can be used for various analyses. The arithmetic device is not limited to use for browsing content, and can be used in a wide range of applications. For example, the arithmetic device may be used for tests such as psychological evaluations, or for stress assessments. Furthermore, analyses may be performed using the results of these evaluations, and the analysis results may be used when providing various services. Furthermore, the arithmetic device may be used in place of a pulse wave measuring device in situations where a pulse wave measuring device would conventionally be used.

As an example, the following describes a case where the arithmetic device is used to select content to be presented to a user when the user browses content, adjust the method of presenting the content, etc. In this case, the arithmetic device is implemented in, for example, the content information providing device 1.

FIG. 1 is a functional configuration diagram showing an example of the functional configuration of a content information providing device according to this embodiment. First, the functional configuration of the content information providing device 1 according to this embodiment will be described with reference to this diagram. The content information providing device 1 is a device used by a user U. The user U browses content by operating the content information providing device 1. The content provided by the content information providing device 1 may be e-books such as manga and novels, videos such as movies, images such as photographs, music, services provided by SNS (Social Networking Service), etc. The content provided by the content information providing device 1 may be content stored on the device itself, or may be web content stored on a server device (not shown). The content information providing device 1 may also be a general-purpose device such as a smartphone, tablet, or smart glasses.

The content information providing device 1 is configured to include at least an imaging unit 2, a calculation unit 3, a control unit 4, and a display unit 5 as functional components. Each of these functional units may be implemented using, for example, electronic circuits. Each functional unit may also include internal storage means such as semiconductor memory or a magnetic hard disk drive, as necessary. Each function may also be implemented using software and a computer with a CPU (Central Processing Unit). All or part of each functional unit may also be implemented using hardware such as an ASIC (Application Specific Integrated Circuit), PLD (Programmable Logic Device), or FPGA (Field-Programmable Gate Array). All or part of each functional unit may also be implemented using a combination of software and hardware.

The imaging unit 2 acquires video information of the user U. This video information includes multiple consecutive frame images. The frame images may be images captured by a CCD (Charge Coupled Devices) camera using a CCD image sensor for video capture, or may be images captured by a CMOS (Complementary Metal Oxide Semiconductor) camera using a CMOS image sensor. The video information captured by the imaging unit 2 is preferably a color video including RGB pixel information. Furthermore, the video information captured by the imaging unit 2 is preferably an image of the user U's skin, and more preferably an image of the user U's face.

The calculation unit 3 analyzes the video information acquired by the imaging unit 2. As a result of analyzing the video information, the calculation unit 3 detects biometric information of the user U. In other words, the calculation unit 3 can be said to detect biometric information of the user U based on video information obtained without contacting the user U (non-contact). The biometric information detected by the calculation unit 3 may include information indicating the stress state of the user U. In the following description, the calculation unit 3 may sometimes be referred to as a calculation device.

The control unit 4 acquires biometric information of the user U from the calculation unit 3. The control unit 4 controls the content to be presented to the user U based on the acquired biometric information of the user U. The control unit 4 can also provide appropriate content depending on the stress level of the user U. For example, the content to be presented to the user U may be determined in advance based on the user U's operation, and the control unit 4 may adjust the content to be presented to the user U based on the user U's biometric information. This adjustment may include adjusting the timing of presenting the content, screen settings, etc. Note that the control unit 4 may select content to be presented to the user U based on the user U's biometric information after the presentation of the content desired by the user U has finished, and present the selected content to the user U.

The display unit 5 presents content to the user U in accordance with the control of the control unit 4. The content presented by the display unit 5 may include image information, video information, sound information, etc. The display unit 5 may be, for example, a liquid crystal display, an organic EL (electroluminescence) display, etc.

Figure 2 is a functional configuration diagram showing an example of the functional configuration of a calculation device according to this embodiment. An example of the functional configuration of the calculation device will be described with reference to this diagram. The calculation device is the calculation unit 3 described above. The calculation unit 3 is configured to include at least an image information acquisition unit 31, an area identification unit 32, a statistical calculation unit 33, a difference calculation unit 34, a Fourier calculation unit 35, and an output unit 36. Each of these functional units may be implemented using, for example, electronic circuits. Each functional unit may also include internal storage means such as semiconductor memory or a magnetic hard disk drive, as necessary. Each function may also be implemented using software and a computer with a CPU. All or part of each functional unit may also be implemented using hardware such as an ASIC, PLD, or FPGA. All or part of each functional unit may also be implemented using a combination of software and hardware.

The video information acquisition unit 31 acquires video information from the imaging unit 2. The video information includes a series of frame images. Each of the frame images captures an image of a person whose biometric information is to be measured. The person whose biometric information is to be measured may be a person to whom content information is provided by the content information providing device 1.

The area identification unit 32 identifies the area to be measured from the area of the person depicted in the frame image included in the video information acquired by the video information acquisition unit 31. Here, the area to be measured is the skin of the person depicted in the frame image. Preferably, the area to be measured may be the face of the person depicted in the frame image. In other words, the area identification unit 32 identifies part of the face of the person depicted in the frame image as the area to be measured.

Note that the processing according to this embodiment does not need to be performed on all frame images included in the frame images contained in the video information acquired by the video information acquisition unit 31. The processing according to this embodiment only needs to be performed on some of the frame images included in the video information. For example, if the frame rate of the video information is 60 FPS (Frames Per Second), processing may be performed on each frame (i.e., at a granularity of 30 FPS).

Figure 3 is a diagram illustrating an example of region identification performed by the calculation device according to this embodiment. Here, an example of a region identified by the region identification unit 32 will be described with reference to the figure. The figure shows a frame image FI included in the video information acquired by the video information acquisition unit 31. First, the region identification unit 32 detects a face portion FP from the frame image FI. Next, the region identification unit 32 identifies a predetermined region from the face portion FP.

Here, it is preferable that the region identification unit 32 identify, as the region to be measured, a region of the person's face captured in the frame image FI that has less fat than the surrounding area. The region that has less fat than the surrounding area may be predetermined, and may be the cheek area, nose area, etc. In the example shown, the region identification unit 32 identifies region AR1, which is the nose area, region AR2, which is the right cheek area, and region AR3, which is the left cheek area, as the regions to be measured. Note that the region identification unit 32 may identify the forehead area as the region to be measured, but the forehead may be hidden by bangs, a hat, etc. depending on the user. Therefore, it is preferable that the region identification unit 32 identify a region that can be measured regardless of the user.

Note that depending on the user's appearance, what the user is wearing, etc., the region identification unit 32 may not be able to identify the predetermined region. Therefore, the region identification unit 32 may be predetermined to identify multiple regions. If multiple regions are predetermined to be identified, subsequent processing may be performed for each of the multiple regions, or processing may be performed for one of the regions based on a predetermined priority for each region (each body part). Returning to Figure 2, we will continue to explain the functional configuration of the calculation unit 3.

The statistical calculation unit 33 acquires information about the region identified by the region identification unit 32. The statistical calculation unit 33 preferably acquires pixel information about the region identified by the region identification unit 32. The pixel information may include independent pixel values for R (red), G (green), and B (blue). Furthermore, the pixel information does not need to include pixel information for all RGB; preferably, it only needs to include pixel information for two of the RGB colors (R and G, G and B, or R and B). The statistical calculation unit 33 performs statistical calculations on at least two of the RGB pixel values (e.g., R and G) for the identified region. The statistical calculation may, for example, involve calculating the average value of multiple pixel values included in the identified region. Note that the statistical calculation performed by the statistical calculation unit 33 is not limited to calculating an average value, and may also involve calculating statistical values such as the mode, maximum value, or minimum value.

The difference calculation unit 34 calculates the difference between at least two pixel values for which statistical calculations, such as calculation of an average value, have been performed by the statistical calculation unit 33. For example, if the statistical calculation unit 33 calculates the average R value and the average G value for pixels included in an area identified by the area identification unit 32, the difference calculation unit 34 calculates the difference between the average R value and the average G value. Note that the area identification unit 32 may calculate the difference between the average G value and the average B value, or the difference between the average R value and the average B value.

Figure 4 is a diagram illustrating a method for acquiring pulse wave components using a computing device according to this embodiment. As shown in the figure, the outermost layer of human skin is covered by the epidermis. Human skin has a three-layer structure: the epidermis, dermis, and subcutaneous tissue, with arteries and veins running through the subcutaneous tissue. Capillaries also run through the area close to the epidermis. Here, the R, G, and B components of light reflected by human skin are reflected to different depths (or one could say that they reach different depths subcutaneously). Specifically, the B component is reflected by the epidermis, which is the shallowest layer. The G component is reflected by a layer deeper than the epidermis where capillaries and arteries run. The R component is reflected by a layer deeper than the epidermis where arteries and veins run.

This embodiment focuses on these characteristics and calculates the difference between any two of the RGB components to extract blood components from image information and further obtain pulse wave components. As described above, by determining areas with less fat than the surrounding area, rather than areas covered with fat, it is possible to extract blood components more accurately. Note that some users may wear makeup on their cheeks or nose. When makeup is worn, it may be difficult to extract blood components from image information. Therefore, the region identification unit 32 may identify areas where makeup is normally not applied, such as the earlobes.

Figure 5 is a diagram showing an example of RGB difference values according to this embodiment. In the illustrated example, the horizontal axis represents time and the vertical axis represents the RGB difference values, showing changes in the difference values over time. Specifically, waveform W1 shows changes in the difference value between R and G. Waveform W2 shows changes in the difference value between R and B. Waveform W3 shows changes in the difference value between G and B. In this way, the difference values change over time (as the subject's stress state changes). Returning to Figure 2, we will continue to explain the functional configuration of the calculation unit 3.

The Fourier calculation unit 35 converts information about the time-dependent difference values calculated by the difference calculation unit 34 into the frequency axis. Specifically, the Fourier calculation unit 35 may perform an overlapping FFT (Fast Fourier Transformation) every second to convert into the frequency axis. In the following description, the Fourier calculation unit 35 may be simply referred to as the calculation unit. The calculation unit may also perform processing to convert the time-dependent difference values into the frequency axis using a known method other than Fourier analysis.

The output unit 36 outputs the pulse wave amplitude value and heart rate of the person being measured based on the results of the conversion performed by the Fourier calculation unit 35.

At least some of the functions of the calculation unit 3 may be configured using a neural network. For example, the above-mentioned calculation unit may be replaced with a neural network. In this case, the calculation unit may be a machine learning model that has been trained in advance to infer the pulse wave amplitude value (or heart rate) from the difference value using a combination of the difference value and the pulse wave amplitude value (or heart rate) as training data. Furthermore, the functions inferred by the machine learning model are not limited to those of the calculation unit, and may further include, for example, at least one of the functions of the difference calculation unit 34, statistical calculation unit 33, and region identification unit 32.

Figure 6 shows an example of pulse wave amplitude and heart rate obtained by the calculation device according to this embodiment. The figure shows the results of an overlap FFT performed every second by the Fourier calculation unit 35. For each waveform shown, the horizontal axis represents frequency, and the vertical axis represents amplitude. The figure shows a spectrum ranging from 0.7 Hz (Hertz) to 4.0 Hz. As shown, the calculation device according to this embodiment can obtain the pulse wave amplitude and heart rate from the peak values between 0.7 Hz and 4.0 Hz. The maximum value of the spectrum is the pulse wave amplitude, and the heart rate [bpm (beats per minute)] is obtained by multiplying the frequency at which the pulse wave amplitude is measured by 60.

Note that if makeup is applied to the area identified by the area identification unit 32, the pulse wave amplitude and heart rate may not be detected accurately from the image information. Therefore, the calculation unit 3 may perform processing based on the results obtained from each of the multiple areas. Specifically, the area identification unit 32 identifies multiple areas, such as the nose, cheeks, and earlobes, using a predetermined algorithm. The multiple areas are assigned a predetermined priority. The calculation unit 3 calculates the pulse wave amplitude and heart rate for each area, and if the calculation result is below a threshold, it determines that the calculation result is incorrect and excludes it from the list of output results. The calculation unit 3 may output the measurement result for the highest-ranked area among the results for which the calculation result is greater than the threshold.

Figure 7 shows a comparison of the time-varying change in pulse wave amplitude obtained based on the RG difference value according to this embodiment with the time-varying change in pulse wave amplitude obtained using a photoplethysmograph. The horizontal axis of the figure represents time, and the vertical axis represents standardized pulse wave amplitude. To obtain the results shown in this figure, a photoplethysmograph (1 kHz) was attached to the subject's earlobe. At the same time, an image of the subject's face was captured using the content information providing device 1 according to this embodiment, and the pulse wave amplitude was measured using both the photoplethysmograph and a calculation device included in the content information providing device 1. Waveform W21 represents the pulse wave amplitude measured using the photoplethysmograph, and waveform W22 represents the pulse wave amplitude measured using the calculation device included in the content information providing device 1 according to this embodiment.

As is clear from the figure, when the pulse wave amplitude value increases over time in the measurement results of the photoplethysmograph, the pulse wave amplitude value also increases in the measurement results of the calculation device. In other words, the calculation device according to this embodiment can obtain measurement results similar to those of a contact-type photoplethysmograph without contact.

The pulse wave amplitude value can also be standardized by first subtracting the average value of the pulse wave amplitude value from the pulse wave amplitude value data, and then dividing by the standard deviation. For example, if the pulse wave amplitude value data is X, μ is the average value of X, and σ is the standard deviation, the standardized pulse wave amplitude value z can be expressed by the following equation (1):

z=(X-μ)/σ...(1)

Figure 8 shows the results of comparing the heart rate obtained by the calculation device according to this embodiment with the heart rate obtained by a photoplethysmography system. The figure also shows the error obtained by comparing the heart rate obtained by the calculation device provided in the content information providing device 1 according to this embodiment with the heart rate obtained by a photoplethysmograph. The figure also shows the results obtained by the method according to this embodiment and the results obtained by a conventional method. The method according to this embodiment shows the results obtained based on the difference value between R and G, the difference value between R and B, and the difference value between G and B. The heart rate shown in the figure is in units of [bpm].

The conventional technology involves acquiring pulse waves by performing independent component analysis (ICA) based on images captured by a webcam. However, this method has the drawback of sometimes being unable to separate pulse wave components.

The calculation method according to this embodiment was found to produce smaller errors than conventional techniques when based on the difference between R and G and when based on the difference between G and B. In particular, when based on the difference between R and G, the error was approximately half or less compared to conventional techniques. Therefore, this embodiment can be said to be capable of accurately extracting reflected components that contain a large amount of pulse wave information. In other words, the calculation method according to this embodiment is capable of obtaining pulse wave information with higher accuracy than conventional techniques.

Note that, according to this embodiment, it is also possible to use a combination of the difference value between R and G, the difference value between R and B, and the difference value between G and B. In this case, the average value of each difference value may be used, or weighting may be performed based on accuracy, etc. The weighting may vary based on the region identified by the region identification unit 32, and the combination of each difference value may differ for each region.

Figure 9 is a flowchart showing the flow of a series of processes performed by the calculation method according to this embodiment. With reference to this figure, we will explain an example of the calculation method performed by the calculation device according to this embodiment.

(Step S11) First, video information is acquired by the video information acquisition unit 31. The video information is acquired, for example, from the imaging unit 2. The video information includes multiple frame images. The frame rate of the video information may be, for example, 30 [FPS] or 30 [FPS], etc.

(Step S12) Next, the area to be measured is identified by the area identification unit 32 for each frame image included in the video information. The area to be measured may be the nose area, right cheek area, left cheek area, etc. It is preferable that the area to be measured is determined in advance. Furthermore, depending on the imaging angle, the user's appearance, etc., the predetermined measurement area may not necessarily be displayed in the frame image, and analysis of the area may not be easy. Therefore, it is preferable that there are multiple areas to be measured. In this case, subsequent processing may be performed based on one of the multiple areas, or a combination of multiple areas.

(Step S13) Next, the statistical calculation unit 33 performs statistical calculations on the RGB pixel values. The statistical calculations performed by the statistical calculation unit 33 may, for example, calculate the average value of pixel values within the area identified by the area identification unit 32. The area on which the statistical calculation unit 33 performs calculations may be multiple. Furthermore, the statistical calculation unit 33 does not necessarily need to perform statistical calculations on all RGB pixel values; it is sufficient to perform statistical calculations on at least any two of the RGB pixel values.

(Step S14) Next, the difference calculation unit 34 calculates a difference value between any two statistically calculated values (e.g., average values) calculated by the statistical calculation unit 33. The difference value calculated by the difference calculation unit 34 may be, for example, the difference value between R and G, the difference value between R and B, or the difference value between G and B. The difference value calculated by the difference calculation unit 34 may also be a combination of these difference values, or a combination of these difference values weighted in a predetermined manner.

(Step S15) Next, the Fourier calculation unit 35 converts the difference value calculated by the difference calculation unit 34 into a frequency axis. Based on the result of the frequency conversion by the Fourier calculation unit 35, the pulse wave amplitude value and heart rate can be calculated.

(Step S16) Finally, the output unit 36 outputs the pulse wave amplitude value and heart rate obtained in step S15. The output destination of the output unit 36 may be, for example, the control unit 4 or a storage device (not shown).

Figure 10 is a functional configuration diagram showing a modified functional configuration of the calculation device according to this embodiment. With reference to this diagram, calculation unit 3A, which is a modified version of calculation unit 3, will be described. In the description of calculation unit 3A, components similar to those in calculation unit 3 will be assigned the same reference numerals and their description may be omitted. Calculation unit 3A differs from calculation unit 3 in that it further includes an inverse Fourier calculation unit 37.

The inverse Fourier calculation unit 37 performs an inverse Fourier transform based on the results of the transformation performed by the Fourier calculation unit 35. The inverse Fourier transform performed by the inverse Fourier calculation unit 37 enables the pulse wave to be obtained. The output unit 36 further outputs the pulse wave calculated by the inverse Fourier calculation unit 37 instead of, or in addition to, the pulse wave amplitude value and heart rate calculated by the Fourier calculation unit 35.

Figure 11 is a block diagram showing an example of the internal configuration of a computing device according to this embodiment. An example of the internal configuration of the computing device will be described with reference to this figure. At least some of the functions of the computing device can be implemented using a computer. As shown in the figure, the computer includes a central processing unit 901, RAM 902, input/output port 903, input/output devices 904 and 905, etc., and bus 906. The computer itself can be implemented using existing technology. The central processing unit 901 executes instructions contained in a program read from RAM 902, etc. In accordance with each instruction, the central processing unit 901 writes data to RAM 902, reads data from RAM 902, and performs arithmetic and logical operations. RAM 902 stores data and programs. Each element included in RAM 902 has an address and can be accessed using the address. RAM is an abbreviation for "random access memory." The input/output port 903 is a port through which the central processing unit 901 exchanges data with external input/output devices, etc. Input/output devices 904 and 905 are input/output devices. The input/output devices 904 and 905 exchange data with the central processing unit 901 via the input/output port 903. The bus 906 is a common communication path used within the computer. For example, the central processing unit 901 reads and writes data from the RAM 902 via the bus 906. Also, for example, the central processing unit 901 accesses the input/output port via the bus 906. Furthermore, all or part of the functional units provided in the arithmetic device according to this embodiment may be implemented using hardware such as an ASIC, PLD, or FPGA. Furthermore, all or part of the functional units may be implemented using a combination of software and hardware. Note that the content information providing device 1 may also have an internal configuration similar to that shown in the figure.

[Summary of this embodiment]
According to the embodiment described above, the computing device according to this embodiment includes the video information acquisition unit 31, which acquires video information including a plurality of consecutive frame images of a person to be measured; the region identification unit 32, which identifies the region of the person captured in the frame images as the measurement target; the statistical calculation unit 33, which performs statistical calculations on at least two of the RGB pixel values of the consecutive frame images for the identified region; the difference calculation unit 34, which calculates the difference between at least two of the pixel values subjected to statistical calculation; the Fourier calculation unit 35, which converts information on the time-dependent difference values calculated by the difference calculation unit 34 into a frequency axis; and the output unit 36, which outputs the pulse wave amplitude and heart rate of the person to be measured based on the results of the conversion by the Fourier calculation unit 35. In other words, the computing device according to this embodiment detects biometric information based on image information. Therefore, according to this embodiment, biometric information can be detected without contact.

Furthermore, according to this embodiment, the area identification unit 32 identifies a part of the face of a person captured in the frame image as the area to be measured. Here, users of this embodiment often use content information providing devices 1 equipped with an imaging unit 2 that captures images of the user's face, such as smartphones, tablet devices, or smart glasses. Therefore, according to this embodiment, by using the imaging unit 2 equipped in the content information providing device 1, biometric information can be detected without the need for new hardware used for biometric information detection. Therefore, according to this embodiment, biometric information can be detected at low cost and without imposing a burden on the user.

Furthermore, according to this embodiment, the region identification unit 32 identifies, within a portion of the face of a person captured in a frame image, an area that has less fat than the surrounding area as the area to be measured. Because the region identification unit 32 identifies, as the area to be measured, an area that has less fat than the surrounding area, the calculation device according to this embodiment can accurately obtain changes in blood vessels present in the dermis and subcutaneous tissue.

Furthermore, according to this embodiment, the statistical calculation unit 33 performs statistical calculations on at least the R and G pixel values of the RGB pixel values of multiple consecutive frame images, and the difference calculation unit 34 calculates the difference between the R and G pixel values for which statistical calculations have been performed. That is, according to this embodiment, biometric information is detected based on the R and G difference values. Here, referring to FIG. 8, of the RGB difference values, the difference value between R and G had the smallest error. Therefore, according to this embodiment, biometric information can be detected with greater accuracy.

Furthermore, according to this embodiment, the statistical calculation unit 33 calculates the average value of at least two of the RGB pixel values of multiple consecutive frame images for the region identified by the region identification unit 32. In other words, the statistical calculation performed by the statistical calculation unit 33 is the calculation of an average value. Therefore, according to this embodiment, biometric information can be easily detected.

In addition, the calculation device according to this embodiment further includes an inverse Fourier calculation unit 37, which performs an inverse Fourier transform based on the results of the transformation performed by the Fourier calculation unit 35 to calculate the pulse wave. The output unit 36 further outputs the pulse wave calculated by the inverse Fourier calculation unit 37. Therefore, according to this embodiment, the pulse wave can be detected without contact.

Note that all or part of the functions of each unit of the computing device according to the above-described embodiments may be realized by recording a program for realizing these functions on a computer-readable recording medium, and then loading and executing the program recorded on the recording medium into a computer system. Note that the term "computer system" here includes hardware such as the OS and peripheral devices.

In addition, "computer-readable recording medium" refers to portable media such as flexible disks, optical magnetic disks, ROMs, and CD-ROMs, as well as storage units such as hard disks built into computer systems. Furthermore, "computer-readable recording medium" may also include devices that dynamically store programs for a short period of time, such as communication lines when transmitting programs over networks like the Internet or communication lines like telephone lines, or devices that store programs for a fixed period of time, such as volatile memory within computer systems that serve as servers or clients in such cases. Furthermore, the above-mentioned programs may be those that implement some of the functions described above, or may be those that can implement the functions described above in combination with programs already stored in the computer system.

The above describes the form for carrying out the present invention using an embodiment, but the present invention is in no way limited to such an embodiment, and various modifications and substitutions can be made within the scope that does not deviate from the spirit of the present invention.

1...Content information providing device, 2...Image capture unit, 3...Calculation unit, 4...Control unit, 5...Display unit, U...User, 31...Video information acquisition unit, 32...Area identification unit, 33...Statistical calculation unit, 34...Difference calculation unit, 35...Fourier calculation unit, 36...Output unit, 37...Inverse Fourier calculation unit

Claims (7)

a video information acquisition unit that acquires video information including a plurality of consecutive frame images of a person being measured;
an area specifying unit that specifies an earlobe of a person as a measurement target area from among areas of the person captured in the frame image;
a statistical calculation unit that performs statistical calculations on at least two pixel values of RGB of the plurality of consecutive frame images for the identified region;
a difference calculation unit that calculates a difference between at least two pixel values for which statistical calculation has been performed;
a calculation unit for converting information about the difference value for each time calculated by the difference calculation unit into information on a frequency axis;
an output unit that outputs a pulse wave amplitude value and a heart rate of the person being measured based on the results of the conversion by the calculation unit ,
the region specifying unit specifies a plurality of regions as measurement target regions in addition to an earlobe of the person photographed in the frame image;
the statistical calculation unit performs statistical calculations for each of the plurality of regions identified by the region identification unit;
the difference calculation unit calculates a difference for each of the plurality of regions identified by the region identification unit;
the calculation unit converts each of the plurality of regions identified by the region identification unit into a frequency axis;
the output unit outputs a pulse wave amplitude value and a heart rate of the person to be measured based on at least one of the plurality of regions identified by the region identification unit.
Computing device. a priority order is set in advance for each of the plurality of regions identified by the region identification unit;
the calculation unit calculates a pulse wave amplitude value and a heart rate for each of the plurality of regions identified by the region identification unit, and determines that the calculation result is incorrect when the calculation result is equal to or less than a threshold value;
the output unit outputs the measurement result at the location with the highest priority among the results for which the calculation result is greater than the threshold.
The computing device of claim 1 . the statistical calculation unit performs statistical calculations on at least R and G pixel values among the R, G, and B pixel values of the plurality of consecutive frame images;
The arithmetic device according to claim 1 , wherein the difference calculation unit calculates the difference between the R and G pixel values on which the statistical calculation has been performed. The computing device according to claim 1 , wherein the statistical computing section computes an average value of at least any two pixel values of each of RGB pixel values of the plurality of consecutive frame images for the identified region. An inverse Fourier calculation unit is further provided which calculates a pulse wave by performing an inverse Fourier transform based on the result of the conversion by the calculation unit,
The calculation device according to claim 1 , wherein the output unit further outputs the pulse wave calculated by the inverse Fourier calculation unit. a video information acquisition step of acquiring video information including a plurality of consecutive frame images of a person to be measured;
an area specifying step of specifying an earlobe of the person as a measurement target area from among areas of the person captured in the frame image;
a statistical calculation step of performing statistical calculations on at least two pixel values of RGB of the plurality of consecutive frame images for the identified region;
a difference calculation step of calculating a difference between at least two pixel values for which statistical calculation has been performed;
a calculation step of converting information about the difference values for each time calculated in the difference calculation step into a frequency axis;
and an output step of outputting the pulse wave amplitude value and the heart rate of the person being measured based on the results converted by the calculation step ,
the region specifying step specifies a plurality of regions as measurement target regions in addition to the earlobe of the person photographed in the frame image;
the statistical calculation step performs a statistical calculation for each of the plurality of regions identified by the region identification step;
the difference calculation step calculates a difference for each of the plurality of regions identified by the region identification step;
The calculation step converts each of the plurality of regions identified by the region identification step into a frequency axis,
The output step outputs a pulse wave amplitude value and a heart rate of the person to be measured based on at least one of the plurality of regions identified by the region identification step.
Calculation method. On the computer,
a video information acquisition step of acquiring video information including a plurality of consecutive frame images of a person to be measured;
an area specifying step of specifying an earlobe of the person as an area to be measured from an area of the person captured in the frame image;
a statistical calculation step of performing statistical calculations on at least two pixel values of RGB of the plurality of consecutive frame images for the identified region;
a difference calculation step of calculating a difference between at least two pixel values for which statistical calculation has been performed;
a calculation step of converting information about the difference values for each time calculated in the difference calculation step into a frequency axis;
an output step of outputting the pulse wave amplitude value and the heart rate of the person being measured based on the results of the conversion in the calculation step ;
the region specifying step specifies a plurality of regions as regions to be measured in addition to an earlobe of the person photographed in the frame image;
the statistical calculation step performs a statistical calculation for each of the plurality of regions identified by the region identification step,
the difference calculation step calculates a difference for each of the plurality of regions identified by the region identification step;
The calculation step converts each of the plurality of regions identified by the region identification step into a frequency axis,
The output step outputs a pulse wave amplitude value and a heart rate of the person to be measured based on at least one of the plurality of regions identified in the region identification step.
program.