JP7641722B2 - How to evaluate your skin - Google Patents

Description

The present invention relates to a method for evaluating skin.

In recent years, advances in measurement techniques have made it possible to measure the internal structure of the skin non-invasively.
For example, in Non-Patent Document 1, the structure of blood vessels is revealed by a tissue transparency technique.

Also known are methods for diagnosing skin conditions based on non-invasively measured internal structures of the skin.
For example, Patent Document 1 discloses a method for diagnosing skin burns, gangrene, and bedsores by non-invasively measuring the internal structure of the skin using photoacoustic waves.

Shibata T, Kajiya K, Sato K, Yoon J, Kang HY. 3D microvascular analysis reveals irregularly branching blood vessels in the hyperpigmented skin of solar lentigo. Pigment cell & melanoma research. 2018;31(6):725 7.

Patent No. 5860534

Incidentally, it is known that the internal morphological and physiological characteristics of the skin, such as the thickness and amount of collagen fibers and the amount of water retained inside the skin, i.e., the internal structure of the skin, affect the appearance of the skin (spots, wrinkles, dullness, etc.).
However, analysis of the internal structure of the skin using tissue transparency techniques requires the removal of the skin, which means that blood and oxygen are lost, making it impossible to measure the viability of the tissue. In addition, it is difficult to ensure a sufficient number of samples for analysis.
In other words, the above-mentioned techniques are limited in their ability to measure the internal structure of skin in a living body, and skin evaluations based on indicators obtained by such techniques do not necessarily reflect the condition of the living body.

In addition, a method for diagnosing skin conditions (presence or absence of burns, gangrene, bedsores, etc.) based on non-invasively measured internal structure of a subject's skin, as described in Patent Document 1, is known, but this method has the problem that it requires expensive equipment to measure the internal structure of the skin.

In view of the above situation, the present invention aims to provide a new skin evaluation method that can output an estimate of the internal structure of the skin of a living body by utilizing information that can be easily obtained from the living body without using expensive equipment.

The present invention, which solves the above problem, is a skin evaluation method that uses multiple pairs of feature amounts obtained from data showing the internal structure of the subject's skin and skin images obtained by photographing the biological surface of the same subject as learning data, inputs the skin images into a model that has learned the relationship between the feature amounts and the skin images, and outputs estimated values of the feature amounts.

In a preferred embodiment of the present invention, the data indicating the internal structure is non-invasively obtained.

In a preferred embodiment of the present invention, the model is a convolutional neural network model that is created by performing deep learning using the skin image as input data and the feature amount as a response variable.

In a preferred embodiment of the present invention, the method includes estimating the contribution of each region of the input skin image to the estimation result by the model by analyzing the model from the output side to the input side based on the input skin image.

In a preferred embodiment of the present invention, the contribution of each region of the skin image to the estimation result is displayed superimposed on the skin image.

In a preferred embodiment of the present invention, the feature is at least one of a feature relating to the blood vessel structure of the skin, a feature relating to the oxygen saturation of the skin, and a feature relating to the melanin structure of the skin.

In a preferred embodiment of the present invention, the feature quantity relating to the vascular structure is a feature quantity relating to the amount of microvascular structure in the upper part of the skin.

In a preferred embodiment of the present invention, the features related to the vascular structure are selected from the average thickness of the blood vessels, the number of blood vessel branches, and the volume of the blood vessels in the upper part of the skin.

In a preferred embodiment of the present invention, the feature related to oxygen saturation is vascular oxygen saturation.

In a preferred embodiment of the present invention, the characteristic quantity relating to the melanin structure is the volume of the tubular streak structure of melanin.

In a preferred embodiment of the present invention, the data indicating the internal structure is measurement data obtained by a photoacoustic method or data generated based on the measurement data.

In a preferred embodiment of the present invention, the method includes estimating the oxygen saturation of the subject's skin based on features related to the blood vessel structure of the skin.

In a preferred embodiment of the present invention, the method includes estimating the oxygen saturation of the skin based on features related to the melanin structure of the skin.

In a preferred embodiment of the present invention, the method includes estimating features related to the blood vessel structure of the skin and/or features related to the melanin structure of the skin based on the oxygen saturation of the anterior skin.

In a preferred embodiment of the present invention, the method further includes estimating an index of the subject's skin aging state based on the estimated value output by the model.

In a preferred embodiment of the present invention, the aging state is evaluated by inputting the estimated value into a correlation model created by providing multiple pairs of the feature amount and age, and calculating an estimated age.

The present invention also provides a skin evaluation program that uses multiple pairs of feature amounts obtained from non-invasively obtained data showing the internal structure of a subject's skin and skin images obtained by photographing the biological surface of the same subject as learning data, inputs the skin images into a model that has learned the relationship between the feature amounts and the skin images, and causes a computer to function to output estimated values of the feature amounts.

The present invention provides a skin evaluation method that easily outputs an estimate of features related to the internal structure of the skin based on a skin image. In particular, when measurement data obtained by a non-invasive method such as a photoacoustic method is used as learning data, estimation that reflects the results of learning based on the internal structure of living skin becomes possible.

1 shows the results of deep learning training to discriminate photoacoustic measurements using facial skin image data as input, (A) training and validation accuracy, (B) loss and data validation. (A) Examples of analysis of vascular structures of subjects with thinner (top) and thicker (bottom) average vascular structures, showing input images (left), Grad-CAM images (center), and Guided Grad-CAM images (right), respectively. (B) Examples of analysis of oxygen saturation of subjects with high oxygen saturation (top) and low oxygen saturation (bottom), showing input images (left), Grad-CAM images (center), and Guided Grad-CAM images (right), respectively. These images show an input image (left), a Grad-CAM image (center), and a Guided Grad-CAM image (right) as examples of analysis of melanin tubular pattern structures of subjects with small (top row) and large (bottom row) melanin tubular pattern structures. (A) A merged image of an image obtained from a 575 nm photoacoustic signal and an image obtained from a 650 nm ultrasound signal. (B) A diagram showing the vascular structure in the upper part of the skin of a subject in his twenties, generated from measurement data obtained by the photoacoustic method. (C) A diagram showing the vascular structure in the upper part of the skin of a subject in his sixties, generated from measurement data obtained by the photoacoustic method. (A) Measured image data, annotated image data, and predicted image data of vascular structures by deep learning that were used as training data in deep learning. (B) Measured image data, annotated image data, and predicted image data of vascular structures by deep learning that were not used as training data in deep learning. (C) A representative example of three-dimensional data generated by combining predicted image data (xy plane) in the z-axis direction. FIG. 1 is a diagram showing the relationship between subject age and the average thickness of the vascular structure in each layer of the skin. 1A and 1B are diagrams showing the spatial distribution of oxygen saturation in the measurement area of a subject in his 20s and 60s, respectively. FIG. 1 is a diagram showing the relationship between the age of a subject and the oxygen saturation in each layer of the skin. FIG. 1 shows the relationship between the average thickness of the vascular structure in each layer of the skin and the oxygen saturation in the corresponding layer. (A) A diagram showing the distribution of melanin structures in the measurement area of a subject in his 20s (B) A diagram showing the distribution of melanin structures in the measurement area of a subject in his 60s (A) A diagram showing the distribution of melanin structures in the measurement area of a subject in his 20s (B). (A) Measured image data, annotated image data, and predicted image data of melanin structure by deep learning, which were used as training data in deep learning. (B) Measured image data, annotated image data, and predicted image data of melanin structure by deep learning, which were not used as training data in deep learning. (C) A representative example of three-dimensional data generated by combining predicted image data (x-y plane) in the z-axis direction. FIG. 1 shows the relationship between subject age and melanin tubular patterning present in the upper part of the skin. FIG. 1 shows the relationship between oxygen saturation in each layer of the skin and the volume of melanin tubular structures in the corresponding layer.

In the present invention, the term "skin" refers to biological tissue including the epidermis layer, the dermis layer, and the subcutaneous fat layer, and is used when focusing particularly on the structure of the tissue.
In the present invention, the term "skin" is synonymous with "dermis" as a biological tissue, but is used without particular attention being paid to the structure of the tissue.
Additionally, "evaluation" in the present invention does not include medical diagnosis.

<1> Skin evaluation method The skin evaluation method of the present invention uses multiple pairs of feature quantities obtained from data showing the internal structure of a subject's skin and skin images obtained by photographing the biological surface of the same subject as learning data, and inputs the skin images into a model that has learned the relationship between the feature quantities and the skin images, thereby outputting estimated values of the feature quantities.
Specifically, skin images obtained by photographing the subject's facial skin in vivo (not a collected sample) are used as input data, and a convolutional neural network model is created by performing deep learning using multiple feature quantities extracted from data showing the internal structure of the same subject's skin as objective variables.
A skin image of a subject, whose features are to be estimated, is input to the model, and an estimated value of the features relating to the subject is output.

That is, in this embodiment, first, in creating a model, features including features relating to the blood vessel structure of the skin, features relating to the oxygen saturation of the skin, and features relating to the melanin structure of the skin are extracted based on data indicating the internal structure of the skin obtained by a non-invasive method such as a photoacoustic method.
Then, skin images are taken of the same subject's body, and a set of the skin image and the feature amount is provided as learning data to create a model that outputs an estimate of the feature amount based on the skin image. The data and feature amount showing the internal structure of the skin will be described later.

Here, the inventors have demonstrated, by a technique using the photoacoustic method, that there is a correlation between the average blood vessel thickness in a measurement area of the skin and the oxygen saturation in the measurement area, and between the oxygen saturation in the measurement area of the skin and the volume of the melanin streak structure in the measurement area (see Figures 9 and 13).
Therefore, by utilizing this correlation, a part of the learning data may be created by mutually estimating the features of other skin structures. For example, a feature related to the blood vessel structure of the skin may be extracted from data showing the internal structure of the skin, and the oxygen saturation of the skin may be estimated based on the feature related to the blood vessel structure of the skin, and each of the features may be used as a feature in the learning data.
Similarly, correlations may be used to estimate features of other skin structures from the estimated values of the features output by the model.

In addition, in a preferred embodiment of the present invention, the model is analyzed from the output side to the input side based on the input skin image, and the contribution of the model to the estimation result is estimated for each region of the input skin image and displayed superimposed on the skin image.
Preferred examples of the analysis method include the Grad-CAM method, the Guided Backpropagation method, and the Guided Grad-CAM method, which is a combination of these. These methods are known as methods for visualizing the area focused on by a model in deep learning. Based on an input image, the method uses class-specific gradient information flowing into the final convolution layer to perform analysis by backpropagation. The Guided Grad-CAM method is a technology that extracts features of interest in deep learning for each pixel of the input image by tracing backward from the output to the input.
This makes it possible to visualize how features related to the internal structure of the skin relate to expressions in a skin image.

<2> Acquisition of data and features showing the internal structure of the skin In this embodiment, the data showing the internal structure of the skin used to create the learning data is obtained by a non-invasive measurement method, but the present invention is not limited to this, and model learning may be performed based on invasively obtained measurement data. The data can be acquired using a known device and a known method as appropriate. As a known non-invasive measurement method, for example, a high-frequency current method, an ATR-FTIR method, a near-infrared spectroscopy, a magnetic resonance imaging method (MRI), an in vivo confocal Raman spectroscopy, an ultrasonic tomography, an optical coherence tomography, an in vivo confocal laser scanning microscope, a multiphoton microscope, an SHG microscope, a CARS microscope, an optical ultrasonic microscope, etc. can be selected.
Furthermore, so long as measurements can be made using the above-mentioned non-invasive measurement method, the type of data indicating the internal structure of the skin is not particularly limited, and examples include data regarding the vascular structure in a certain measurement area of the subject's skin, data regarding blood flow, data regarding oxygen saturation, data regarding the microspatial distribution of melanin, data regarding moisture, data regarding the viscoelasticity of the skin, data regarding specific substances, etc.
The obtained data showing the internal structure of the skin may be used as raw data, or may be subjected to general statistical processing (such as obtaining the average or standard deviation) or processing such as imaging.
In a preferred embodiment of the present invention, a photoacoustic method is employed as the non-invasive measurement method, and the data indicating the internal structure of the skin is image data acquired by the photoacoustic method.

Here, any location can be selected as the measurement region for obtaining data showing the internal structure of the skin, but it is preferable to select skin in a location that is easily exposed to light in daily life (e.g., the face or the back of the hand). It is also preferable to select skin in a location that is not easily exposed to light in daily life (e.g., the inside of the upper arm).
When the former is selected, the overall skin condition taking into account the effects of light can be evaluated, and when the latter is selected, the (physiological) skin condition can be evaluated with the effects of light minimized.
In addition, a normal area free of visible blemishes, pigmentation, wounds, or other abnormalities is set as the measurement area.
The size of the measurement area can be preferably set to 5 mm×5 mm to 15 mm×15 mm.

In the present invention, when image data is used as data showing the internal structure of the skin, it is preferable that the image data for extracting features is three-dimensional data. Such three-dimensional data may be the measured image data itself obtained by a photoacoustic method or the like, or may be processed image data obtained by processing the measured image data.

When image data is used as data indicating the internal structure of the skin, the image data of the skin may be image data including all layers, namely the epidermis layer, dermis layer, and subcutaneous fat layer, or image data obtained by arbitrarily dividing this in the depth direction of the skin may be used.
For example, image data arbitrarily divided in the depth direction of the skin can be image data in which the epidermis layer to the subcutaneous fat layer is divided into an upper portion (the region from the epidermis to the papillary dermis), a middle portion (the boundary region from the reticular dermis to the subcutaneous fat layer), and a lower portion (the upper layer of the subcutaneous fat layer).

The skin features obtained from data indicating the internal structure of the skin and used in creating the model of the present invention include, for example, features related to blood vessel structure, features related to blood flow, features related to oxygen saturation, features related to melanin structure, and features related to moisture.
Among these, preferred forms include a feature amount relating to a blood vessel structure, a feature amount relating to oxygen saturation, and a feature amount relating to a melanin structure.
When using the feature amount relating to the blood vessel structure, it is preferable to use data including measurement data of the skin, and the data is preferably image data.
When using features related to the melanin structure, it is preferable to use data including measurement data of the epidermis layer, and the data is preferably image data.
When the feature amount relating to the oxygen saturation is used, it is preferable to use data including measurement data of the epidermis layer or measurement data of the subcutaneous fat layer. The data is preferably image data.

In the evaluation method of the present invention, one or more of the features related to blood vessel structure, the features related to oxygen saturation, and the features related to melanin structure may be used.

As the feature quantity related to the vascular structure, it is preferable to use the feature quantity related to the microvascular structure, and in particular, it is preferable to use the feature quantity related to the microvascular structure in the upper part of the skin. Such feature quantities are preferably the number of vascular branches, the curvature of the blood vessels, the total amount (volume) of blood vessels, the number of microvessels, the density of blood vessels, the average thickness of blood vessels, etc., and among these, it is preferable to use the number of vascular branches, the total amount (volume) of blood vessels, the number of microvessels, and the average thickness of blood vessels.

The feature quantity relating to the blood vessel structure can be extracted from data relating to the blood vessel structure in the measurement region measured by the non-invasive measurement method.
Image data for extracting the feature amount related to the blood vessel structure can be generated by, for example, irradiating a measurement site with pulsed light including two wavelengths of 575 nm and 650 nm, and using a difference value obtained by subtracting the photoacoustic signal of 650 nm from the photoacoustic signal of 575 nm, to generate three-dimensional image data showing the blood vessel structure inside the skin. For example, such three-dimensional image data (measurement image data) can be generated using HadatomoZ (Hadatomo is a registered trademark) by Advantest Corporation. Then, by analyzing the generated image, the feature amount related to the blood vessel structure can be extracted.
Such measured image data may be used to quantify features related to the blood vessel structure.
In a preferred embodiment, the measurement image data is processed to improve the accuracy of extraction of features relating to the blood vessel structure.

For example, a method can be used to use a deep learning model to decompose measurement image data into horizontal image data (planar image data), generate extracted planar image data by extracting elements that indicate vascular structure from this decomposed measurement planar image data, and combine this extracted planar image data to generate extracted three-dimensional image data from which the vascular structure has been extracted.
A deep learning model can be used to generate the extracted planar image data from the measured planar image data.
In this case, a data curator visually evaluates the measurement planar image data to create annotated image data that extracts the vascular structure.
A necessary number of measurement plane image data and corresponding annotation image data are used as training image data to perform deep learning. As a deep learning model, a Unet, which is a network having an encoder-decoder model structure, can be used.

Feature quantities related to oxygen saturation can include the oxygen saturation of the measurement area (e.g., any layer of the skin) and the amount of area containing microvessels with high oxygen saturation.

Image data for extracting the feature amount related to oxygen saturation can be obtained by, for example, irradiating a measurement site with pulsed light including two wavelengths of 575 nm and 586 nm, and using the difference in absorption coefficient between oxyhemoglobin and deoxyhemoglobin to generate a measurement image in which the ratio of oxyhemoglobin to the whole is visualized by color coding or the like. For example, such three-dimensional image data (measurement image data) can be generated using HadatomoZ (Hadatomo is a registered trademark) by Advantest Corporation. Then, by analyzing the generated image, the feature amount related to oxygen saturation can be extracted.
Such measurement image data may be used to quantify the feature amount relating to the oxygen saturation level.
Furthermore, the oxygen saturation level at a certain position (region) may be calculated based on the amplitude ratio of the photoacoustic signals at each wavelength at that position (region) (see the examples).

The feature quantity related to the melanin structure is a feature quantity related to a structure formed by the aggregation of melanin granules. Here, in the present invention, the feature quantity related to the melanin structure does not refer to a structure such as a melanin aggregation related to visible spots or freckles, but rather to a structure in a space of less than 1 mm square that is usually not visible to the naked eye.
Photoacoustic methods generally cannot measure individual melanin granules.
This structure has a size ranging from 10 to several hundreds of μm, and is preferably a tubular ripple structure (see FIG. 10B).
The feature amount relating to the melanin structure includes a feature amount relating to the amount of the structure present, such as volume, density, and number.

Image data for extracting features related to the melanin structure can be obtained, for example, by irradiating a measurement site with 650 nm pulsed light and generating a measurement image based on the obtained photoacoustic data. For example, such three-dimensional image data (measurement image data) can be generated using HadatomoZ (Hadatomo is a registered trademark) by Advantest Corporation. Then, by analyzing the generated image, features related to the melanin structure can be extracted.
Such measured image data may be used to quantify features related to the melanin structure.

<3> Evaluation method (estimation of internal structure)
Furthermore, the inventors have demonstrated, by using the photoacoustic method, that there is a correlation between the average blood vessel thickness in a measurement area of the skin and the oxygen saturation in the measurement area, and between the oxygen saturation in the measurement area of the skin and the volume of the melanin streak structure in the measurement area (see FIGS. 9 and 13 ).
Therefore, the present invention provides a skin evaluation method that includes estimating features related to each other's skin structures by analyzing at least one of features related to the blood vessel structure of the skin, features related to the oxygen saturation of the skin, and features related to the melanin structure in the skin.

Specifically, the higher the average thickness of blood vessels in a measurement area of the skin, the lower the oxygen saturation of the blood vessels in that measurement area (see FIG. 9 ). This relationship is particularly noticeable in the upper and lower parts of the skin.
Therefore, by measuring or estimating the average thickness of blood vessels in a measurement area of the skin, the oxygen saturation of the blood vessels in that measurement area can be estimated.
As described above, the oxygen saturation in the measurement area of the skin decreases with age, so the skin (particularly the aging state of the skin) can be evaluated by estimating the oxygen saturation of the blood vessels in the measurement area based on the measurement data of the average thickness of the blood vessels in the measurement area of the skin. The evaluation of the aging state will be described later.

Furthermore, based on the relationship between the average thickness of blood vessels in the measurement area and the oxygen saturation in the measurement area, the average thickness of blood vessels in the measurement area can be estimated by measuring or estimating the oxygen saturation in the measurement area of the skin.
As described above, the average thickness of blood vessels in a measurement area of the skin increases with age, so the skin (in particular the aging state of the skin) can be evaluated by estimating the average thickness of blood vessels in the measurement area of the skin based on the measurement data of oxygen saturation in that measurement area.

In addition, there is a correlation that the higher the oxygen saturation in the measurement area of the skin, the smaller the volume of the melanin streak structure present in the measurement area (see FIG. 13).
Therefore, by measuring or estimating the oxygen saturation in a measurement area of the skin, it is possible to estimate the volume of the melanin markings observed in said measurement area.
As described above, the volume of the melanin striated structure observed in a measurement area of the skin increases with age, so the skin (particularly the aging state of the skin) can be evaluated by estimating the volume of the melanin striated structure observed in the measurement area of the skin based on the measurement data of oxygen saturation in the measurement area of the skin.

Furthermore, based on the relationship between the oxygen saturation in the measurement area and the volume of the melanin streak structure present in the measurement area, the oxygen saturation in the measurement area can be estimated by measuring the volume of the melanin streak structure observed in the measurement area of the skin.
As described above, the oxygen saturation in the measurement area of the skin decreases with age, so the skin (particularly the aging state of the skin) can be evaluated by estimating the oxygen saturation in the measurement area based on the measurement data of the volume of the melanin pattern structure observed in the measurement area of the skin.

In this way, by mutually estimating features related to the internal structure of the skin, in creating learning data for a model that estimates features from a skin image in the present invention, it is possible to create learning data related to one feature from another feature. In addition, after estimating a feature from a skin image using a model, it is possible to further estimate another feature. The preferred forms of each feature and its extraction method are as described in <2>.

<4> Evaluation of aging state The skin evaluation method of the present invention may further include a step of evaluating the aging state of the subject's skin based on estimated values of features related to the internal structure of the subject's skin output by the model.
In a preferred embodiment of the present invention, the aging state is evaluated by inputting the estimated value into a correlation model created by providing a plurality of pairs of the feature amount and age, and calculating an estimated age.
According to the present invention, since the aging state of the skin can be evaluated from an estimated value based on a skin image, the aging state of the skin of a subject can be evaluated simply and non-invasively.

In the present invention, examples of "skin aging state" include the degree (e.g., size, number, depth, etc.) of skin aging traits (e.g., age spots, wrinkles, dullness, etc.) visible to a third party, prediction of the occurrence of aging traits that may appear in the future, the subject's impression of aging (apparent age), etc.

As shown in the examples, the inventors have clarified that each of the aforementioned features has a relationship with age. Therefore, this relationship can be utilized to evaluate the aging state of the skin based on the aforementioned features.

In this embodiment, the aging state is evaluated from features estimated based on skin images using a trained convolutional neural network model. When creating training data used to train the model, the method of extracting features from data showing the internal structure of the skin is not particularly limited, and features may be extracted by applying general statistical processing (e.g., calculation of the average, standard deviation, etc.) to data showing the internal structure of the skin obtained by a non-invasive measurement method, or features may be extracted by analyzing the image after processing such as imaging.

In the present invention, in the evaluation of the aging state of the skin, it is preferable to create a correlation model showing the correlation between the feature amount and age in advance from the viewpoint of simplicity. In this way, the feature amount estimated from the skin image can be inputted into the correlation model to calculate the estimated age. It should be noted that, in addition to age, any index related to any aging state having a correlation with the feature amount can be used.
The analysis of correlations and the creation of correlation models are described below.

The feature amount (evaluation index) used in the evaluation of the aging state in the present invention can be determined as follows.
First, a non-invasive measurement method is used to measure the measurement area of the skin of multiple subjects of different ages to obtain data on the internal structure of the skin. The data on the internal structure of the skin is preferably image data or the obtained data is visualized. When the data on the internal structure of the skin is image data, the image may be a two-dimensional image such as a cross-sectional image of the skin, or a three-dimensional image.

As a non-invasive measurement method, the various devices and methods described above can be used, but it is preferable to employ a photoacoustic method.
By adopting the photoacoustic method, it is possible to more accurately measure structures deep inside the body that could not be measured using conventional methods.
For example, when obtaining image data of the internal structure of the skin, pulsed light is irradiated onto the measurement area of the skin of multiple subjects of different ages, the generated ultrasound is measured, and image data of the internal structure of the skin is generated for each subject of each age based on the measured ultrasound data inside the skin.

Next, parameters contained in the obtained measurement data are extracted, and parameters correlated with actual age or apparent age are searched for.
The method of extracting parameters from the measurement data is not particularly limited, and known analysis methods can be used. For example, when image data is used as the measurement data, a method of visually comparing images by age group and extracting parameters of skin that shows changes can be adopted. In addition, parameters can be extracted by extracting features including color distribution, luminance distribution, spatial frequency intensity, etc. through analysis using general-purpose image processing software.

Here, when determining the features (evaluation index) used to evaluate the aging state, features estimated from skin images using a model may be used instead of parameters (features) extracted from data showing the internal structure of the skin measured using a non-invasive method.

Next, the correlation between the extracted parameters and an index indicating an aging state such as apparent age or actual age is analyzed. In this analysis, a feature amount estimated from a skin image by a model may be used, or a feature amount obtained from data indicating the internal structure of the skin measured by a non-invasive method may be used. Note that, as the apparent age, a value obtained by appropriately converting age into a generation, score, or the like may be used.
As the analysis method, a method generally used for analyzing correlation, such as regression analysis, can be adopted.

In addition, when image data is used as the measurement data, correlations can be analyzed by machine learning using training data that is image data showing the internal structure of the skin obtained by a non-invasive measurement method, labeled with data showing the degree of aging, such as apparent age or actual age.

As a result of the analysis, the feature values that are determined to be correlated with age are determined as skin evaluation indices, and a correlation model that explains the correlation between the feature values and age is created. Examples of correlation models include multiple regression equations, discriminant equations, and neural networks.

The inventors' analysis specifically found a relationship in which the older the actual age, the thicker the average thickness of the blood vessels at the measurement site of the subject (the thicker the blood vessels) (see FIG. 6).
They also found that the older the person is, the smaller the volume of blood vessels is, and the older the person is, the fewer the number of blood vessel branches in the upper part of the skin is.
It was also found that the higher the actual age, the lower the oxygen saturation in the measurement area, and this relationship was particularly strong in the upper and lower parts of the skin (see FIG. 8).

It was also found that the older the actual age, the more melanin striated structures were observed per unit skin area (the larger the volume) (see Figure 12).

Therefore, these features can be particularly preferably used as features (evaluation indices) for use in the aging state evaluation method of the present invention.

In the present invention, in addition to the above feature amounts, items known as indices for evaluating the aging state of the skin may also be measured and taken into consideration when evaluating the aging state of the skin.
For example, the moisture content, viscoelasticity, and degree of aging characteristics of the skin can be non-invasively measured using known methods, and these measurement results can be combined with at least one of the results of measurement of the above-mentioned features using a photoacoustic method to evaluate the aging state of the subject's skin.

A preferred method for evaluating skin from features is, for example, to store in advance in a computer an equation or model that indicates the correlation between features and estimated values of the aging state (apparent age or score) based on the correlation analysis described above, and to input the features into the equation or model to output an estimated value of the aging state. In the present invention, by using features estimated based on a skin image as the features input here, it is possible to estimate the aging state from the skin image through the estimation of the features.

Photoacoustic measurement image data was obtained from normal cheek areas (9 mm x 9 mm) of subjects (80 Japanese women in their 20s to 60s), excluding areas where blemishes, red spots, etc. were visible to the naked eye, using the following method, and various features were extracted.
Furthermore, an image (1200px x 1200px) of the area including the measurement area using the photoacoustic method was extracted from the facial image of the same subject, and an enlarged skin image (640px x 480px) of the cheek area was obtained using a 30x video microscope, and deep learning was performed using the skin image and features as learning data.
In addition, the correlation between the extracted features and age was analyzed to analyze features related to the aging state.

<1> Acquisition of Photoacoustic Measurement Images For the photoacoustic measurements, a Hadatomo Z photoacoustic measurement device manufactured by Advantest Corporation was used. This device is capable of dual-wavelength photoacoustic imaging and ultrasonic imaging. The measurements were performed with a scanning step of 30 μm.

Two dual wavelength lasers were used for the measurements: (i) 575 nm and 586 nm, and (ii) 575 nm and 650 nm.
For vascular imaging, the 575 nm photoacoustic signal minus the 650 nm photoacoustic signal was used.
For imaging of oxygen saturation distribution, the difference in absorption coefficient between oxyhemoglobin and deoxyhemoglobin obtained from photoacoustic signals at 575 nm and 586 nm was used.
For imaging of melanin distribution, a photoacoustic signal at 650 nm was used.

<2> Image generation for analysis of vascular structure and melanin structure In order to extract features related to vascular structure and melanin structure, analysis image data was generated based on image data obtained by photoacoustic measurement using MATLAB (registered trademark) 2020a (MathWorks, USA), Python 3.7, and Tensorflow.
Three-dimensional image data obtained by photoacoustic measurements was reconstructed into horizontal (x-y plane) images from images stacked on the z-axis. To extract vascular and melanin structures from the reconstructed images, Unet, a network with an encoder-decoder model structure, was used as a deep learning segmentation method.
Images of the measurement area where blood vessel structures and melanin structures were observed were cut out from the x-y plane images at multiple arbitrary positions on the z-axis, and extracted images were created as annotated image data (training data) for areas visually regarded as blood vessel structures or melanin aggregate structures by the data curator. These extracted images were used as output data in deep learning and their class labels. These extracted images were 90 and 30 examples of blood vessel structures and melanin structures, respectively. Next, a network with 5 iterations and 2 × 2 pooling was constructed to extract blood vessels and melanin structures using the original x-y plane images as input data. The objective function was assumed to be a Dice function calculated by the following equation (1).
Here, y represents a pixel vector of the original xy plane image, and y represents a pixel vector of the annotated image data.

Using the constructed network, x-y planar image data was extracted from the photoacoustic data acquired from the subject, with vascular and melanin structures extracted, and this extracted planar image data was then integrated again in the depth direction to create three-dimensional data.

Based on the three-dimensional data created, parameters such as length, thickness, number of branches, and volume were measured. Various parameters were measured for each of the three equal parts in the depth direction of the skin: upper, middle, and lower.

<3> Calculation of Oxygen Saturation The spatial distribution of oxygen saturation SO ₂ (r) can be calculated based on the ratio of oxygenated hemoglobin _HbO2 to deoxygenated hemoglobin Hb, as shown in formula (2).
Here, r is position information, εHb is the molar absorption coefficient of deoxygenated hemoglobin, εΔHb is the difference between the molar absorption coefficients of oxyhemoglobin and deoxyhemoglobin, and μa(r) is the photoacoustic signal. λ ₁ and λ ₂ are laser wavelengths, the wavelength λ ₁ is 575 nm, and the wavelength λ ₂ is 586 nm. Assuming that the molar absorption coefficients of deoxyhemoglobin and oxyhemoglobin corresponding to the wavelengths λ ₁ and λ ₂ are known values, the spatial distribution of oxygen saturation SO ₂ (r) can be calculated based on the amplitude ratio of the photoacoustic signal at each wavelength. The skin was divided into upper, middle, and lower parts in the depth direction, and various parameters related to oxygen saturation were measured for each part.

<4> Creation of a feature estimation model based on skin images A VISIA (Canfield, USA) was used to acquire face images. From the acquired face images, an image (1200px x 1200px) of an area including the measurement area by the photoacoustic method was extracted, and an enlarged skin image (640px x 480px) of the cheek part, which was the measurement area, was acquired using a 30x video microscope.

Deep learning was performed using MATLAB (registered trademark) 2020a and a deep learning toolbox. First, a top group of 20 people and a bottom group of 20 people were extracted from 80 Japanese female subjects in their 20s to 60s based on the measured values of the features. Then, images of the area including the measurement area of the features and enlarged skin images extracted from the face images of a total of 40 subjects in the top and bottom groups were used as input data, and a convolutional neural network model was trained to determine whether the measured values were in the top or bottom group. Here, the volume of the vascular structure, oxygen saturation, and the volume of the tubular pattern structure of melanin obtained by the photoacoustic method were adopted as the feature values, and a neural network model was created to distinguish the top group and the bottom group from the image of the area including the measurement area by the photoacoustic method and the enlarged skin image for each of them.
Furthermore, we used GoogleNet, a pre-trained model for transfer learning, to mitigate the impact of small sample sizes.
Furthermore, through repeated training, the prediction accuracy improved (to over 95%). Also, a study using validation data that was not used for learning verified that the prediction accuracy was about 75%. Figure 1 shows a representative example of the results of deep learning on the relationship between melanin structure and skin images.

<5> Analysis of the Relationship between Skin Images and Features Related to the Internal Structure of the Skin In order to visualize what image features affect classification in deep learning, the inventors performed visualization using the Grad-CAM method.

The relationship between vascular structures and facial and skin features is shown in Figure 2(A). In subjects with thinner average vascular structures, the heat map shown in the Grad-CAM image tended to expand uniformly (upper image in Figure 2(A)), whereas in subjects with thicker average vascular structures, the heat map became localized (lower image in Figure 2(A)).
A heterogeneous distribution of cloud-like features was observed in subjects with a thicker average vascular structure as shown by Guided Grad-CAM images, whereas such features were not observed in subjects with fine vascular structure.
Furthermore, in cases where the average thickness of the vascular structures was greater, the presence of color shading and unevenness was found to be perceived in the facial image (input image) consistent with the areas shown in the Guided Grad-CAM image. It is suggested that the thickness of the vascular structures observed in this study may lead to uneven coloring on the face that is perceived as sagging skin. The arrows in the Guided Grad-CAM image of the subject with the greater average thickness of the vascular structures indicate an uneven distribution of cloud-like features consistent with the presence of color shading and unevenness in the input skin image.

Next, we investigated the relationship between oxygen saturation and appearance (Figure 2B). The heat maps shown in the Grad-CAM images of subjects with high oxygen saturation tended to be wide and uniform (top image in Figure 2B). In contrast, the heat maps of subjects with low oxygen saturation tended to be separated (bottom image in Figure 2B).
Furthermore, as shown in the Guided Grad-CAM images, the distribution of features is unevenly distributed in the shape of a cloud in subjects with low oxygen saturation, whereas it is uniformly distributed in subjects with high oxygen saturation. This suggests that the oxygen saturation obtained in this study leads to relatively macroscopic color unevenness in the face. The arrows in the Guided Grad-CAM images indicate the uneven distribution of features in the shape of a cloud in subjects with low oxygen saturation, which is consistent with the presence of color shading and non-uniformity in the input images.

We also investigated the relationship between the melanin tubular pattern structure and facial skin images. We found that the features were widely distributed in a cloud-like manner, as shown in the Guided Grad-CAM images of subjects with large amounts of melanin structures (lower image in Figure 3). In addition, color unevenness was observed in areas where the input image had features, suggesting that the melanin tubular pattern structure found here is expressed as the perspective of color unevenness in the skin image captured by the skin camera (arrow in the figure). In this way, we have demonstrated the possibility that the presence of the melanin tubular pattern structure is involved in the generation of uneven color spots on the skin.

<6> Statistical Analysis The various parameters and subject ages obtained above were analyzed using JMP software version 14.0 (SAS Institute, USA). Correlations between data were evaluated by Pearson's correlation analysis.

4A shows a merged image of an image obtained from a 575 nm photoacoustic signal and an image obtained from a 650 nm ultrasound signal. From this merged image, it was found that the photoacoustic measurement can identify the vascular structure from the upper to lower depths of the subcutaneous tissue.

4B and 4C are representative measurement images showing vascular structures extracted based on the difference value obtained by subtracting the photoacoustic signal at 650 nm from the photoacoustic signal at 575 nm, where (B) is an image of a subject in his 20s and (C) is an image of a subject in his 60s.
In particular, as is clear from the top view, FIG. 4 shows that the area with the microvascular structure seen in the subject in his 20s (blood vessels indicated by white triangles in FIG. 4(B)) was not seen in the subject in his 60s (area indicated by white arrows in FIG. 4(C)).

5 shows a representative example of measured image data of an x-y plane including a vascular structure obtained by photoacoustic measurement using a network constructed using the above-mentioned deep learning, and predicted image data of the vascular structure obtained by inputting the measured image data into the network. Annotated image data of the vascular structure visually extracted by a data curator from the measured image data is also shown. (A) is a predicted image obtained from measured image data used as training data in deep learning, and (B) is a predicted image obtained from measured image data not used as training data in deep learning.
This confirmed that it is possible to generate predicted image data with a high similarity to the correct data created by the analyst even from measured image data that has not been used as learning data.

(C) is a representative example of three-dimensional data generated by combining predicted image data (xy plane) in the z-axis direction.
As a result of correlation analysis between various parameters extracted from this three-dimensional data and the subjects' ages, correlations between some parameters and ages were confirmed.

As shown in FIG. 6, the average thickness of the vascular structure in each layer of the skin was observed to increase with age.
It was also observed that with increasing age, the volume of the vasculature in the upper part decreased and the volume of the vasculature in the lower part increased (data not shown).
It was also observed that the number of branches decreased with increasing age (data not shown).
In other words, with age, the number of microvessels in the upper part of the skin decreases and the volume of the entire vascular structure also decreases, whereas in the lower part of the skin, the number of microvessels decreases but the volume of the entire vascular structure does not decrease, suggesting that the changes in the vascular structure with age differ depending on the layer of the skin (upper, middle, and lower).

From the above, it became clear that the feature value indicating the amount of microvessels can be used as an index for evaluating the state of aging.
It was also revealed that the average thickness of the vascular structure and the number of branches can be used as the feature quantities.
It was also revealed that the volume of the vascular structure in the upper part of the skin can be used as an index for evaluating the state of aging.
It was also revealed that any combination of the average thickness, number of branches, and volume of the vascular structure in the upper part of the skin can be used as an index for evaluating the aging state.

<8> Analysis of oxygen saturation level Photoacoustic measurement image data of subjects in their twenties and sixties are shown in Fig. 7. It was revealed that the spatial distribution of oxygen saturation level can be visualized by photoacoustic imaging.

In FIG. 7, (A) is measured image data showing the spatial distribution of oxygen saturation in blood vessels of a subject in his twenties, and (B) is measured image data showing the spatial distribution of oxygen saturation in blood vessels of a subject in his sixties.
7, comparing (A) and (B), the oxygen saturation in the measurement area of the subject in his 60s was generally lower than the oxygen saturation of blood vessels in the measurement area of the subject in his 20s. In addition, in the measurement image data of the subject in his 20s, an area where microvessels exist with high oxygen saturation (indicated by a triangle in (A)) was observed, but in the measurement image data of the subject in his 60s, this area did not exist (indicated by an arrow in (B)).

For statistical analysis, oxygen saturation was calculated using the method described above, and the correlation with the previously measured vascular structure features and age was analyzed.

As shown in FIG. 8, it has become clear that oxygen saturation in each layer of the skin tends to decrease with age.
In other words, it became clear that oxygen saturation can be used as an evaluation index for the aging state of the skin.

In addition, as shown in Figure 9, a negative correlation was found between the average blood vessel thickness and oxygen saturation in the upper and lower parts of the skin.

Therefore, it is possible to estimate the oxygen saturation level from the average blood vessel thickness, and conversely, it is possible to estimate the average blood vessel thickness by measuring the oxygen saturation level.
The analysis also suggests that age-related loss of microvasculature induces a decrease in oxygen saturation, and that this occurs at least in the upper and lower layers of the skin.

From the above, it became clear that the feature quantity indicating the distribution of oxygen saturation in the measurement area can be used as an index for evaluating the aging state.
It was also revealed that the oxygen saturation and the volume of microvascular regions with high oxygen saturation can be used as the feature quantities.
In particular, because oxygen saturation in the upper and lower layers is also related to the average thickness of blood vessels, which is related to the aging state, it was suggested that a more accurate evaluation could be made by combining the oxygen saturation in these layers with features related to the vascular structure.

<9> Analysis of Melanin Structure FIG. 10 shows measurement image data acquired by a photoacoustic method using pulsed light with a wavelength of 650 nm.

In FIG. 10, (A) is measured image data showing the distribution of melanin structures of a subject in his twenties, and (B) is measured image data showing the distribution of melanin structures of a subject in his sixties.
As shown in (A), melanin structures were less abundant in the skin of subjects in their twenties.
In contrast, as shown in (B), a tubular striated structure (indicated by an arrow) with a toroidal cavity was observed in the skin of a subject in his 60s.

11 shows a representative example of measured image data of an x-y plane including a melanin structure obtained by photoacoustic measurement using a network constructed using the above-mentioned deep learning, and predicted image data of the melanin structure obtained by inputting the measured image data into the network. Annotated image data of the melanin structure visually extracted by a data curator from the measured image data is also shown. (A) is predicted image data obtained from measured image data used as training data in deep learning, and (B) is predicted image data obtained from measured image data not used as training data in deep learning.
This confirmed that it is possible to generate predicted image data with a high similarity to the correct data created by the analyst even from measured image data that has not been used as learning data.

(C) is a representative example of three-dimensional data generated by combining predicted image data (xy plane) in the z-axis direction.
Correlation analysis of various parameters extracted from this three-dimensional data and the age of the subjects confirmed a correlation between the volume of the melanin tubular pattern structure and age.

As shown in FIG. 12, it was revealed that the volume of the melanin structure in the tubular pattern of melanin present in the surface layer of the skin increases with aging.
The three-dimensional distribution of melanin in normal skin has not been studied before, and there have been no reports on the melanin structure on the scale of 10 to several hundred μm demonstrated in this study.
This newly discovered feature value related to the characteristic melanin structure has been shown to be useful for evaluating the state of aging.

From the above, it became clear that the volume of melanin in the surface layer, in particular the volume of the tubular melanin pattern, can be used as an evaluation index to assess the state of aging.

We also analyzed the correlation between the above-mentioned oxygen saturation and the volume of the tubular pattern structure of melanin.
As shown in FIG. 13, it was found that the volume of melanin tubular striations increased with decreasing oxygen saturation.
Therefore, the spatial distribution of oxygen saturation can be estimated based on the volume of the melanin tubular structures.
Conversely, the volume of melanin tubular structures can be estimated based on oxygen saturation.

The present invention can be used in a method for evaluating skin in relation to cosmetics.

Claims (16)

a step of inputting a plurality of pairs of feature amounts obtained from data showing an internal structure of the skin of a subject and skin images obtained by photographing a biological surface of the same subject as learning data into a model that has learned a relationship between the feature amounts and the skin images, thereby outputting an estimate of the feature amounts;
A skin evaluation method, wherein the feature is a feature relating to a tubular pattern structure of melanin in the skin. a step of inputting a plurality of pairs of feature amounts obtained from data showing an internal structure of the skin of a subject and skin images obtained by photographing a biological surface of the same subject as learning data into a model that has learned a relationship between the feature amounts and the skin images, thereby outputting an estimate of the feature amounts;
A skin evaluation method, wherein the feature is a feature related to the vascular structure of the skin selected from an average thickness of blood vessels, a number of blood vessel branches, a volume of blood vessels in the upper part of the skin, and a curvature of blood vessels. a step of inputting a plurality of pairs of feature amounts obtained from data showing an internal structure of the skin of a subject and skin images obtained by photographing a biological surface of the same subject as learning data into a model that has learned a relationship between the feature amounts and the skin images, thereby outputting an estimate of the feature amounts;
A method for evaluating skin, wherein the feature amount is the amount of area in the skin where microvessels with high oxygen saturation are present. The skin evaluation method according to claim 1, wherein the feature is the volume of a tubular melanin pattern. The skin evaluation method according to claim 1 or claim 4, further comprising a step of estimating the oxygen saturation of the skin based on the estimated value of the feature quantity indicating the volume of the tubular pattern structure of melanin in the skin. The skin evaluation method according to claim 2, further comprising a step of estimating the oxygen saturation of the subject's skin based on the estimated value of the feature quantity indicating the average thickness of the blood vessels in the skin. The skin evaluation method according to any one of claims 1 to 6, wherein the data indicating the internal structure is non-invasively obtained data. The skin evaluation method according to any one of claims 1 to 7, wherein the model is a convolutional neural network model created by performing deep learning using the skin image as input data and the feature amount as a response variable. The skin evaluation method according to claim 8, further comprising estimating the contribution of each region of the input skin image to the estimation result by the model by analyzing the input skin image from the output side to the input side of the model. The skin evaluation method according to claim 9, further comprising displaying the contribution of each region of the skin image to the estimation result by superimposing it on the skin image. The skin evaluation method according to any one of claims 1 to 9, wherein the data indicating the internal structure is measurement data obtained by a photoacoustic method or data generated based on the measurement data. The skin evaluation method according to any one of claims 1 to 11, further comprising estimating an index of the subject's skin aging state based on the estimated value output by the model. The skin evaluation method according to claim 12, wherein the aging state is evaluated by inputting the estimated value into a correlation model created by providing multiple pairs of the feature amount and age, and calculating an estimated age. 1. A skin assessment program, comprising:
a computer is caused to function so as to output an estimate of the feature amount by inputting the skin image into a model that has learned the relationship between the feature amount and the skin image, the model using a plurality of pairs of feature amounts obtained from data showing the internal structure of the skin of the subject and skin images obtained by photographing the biological surface of the same subject as learning data;
A skin evaluation program, wherein the feature is a feature relating to a tubular pattern structure of melanin in the skin. 1. A skin assessment program, comprising:
a computer is caused to function so as to output an estimate of the feature amount by inputting the skin image into a model that has learned the relationship between the feature amount and the skin image, the model using a plurality of pairs of feature amounts obtained from data showing the internal structure of the skin of the subject and skin images obtained by photographing the biological surface of the same subject as learning data;
A skin evaluation program, wherein the feature is a feature related to the vascular structure of the skin selected from an average thickness of blood vessels, a number of blood vessel branches, a volume of blood vessels in the upper part of the skin, and a curvature of blood vessels. 1. A skin assessment program, comprising:
a computer is caused to function so as to output an estimate of the feature amount by inputting the skin image into a model that has learned the relationship between the feature amount and the skin image, the model using a plurality of pairs of feature amounts obtained from data showing the internal structure of the skin of the subject and skin images obtained by photographing the biological surface of the same subject as learning data;
A skin evaluation program, wherein the feature amount is the amount of area in the skin where microvessels with high oxygen saturation are present.