JP7731228B2 - Skin image analysis method - Google Patents

Description

Article 30, paragraph 2 of the Patent Act applies. [Public Fact 1] Date of publication: January 22, 2021. Posting address: https://www.kao.com/jp/corporate/news/rd/2021/20210122-001/ [Public Fact 2] Date of holding: January 22, 2021. Meeting name: Japan Vision Society 2021 Winter Meeting

The present invention relates to skin image analysis technology that analyzes human skin images using a discriminant model obtained through machine learning.

The following Patent Document 1 discloses a method for detecting a face region from a person's image, extracting image features (gradient histograms) of areas within the detected face region where wrinkles are likely to occur, and identifying classifications of person attributes (age and gender) using the features and a learning device (SVM (Support Vector Machine)).
Patent Document 2 listed below discloses a method for predicting the age of a user by analyzing an image of the user using a convolutional neural network (CNN) and recommending a product to be applied to an area of the skin to achieve a target skin age.
Patent Document 3 listed below discloses a method for determining and displaying the apparent skin age by performing processes for identifying the position of a face and masking macroscopic features (eyes, nose, mouth, etc.) on a digital image containing a human face, and then analyzing the image using a convolutional neural network (CNN).

JP 2013-58060 A Special table 2019-512797 publication Special Publication No. 2020-522810

As in the above-described method, when the entire face image of a user or an image of a region to be evaluated is input into a neural network or a learning device, the skin attributes and skin age for the entire input image are predicted. However, the condition of the skin may vary locally. For example, when applying cosmetics, it is difficult to apply them evenly to the entire face, and even if they are applied evenly, the appearance may vary in parts.
From this perspective, there is still room for improvement in each of the above-mentioned methods, in that they mainly only evaluate the entire skin region that is the subject of evaluation.
The present invention provides a skin image analysis technique that obtains skin condition or attribute indices for each skin patch.

According to the present invention, a skin image analysis method can be provided in which one or more processors that can use a trained discrimination model that has been machine-learned based on multiple training data including multiple combinations of correct answer information indicating common attributes or conditions of the subject skin in a training skin image and a group of training skin patch images of a predetermined image size extracted from the training skin image perform the following steps: acquiring an evaluation skin image showing the skin of a subject; acquiring a group of skin patch images of a predetermined image size from the acquired evaluation skin image; normalizing the pixel values of each of the acquired skin patch images; and inputting each normalized skin patch image into the discrimination model to acquire an index value for the skin attribute or skin condition for each skin patch image.
Furthermore, according to the present invention, there can be provided a method for training a discriminant model that discriminates the skin attributes or skin condition of an input skin patch image of a predetermined image size, the method including the steps of: acquiring a plurality of combinations of teacher skin images and correct answer information indicating common attributes or conditions of the subject skin of the teacher skin images; acquiring a group of teacher skin patch images of a predetermined image size from the acquired teacher skin images, each of which contains a skin area at a predetermined rate or more; normalizing the pixel values of each of the acquired teacher skin patch images; generating a group of teacher data that associates the correct answer information corresponding to the original teacher skin image from which each teacher skin patch image was acquired with each of the normalized teacher skin patch images; and training the discriminant model using the group of teacher data.
In addition, a skin image analysis device that is equipped with at least one or more of the above-mentioned processors and memories and is capable of executing the above-mentioned skin image analysis method, and a discriminant model learning device that is capable of executing the above-mentioned discriminant model learning method may also be provided.

The above aspect provides a skin image analysis technology that acquires index values for skin condition or attributes for each small area of skin.

FIG. 1 is a diagram conceptually illustrating an example of the hardware configuration of an information processing device capable of executing a skin image analysis method and a discriminant model learning method according to an embodiment of the present invention. 4 is a conceptual diagram illustrating a processing flow of a skin image analysis method according to the present embodiment. 1 is a flowchart of a skin image analysis method according to the present embodiment. FIG. 10 is a diagram showing an example of a color map image as index value distribution information of a group of skin patch images. FIG. 2 is a conceptual diagram illustrating a processing flow of a discriminant model learning method according to the present embodiment. 10 is a flowchart of a method for learning a discriminant model according to the present embodiment. 10 is a graph showing the relationship between the size of a skin region and the discrimination accuracy of a discriminant model. 10 is a graph showing the relationship between the makeup feel score obtained by human visual evaluation and the makeup feel score obtained by the skin image analysis method according to the present embodiment. 10 is a graph showing representative index values of makeup appearance obtained by the skin image analysis method according to the present embodiment for each of eight types of skin images for evaluation that differ in the manner in which cosmetics are applied. FIG. 10 is a diagram showing a color map image showing the index value distribution of a group of skin patch images for each of eight types of skin images for evaluation that have different cosmetic application patterns. 10 is a graph showing the relationship between estimated age ranges and actual age ranges obtained by the skin image analysis method according to the present embodiment. FIG. 10 is a diagram showing the relationship between the evaluation skin images of five subjects and a color map image showing the index value distribution of a group of skin patch images. FIG. 10 is a diagram showing color map images of skin images for evaluation taken immediately after application of two types of foundation cosmetics and 4 hours, 8 hours, and 10 hours after application. 10 is a graph showing the relationship between the number of epochs and the discrimination accuracy of the discriminant model in the discriminant model learning method according to the present embodiment. This is a graph comparing the estimation accuracy of a trained model that estimates a visual skin assessment score by inputting a face image and a trained model that estimates a visual skin assessment score by inputting a skin patch image. 10 is a radar chart showing the correlation between the estimated visual evaluation score obtained from the skin patch type discrimination model in Example 8 and age, and the change in the estimated visual evaluation score following the application of a cosmetic material.

The following describes a preferred embodiment of the present invention (hereinafter referred to as the present embodiment). The embodiment described below is merely an example, and the present invention is not limited to the configuration of the following embodiment.

The skin image analysis method (hereinafter referred to as the present analysis method) and the discriminant model learning method (hereinafter referred to as the present learning method) according to this embodiment are executed by one or more processors provided in one or more information processing devices.
FIG. 1 is a diagram conceptually illustrating an example of the hardware configuration of an information processing device 10 that can execute the present analysis method and the present learning method.
The information processing device 10 is a so-called computer, and includes a CPU 11, a memory 12, an input/output interface (I/F) 13, a communication unit 14, etc. The information processing device 10 may be a desktop personal computer (PC), a portable terminal such as a portable PC, a smartphone, or a tablet, or a dedicated computer.

The CPU 11 is a so-called processor, and may include not only a general CPU (Central Processing Unit) but also an application specific integrated circuit (ASIC), a DSP (Digital Signal Processor), a GPU (Graphics Processing Unit), etc. The memory 12 is a RAM (Random Access Memory), a ROM (Read Only Memory), an auxiliary storage device (such as a hard disk), etc.
The input/output I/F 13 can be connected to user interface devices such as a display device 15 and an input device 16. The display device 15 is a device such as an LCD (Liquid Crystal Display) or a CRT (Cathode Ray Tube) display that displays a screen corresponding to drawing data processed by the CPU 11 or the like. The input device 16 is a device such as a keyboard or mouse that accepts input of user operations. The display device 15 and the input device 16 may be integrated and realized as a touch panel.
The communication unit 14 communicates with other computers via a communication network and exchanges signals with other devices such as a printer, etc. The communication unit 14 can also be connected to a portable recording medium, etc.

The hardware configuration of the information processing device 10 is not limited to the example shown in Figure 1. The information processing device 10 may include other hardware elements not shown. Furthermore, the number of each hardware element is not limited to the example shown in Figure 1. For example, the information processing device 10 may have multiple CPUs 11. Furthermore, the information processing device 10 may be realized by multiple computers consisting of multiple housings.

The information processing device 10 can execute the present analysis method and the present learning method by having the CPU 11 execute a computer program stored in the memory 12. This computer program is installed from a portable recording medium such as a CD (Compact Disc) or a memory card, or from another computer on a network, via the input/output I/F 13 or the communication unit 14, and stored in the memory 12.
For ease of explanation, this specification provides an example in which the analysis method and the learning method are executed on a common information processing device 10, but the analysis method and the learning method may also be executed on separate information processing devices having the same or different hardware configurations. Furthermore, the analysis method and the learning method may each be implemented as separate computer programs.

The information processing device 10 (CPU 11) can use a trained discrimination model that has been machine-learned based on training data, and can also train the discrimination model by executing the present learning method.
The "discrimination model" here is a model obtained by machine learning using training data, i.e., supervised learning, and can be expressed as an AI (Artificial Intelligence) model, a machine learning (ML) model, etc.
The discriminant model used in this embodiment is only required to be able to output information capable of discriminating the skin attributes or skin conditions of a skin patch image in response to the input of the skin patch image, and there are no limitations on its data structure, learning algorithm, etc. Note that the skin attributes and skin conditions that can be discriminated based on the skin patch images input to the discriminant model and the output from the discriminant model will be described later.

The discriminant model may be, for example, a regression equation obtained by regression analysis, or may be composed of a convolutional neural network including a deep neural network, and may be realized by a combination of a computer program and parameters, or a combination of multiple functions and parameters. Furthermore, when the discriminant model is constructed using a neural network, and the input layer, intermediate layer, and output layer are considered as units of a single neural network, the discriminant model may refer to a single neural network or a combination of multiple neural networks. The discriminant model may also be composed of a combination of multiple multiple regression equations, or a single multiple regression equation. The discriminant model may also be composed of a combination of a neural network and another model, such as a regression equation or a discriminant equation, and may output a value obtained by inputting a value obtained as an output of the neural network into the other model. This embodiment provides an example in which the discriminant model is composed of a deep neural network.

This discrimination model may be stored in the memory 12 within the information processing device 10, or may be stored in the memory of another computer that the information processing device 10 can access via communication.
In this way, the information processing device 10 can be described as a skin image analysis device that can use the discriminant model and execute the present analysis method, and can also be described as a discriminant model learning device that can learn the discriminant model.
In the following description, the CPU 11 will be described as the entity that executes the present analysis method and the present learning method.

[Skin image analysis method (this analysis method)]
Here, this analysis method will be explained with reference to FIGS.
Fig. 2 is a conceptual diagram of the processing flow of the skin image analysis method according to this embodiment, and Fig. 3 is a flowchart of the skin image analysis method according to this embodiment. This analysis method is executed by an information processing device 10 (CPU 11) as exemplified in Fig. 1.
As described above, the CPU 11 can use a trained discrimination model that discriminates the skin attributes or skin conditions of a skin patch image of a predetermined image size in response to the input of the skin patch image. This discrimination model is machine-trained based on multiple training data sets that include multiple combinations of correct answer information indicating common attributes or conditions of the subject skin of the training skin images and a group of training skin patch images of a predetermined image size extracted from the training skin images. Note that this discrimination model can be trained using the present training method, which will be described later.

The teacher skin image is an image used as teacher data and shows human skin. The skin shown in the teacher skin image may include skin that is the target of skin attribute or skin condition discrimination, such as the skin of a human face, the skin of a human neck, the skin of a human arm, the skin of the back or palm of a human hand, the skin of a human leg, etc. The teacher skin image may also show other body parts other than the skin that is the target of discrimination, such as the eyes, nostrils, mouth, etc.
The attributes of the skin include, for example, attributes of the person who has the skin, such as age and sex, and the body part to which the skin belongs (cheek, forehead, face, neck, leg, etc.).
The skin condition may be, for example, a condition indicating whether the skin is bare or made up, a condition indicating whether the makeup has come off, a condition indicating whether the skin is moisturized, a condition indicating whether the skin is dry, etc. The skin condition targeted in this embodiment is preferably one that indicates the overall impression of appearance rather than one that indicates the presence of local features such as blemishes or wrinkles.
The teacher skin patch image group is an image collection including a plurality of skin patch images extracted from one teacher skin image.
A skin patch image is a small section image that contains a skin region (an image region showing skin) at a predetermined rate or more. The predetermined rate is set to be at least 90%. The skin patch image has a predetermined shape, and is preferably rectangular for ease of processing, but the shape is not limited to a circle, ellipse, or the like.
The image size of the skin patch image is predetermined, and is set to, for example, 224 pixels x 224 pixels. However, the size of the skin patch image is not limited to this example and may be determined appropriately depending on the configuration of the discriminant model, etc.

In this analysis method, first, the CPU 11 acquires a skin image for evaluation (S31).
The evaluation skin image may be an image showing the skin of the subject, or may show other parts of the subject. The image size of the evaluation skin image is not limited as long as multiple skin patch images can be extracted from the evaluation skin image.
The skin image for evaluation acquired in step (S31) is acquired as image data in, for example, JPEG (Joint Photographic Experts Group) format, BMP (Bitmap image) format, TIFF (Tagged Image File Format), GIF (Graphic Interchange Format), or other format. However, the data format of the skin image for evaluation is not limited. Furthermore, the acquired skin image for evaluation is preferably a color image captured with a camera that captures visible light in order to determine the appearance or attributes of the skin.
The CPU 11 may acquire the skin image for evaluation from a camera that captured the image, or may acquire the image from another computer or a portable recording medium.

Next, the CPU 11 acquires a group of skin patch images from the evaluation skin images acquired in step S31 (S32). The number of skin patch images included in the group of skin patch images is preferably large because it corresponds to the number of acquired index values described below, but is not particularly limited.
In addition, it is preferable that the predetermined image size of each skin patch image obtained in step (S32) is the same as the image size of each skin patch image forming the above-mentioned teacher skin patch image group, but it does not necessarily have to be the same as that image size.

The skin area of the subject appearing in each skin patch image included in the skin patch image group acquired in step (S32) and each teacher skin patch image included in the teacher skin patch image group of the teacher data is preferably a skin area of the subject of 1 cm ² or more and 20 cm ² or less, and more preferably a skin area of 3 cm ² or more and 19 cm ² or less. As will be described later as an example, the inventors have verified the discrimination accuracy of the discriminant model by varying the size of the skin area of the subject appearing in the teacher skin patch image. As a result, they have newly discovered that the size of the skin area affects the discrimination accuracy of the discriminant model, and have newly discovered that using a skin area size in the above-mentioned range can improve the discrimination accuracy of the discriminant model.
Therefore, it is preferable that the evaluation skin images and teacher skin images from which the individual skin patch images and individual teacher skin patch images are obtained are captured under shooting conditions such as resolution that the skin patch images and teacher skin patch images having a predetermined image size show the skin area in the above-mentioned range. Furthermore, by capturing an image with a color chart of a predetermined size (e.g., 1 cm square) attached to the subject's skin, it is possible to grasp the relationship between the pixel size on the evaluation skin images and teacher skin images and the actual size of the subject's skin.

In step (S32), the set of skin patch images can be obtained by various methods.
For example, one method is to divide the evaluation skin image into rectangular small-partition images of a predetermined image size, and then cut out all rectangular small-partition images that contain a skin region (image region showing skin) at a predetermined ratio or more from the group of rectangular small-partition images as skin patch images. In this method, small-partition images may be cut out at an image size larger than the final image size, and then skin patch images of the final image size may be further cut out from the small-partition images.
Another method involves acquiring a group of skin patch images by moving a cropping window of a predetermined image size on the evaluation skin image and cropping small section images within the window when a predetermined percentage or more of the skin region is included within the window. In this method, the same skin region may be partially included in overlapping skin patch images. In this case, determining the skin region when the cropping window is moved by a predetermined amount or more from the previously cropped position can prevent the same skin region from occupying a large portion of the skin patch images.
Furthermore, the skin patch image group may be acquired by other methods, and the method for acquiring the skin patch image group is not limited.

Next, CPU 11 normalizes the pixel values of each skin patch image acquired in step (S32) (S35). For example, CPU 11 can normalize the pixel values of one skin patch image by calculating the mean value and variance of the pixel values (RGB values) for that skin patch image and performing linear calculations on each pixel value of that skin patch image using the mean value and variance. In this case, the mean value and variance of the pixel values calculated for all skin patch images acquired in step (S32) may be used. Furthermore, as for the normalization method, not only general normalization methods that set the mean value to zero and the variance to 1 may be used, but also normalization methods based on normalization parameters determined for each discriminant model. Various known methods can be used for the normalization method in step (S32).

Next, the CPU 11 inputs each normalized skin patch image into a discriminant model (S37), and obtains index values of skin attributes or skin conditions for each skin patch image (S38).
The skin attributes or skin conditions indicated by the acquired index values depend on the discriminant model used. For example, when a discriminant model trained with correct answer information indicating either bare skin or made-up skin as the common skin condition of the subject skin of the teacher skin images is used, in (S38), an index value for bare skin feel (bare skin-likeness) or made-up feel (made-up skin-likeness) is acquired as the skin condition index value for each skin patch image. When a discriminant model trained with correct answer information indicating either male or female as the common attribute of the subject skin of the teacher skin images is used, an index value for male skin-likeness or female skin-likeness is acquired as the skin attribute index value for each skin patch image. When a discriminant model trained with correct answer information indicating the subject's actual age or actual age range (age group) as the common attribute of the subject skin of the teacher skin images is used, an index value for the skin attribute indicating the skin's apparent age or skin's apparent age range, or an index value for the skin's apparent age range, is acquired as the skin attribute index value for each skin patch image. Furthermore, when a discrimination model trained with correct answer information indicating either immediately after application of a cosmetic product or a predetermined time after application of a cosmetic product as a common skin condition for all subject skin images is used, the degree of makeup smearing is acquired as a skin condition index value for each skin patch image. Furthermore, when a discrimination model trained with correct answer information indicating either before application of a skin care product or after application of a skin care product is used as a common skin condition for all subject skin images, the index value of skin moisturization is acquired as a skin condition index value for each skin patch image.

In this way, the index values obtained for each skin patch image in step (S38) include one of a first index value indicating the degree of male or female skin-likeness, a second index value indicating the degree of bare skin-likeness or made-up skin-likeness, a third index value indicating the degree of skin-likeness for each age group, a fourth index value indicating the degree of makeup breakdown, or a fifth index value indicating the degree of moisturized skin.
However, the index value output from the discriminant model is not limited to this example, and may be any value that can be an index of a skin attribute or skin condition. For example, in (S38), a binary value indicating whether or not a certain skin condition or certain skin attribute exists may be acquired as the index value. However, in this embodiment, it is preferable that the index value obtained indicates an overall appearance impression rather than indicating the presence of local features such as blemishes or wrinkles.

While Figure 3 shows multiple steps (processes) in sequence, the order in which the steps of this analysis method are performed is not limited to the example of Figure 3. For example, rather than performing steps (S37) and (S38) after normalization has been completed for all skin patch images in the skin patch image group, normalization (S35), input to the discriminant model (S37), and acquisition of index values (S38) may each be performed in a single flow for each individual skin patch image. Furthermore, rather than performing steps (S35) and subsequent steps after acquiring all of the skin patch image group in step (S32), steps (S35) and subsequent steps may be performed each time a skin patch image is acquired in step (S32).

In this analysis method, as described above, a group of skin patch images of a predetermined image size obtained from the evaluation skin images is input into a trained discrimination model that has been machine-trained using a training data set consisting of ground truth information indicating the common attributes or conditions of the subject skin in the training skin images and a group of training skin patch images extracted from the training skin images, thereby obtaining index values for the skin attributes or skin conditions for each skin patch image. By obtaining index values for the skin attributes or skin conditions for each skin patch image using a discrimination model trained using ground truth information indicating the common attributes or conditions of the entire skin, it is possible to analyze the apparent skin conditions or skin attributes obtained from the entire subject skin captured in the evaluation skin images, while minimizing the effects of variations in the shooting environment of the evaluation skin images, such as lighting, and differences in the characteristics of local areas of the face.

The CPU 11 can process the index values of each skin patch image acquired in step (S38) as follows.
For example, the CPU 11 can generate index value distribution information that indicates the distribution of index values of each skin patch image according to the position of the skin region of each skin patch image. This index value distribution information may be, for example, a graph or map in which index values are arranged according to the position of the skin region of each skin patch image. This index value distribution information allows the distribution of index values of skin conditions or skin attributes to be understood.

FIG. 4 is a diagram showing an example of a color map image as index value distribution information of a group of skin patch images.
As shown in Figure 4, the CPU 11 can generate a color map image as the index value distribution information in which the index values of each skin patch image are mapped to the evaluation skin image acquired in step (S31) according to the position of the subject skin area of each skin patch image.
Although Fig. 4 is difficult to understand due to the constraints of the drawing, it is necessary to use a binary image, which indicates that the closer to black the color, the higher the index value of the skin condition or skin attribute, and the closer to white the color, the lower the index value. While Fig. 4 is shown in black and white, in the color map image, the position of each skin patch image on the evaluation skin image is represented by a color corresponding to its index value. This color map image allows users to visually grasp the distribution of index values, and because the distribution is displayed on the evaluation skin image, users can intuitively grasp the index of the skin condition or skin attribute for each skin region.

The CPU 11 can also calculate a representative evaluation value for the skin attributes or skin condition of the person being evaluated based on the index values of each acquired skin patch image. A single representative evaluation value may be calculated for the evaluation skin image acquired in step (S31), or the subject skin area of the evaluation skin image may be divided into several parts (e.g., cheeks, forehead, around the mouth, nose, etc.) that are larger than each skin patch image, and a representative evaluation value for each part may be calculated using the index values of the skin patch images belonging to each part. The representative evaluation value may be, for example, a statistical value such as the average, standard deviation, or variance of the index values of each skin patch image.
By obtaining a representative evaluation value from the index values of each skin patch image in this way, it is possible to easily grasp the overall evaluation of the skin condition or skin attributes of the entire skin or each part of the skin of the person being evaluated that appears in the evaluation skin image.

Furthermore, it is also possible to specify a skin region for which the representative evaluation value is to be calculated. In this case, the CPU 11 further executes a step of acquiring region designation information for designating a desired image region in the evaluation skin image acquired in step (S31), and calculates a representative evaluation value for the skin attribute or skin condition of the image region based on index values acquired for a group of skin patch images corresponding to the image region indicated by the region designation information acquired in that step. The CPU 11 displays the evaluation skin image on the display device 15 and identifies a designated image region on the displayed evaluation skin image using the input device 16, thereby obtaining the region designation information. The CPU 11 can also acquire such region designation information from another computer via communication.
In this way, a representative evaluation value of the skin condition or skin attribute can be obtained for a skin area that is of interest to the person being evaluated or other users.

[Discrimination model learning method (this learning method)]
Next, a method for learning the discriminant model used in the above-described analysis method will be described with reference to FIGS.
Fig. 5 is a conceptual diagram of the process flow of the discriminant model training method according to this embodiment, and Fig. 6 is a flowchart of the discriminant model training method according to this embodiment. This training method is executed by an information processing device 10 (CPU 11) as exemplified in Fig. 1.
As described above, this embodiment utilizes a discriminant model configured with a deep neural network that discriminates the skin attributes or skin condition of a skin patch image of a predetermined image size in response to the input of the skin patch image. When this learning method is executed, this discriminant model may be stored in the memory 12 within the information processing device 10, or may be stored in the memory of another computer that the information processing device 10 can access via communication.

First, the CPU 11 acquires a plurality of combinations of teacher skin images and correct answer information indicating attributes or states common to the subject skin of the teacher skin images (S61).
It is preferable that this teacher skin image be captured under shooting conditions such as resolution such that the teacher skin patch image having a predetermined image size shows a skin area of the subject of 1 ^cm2 or more and 20 cm2 or ^less , as described above. In addition, by capturing an image with a color chart of a predetermined size (for example, 1 cm square) attached to the skin of the subject of the teacher skin image, it is possible to grasp the relationship between the pixel size on the teacher skin image and the actual size of the subject's skin.
The "teacher skin image" and the "attributes" and "conditions" common to all subject skins are as described above. For example, the correct answer information indicating the common condition of all subject skins in the teacher skin image indicates either bare skin or made-up skin. Other examples of correct answer information for skin conditions include either the condition immediately after makeup application or a predetermined time after makeup application, or either the condition before or after application of a skin care product. Furthermore, examples of correct answer information for skin attributes include either male or female, or the actual age or actual age range of the subject.
Note that "common to subject skin" means that one correct answer information indicating the attributes or state of the subject skin of one teacher skin image is assigned to one teacher skin image, and that one correct answer information is commonly associated with a group of teacher skin patch images obtained from that teacher skin image as described below.

Next, the CPU 11 executes steps (S63) to (S67) for each combination of the teacher skin image and correct answer information obtained in step (S61).

The CPU 11 acquires a group of teacher skin patch images of a predetermined image size that contain a predetermined ratio or more of skin regions from the teacher skin image related to the target combination (S63). The number of teacher skin patch images acquired from one teacher skin image is preferably as large as possible from the viewpoint of increasing the number of teacher data, but is not particularly limited.
Furthermore, the predetermined image size of each skin patch image forming the skin patch image group is determined according to the configuration of the discriminant model to be trained.
Here, the skin area of the subject appearing in each of the teacher skin patch images included in the teacher skin patch image group acquired in step (S63) is preferably a skin area of the subject of 1 ^cm to 20 ^cm , more preferably a skin area of 3 ^cm to 19 ^cm . The range of the skin area of the subject appearing in such teacher skin patch images is as described above.

The method for acquiring the teacher skin patch image group from the teacher skin image in step (S63) may be the same as or different from step (S32) in this analysis method. As with the method for acquiring the skin patch image group described in step (S32), various methods can be used to acquire the teacher skin patch image group. Examples of each acquisition method are as described in step (S32), and therefore will not be described here. There are no restrictions on the method for acquiring the teacher skin patch image group.

Next, CPU 11 normalizes the pixel values of each teacher skin patch image acquired in step (S63) (S65). For example, CPU 11 can normalize the pixel values of a teacher skin patch image by calculating the average value and variance of the pixel values (RGB values) for one teacher skin patch image and then performing linear calculations on each pixel value of the teacher skin patch image using the average value and variance. The normalization method used in step (S65) is preferably the same as the normalization method used for skin patch images in step (S35) of this analysis method. Various well-known methods can also be used for the normalization method used in step (S65).

Once normalization (S65) is complete for all teacher skin images included in the group of teacher skin patch images obtained in step (S63), the CPU 11 generates teacher data (S67) that commonly associates the correct answer information corresponding to the teacher skin images related to the target combination with the teacher skin patch images normalized in step (S65).

In this way, by executing steps (S63) to (S67) for all combinations of teacher skin images and correct answer information obtained in step (S61), a group of teacher data is generated in which correct answer information corresponding to the original teacher skin image from which each teacher skin patch image was obtained is associated with each of the normalized teacher skin patch images.
The CPU 11 trains the discriminant model using the generated training data group (S69). In this embodiment, the discriminant model is trained by deep learning. However, the specific learning algorithm for the discriminant model is not limited in any way.

While Figure 6 shows multiple steps (processing) in sequence, the order in which the steps of this learning method are performed is not limited to the example of Figure 6. For example, rather than performing step (S67) after normalization has been completed for all teacher skin patch images in the teacher skin patch image group, normalization (S65) and association with correct answer information (S67) may be performed in a single flow for each individual teacher skin patch image. Also, rather than performing step (S65) and subsequent steps after acquiring all of the teacher skin patch image group in step (S63), step (S65) and subsequent steps may be performed each time a teacher skin patch image is acquired in step (S63).

In this analysis method, as described above, a discriminant model is trained using training data in which correct answer information indicating the attributes or condition of the subject skin of a group of training skin patch images obtained from a single training skin image is commonly associated with the group of training skin patch images obtained from the training skin image.
This allows for the efficient construction of a highly accurate discrimination model, since a large number of training skin patch images can be obtained from a small number of training skin images, whereas a large amount of training data is required to improve the accuracy of the discrimination model.
Furthermore, by making the correct answer information associated with each skin patch image correct answer information that indicates the common attributes or state of the entire skin appearing in the original teacher skin image, it is possible to suppress the effects of variations in the shooting environment of the teacher skin image, such as lighting, and differences in the characteristics of local areas of the face, thereby constructing a highly accurate discrimination model.

Some or all of the above-described embodiments and variations may also be specified as follows. However, the above-described embodiments and variations are not limited to the following descriptions.

<1> One or more processors that can use a trained discrimination model that has been machine-learned based on a plurality of training data including a plurality of combinations of correct answer information indicating a common attribute or state of subject skin of a training skin image and a group of training skin patch images of a predetermined image size extracted from the training skin image,
A step of acquiring an evaluation skin image showing the skin of the person to be evaluated;
acquiring a group of skin patch images of a predetermined image size from the acquired skin images for evaluation;
normalizing pixel values of each of the acquired skin patch images;
inputting each normalized skin patch image into the discriminant model to obtain a skin attribute or skin condition index value for each skin patch image;
Perform skin image analysis method.

<2> The one or more processors
The skin image analysis method according to <1> further comprises: generating index value distribution information indicating a distribution of the acquired index values according to the position of the subject skin region of each skin patch image.
<3> A color map image is generated as the index value distribution information by mapping each of the acquired index values onto the acquired skin image for evaluation according to the position of the skin region of each skin patch image.
The skin image analysis method according to <2>.
<4> The one or more processors
A step of calculating a representative evaluation value of the skin attributes or skin condition of the subject based on the index value for each of the acquired skin patch images;
<4> The skin image analysis method according to any one of <1> to <3>, further comprising:
<5> The one or more processors
acquiring area designation information that designates a desired image area in the acquired skin image for evaluation;
calculating a representative evaluation value of a skin attribute or a skin condition of an image region based on the index values acquired for a group of skin patch images corresponding to the image region indicated by the acquired region designation information;
<4> The skin image analysis method according to any one of <1> to <4>, further comprising:
<6> The correct answer information indicating a common state of subject skin of the teacher skin image included in the teacher data indicates either bare skin or made-up skin,
a bare skin feel or a makeup feel index value is acquired for each of the skin patch images as a skin condition index value;
<5> The skin image analysis method according to any one of <1> to <5>.
<7> The correct answer information indicating a common attribute of subject skin of the teacher skin image included in the teacher data indicates either male or female,
An index value of male skin-likeness or female skin-likeness is acquired for each of the skin patch images as an index value of a skin attribute.
<5> The skin image analysis method according to any one of <1> to <5>.
<8> The correct answer information indicating a common attribute of the subject's skin of the teacher skin image included in the teacher data indicates an actual age or an actual age range of the subject of the teacher skin image,
For each of the skin patch images, an index value for each skin appearance age range or an appearance age is acquired as a skin attribute index value.
<5> The skin image analysis method according to any one of <1> to <5>.
<9> The correct answer information indicating a common state of the subject skin of the teacher skin image included in the teacher data indicates either a state immediately after makeup application or a state after a predetermined time has elapsed since makeup application,
A makeup smearing degree is acquired for each of the skin patch images as an index value of skin condition.
<5> The skin image analysis method according to any one of <1> to <5>.
<10> The correct answer information indicating a common state of the subject skin of the teacher skin image included in the teacher data indicates either a state before application of a skin care agent or a state after application of a skin care agent,
A skin moisturizing index value is acquired for each of the skin patch images as a skin condition index value.
<5> The skin image analysis method according to any one of <1> to <5>.
<11> The skin area shown in the skin patch image is a skin area of the subject that is 1 ^cm2 or more and 20 ^cm2 or less.
<10> The skin image analysis method according to any one of <1> to <10>.
<12> The discrimination model is configured by a convolutional neural network including a deep neural network.
<12> The skin image analysis method according to any one of <1> to <11>.

<13> A method for learning a discriminant model that discriminates a skin attribute or a skin condition of a skin patch image in response to an input of the skin patch image of a predetermined image size, the method comprising:
obtaining a plurality of combinations of teacher skin images and correct answer information indicating attributes or states common to the subject skin of the teacher skin images;
a step of acquiring a group of teacher skin patch images of a predetermined image size, each containing a skin region at a predetermined ratio or more from the acquired teacher skin images;
normalizing pixel values of each of the acquired teacher skin patch images;
generating a group of teacher data in which the correct answer information corresponding to the teacher skin image from which each teacher skin patch image was obtained is associated with each of the normalized teacher skin patch images;
a step of training the discriminant model using the training data group;
A method for training discriminant models including:
<14>
A skin image analysis device comprising at least the one or more processors and a memory,
A skin image analysis device capable of executing the skin image analysis method according to any one of <1> to <12>.
<15> A discriminant model learning device capable of executing the discriminant model learning method according to <13>.

The above content will be explained in more detail below with reference to several examples, but the description of each example below does not impose any limitations on the above content.
In the following examples, a discriminant model constructed by transfer learning (fine tuning including fine adjustment of weights) based on a 16-layer convolutional neural network model called VGG16 was used. VGG16 is a model that inputs rectangular images of 224 x 224 pixels and classifies them into approximately 1,000 categories, and was adjusted by transfer learning to classify the images into two categories indicating skin attributes or skin conditions. However, as mentioned above, the configuration of the discriminant model is not limited to this example.

In Example 1, the size of the skin area of the subject appearing in the teacher skin patch image (hereinafter referred to as skin area size) was varied to verify the discrimination accuracy of the discrimination model. In Example 1, each teacher skin patch image and each evaluation skin patch image was a rectangular image of 224 x 224 pixels, and nine types of skin area sizes were used: 4.5 mm square (mm square), 8.9 mm square, 11.1 mm square, 13.4 mm square, 16.0 mm square, 17.8 mm square, 26.7 mm square, 35.6 mm square, and 44.5 mm square.
Teacher skin images of 269 people were captured under shooting conditions such that each teacher skin patch image had these nine skin region sizes, and teacher skin patch image groups were obtained from each teacher skin image, and discriminant models were trained using the above-mentioned training method. That is, nine discriminant models were constructed that were trained using nine teacher data groups corresponding to the nine skin region sizes.
On the other hand, evaluation skin images of the same 269 people were captured under four different lighting conditions, and a group of evaluation skin patch images was obtained from each evaluation skin image. Each evaluation skin patch image was input into a trained discrimination model, and the output was evaluated. Specifically, each skin patch image corresponding to nine skin region sizes was input into a discrimination model trained with training data for the corresponding skin region size.

7 is a graph showing the relationship between the size of the skin region and the discrimination accuracy of the discrimination model. As shown in FIG. 7, Example 1 demonstrated that the size of the skin region affects the discrimination accuracy of the discrimination model. In particular, it was confirmed that the highest discrimination accuracy was achieved when the skin region size was 17.8 mm square.
It was also demonstrated that if the skin area size is larger than 8.9 mm square (approximately 0.79 cm ² ), a discrimination accuracy of approximately 60% or more can be achieved, and if the skin area size is in the range of 16.0 mm square (approximately 2.56 cm ² ) or more and 44.5 mm square (approximately 19.8 cm ² ) or less, a discrimination accuracy of 90% or more can be achieved.
Therefore, it can be said that the skin area of the subject appearing in the evaluation skin patch image and the teacher skin patch image is preferably a skin area of the subject of 1 ^cm2 or more and 20 ^cm2 or less, and more preferably a skin area of 3 ^cm2 or more and 19 ^cm2 or less.

In Example 2, the accuracy of the makeup feel index values for each skin patch image obtained by this analysis method was verified by comparing with the discrimination accuracy based on human visual evaluation.
In the visual evaluation, five trained judges viewed the facial images and judged whether or not base makeup was applied. For the facial images, frontal faces of 269 subjects were photographed both with and without base makeup (bare skin), resulting in a total of 538 frontal facial images.
On the other hand, the 538 frontal face images were used as training skin images, and correct answer information on whether or not base makeup cosmetics had been applied was attached to each training skin image.A trained discrimination model was constructed by learning using a training data group in which a group of skin patch images and their correct answer information were associated for each training skin image.
The discriminant model was a model that outputs two index values: bare skin resemblance and made-up skin resemblance (resemblance that base makeup cosmetic material has been applied).
In contrast, for the 269 subjects, left and right side-facing facial images not used as training data were captured with and without base makeup applied, resulting in a total of 1,076 side-facing facial images being prepared as evaluation skin images. Then, for each evaluation skin image, the present analysis method using the trained discriminant model was executed to obtain index values for bare skin-likeness and made-up skin-likeness for each skin patch image, and the skin condition (bare skin or made-up skin) with the largest average index value for each subject was determined as the estimated skin condition (bare skin or made-up skin) in the present analysis method.

As a result, the highest discrimination accuracy among the five judges was 93.1%, while the discrimination accuracy using the index values obtained with this analysis method was 97.2%. This demonstrates that the accuracy of the index values for bare skin or made-up skin (make-up look) for each skin patch image obtained with this analysis method is higher than the discrimination accuracy achieved by human visual evaluation.

In Example 2, the correlation between the judgment based on human visual evaluation and the judgment based on the index value obtained by this analysis method was further verified. In the visual evaluation, each judge rated the makeup feel using a 6-point score (bare skin being 0, and the makeup feel (degree of makeup-likeness) being indicated on a 5-point scale from 1 to 5). The index value (a value ranging from -5 to 20) for bare skin-likeness or makeup-likeness (makeup feel) obtained by this analysis method was used directly as the makeup feel score. The correlation between the average makeup feel scores of the five judges and the makeup feel score obtained by this analysis method was then analyzed.
8 is a graph showing the relationship between the makeup feel score obtained by human visual evaluation and the makeup feel score obtained by the present analysis method. In Fig. 8, the vertical axis shows the makeup feel score obtained by the present analysis method, and the horizontal axis shows the makeup feel score obtained by human visual evaluation.
As a result, as shown in FIG. 8, a correlation was observed between the average makeup feel scores of the five judges and the makeup feel score obtained by this analysis method, with a correlation coefficient of 0.72.

In Example 3, it was verified whether the index value of the makeup feel for each skin patch image obtained by this analysis method is applicable to various cosmetic application modes.
As skin images for evaluation, in addition to a facial image of the subject's bare skin, makeup facial images of seven different cosmetic application modes were prepared. The makeup facial images of the seven cosmetic application modes are a makeup facial image after application of a skin care cosmetic (denoted as "skin care"), a makeup facial image after application of a skin care cosmetic and a sunscreen cosmetic (denoted as "+sunscreen"), a makeup facial image after application of a skin care cosmetic, a sunscreen cosmetic, and a first base makeup cosmetic (denoted as "base"), a makeup facial image after application of a skin care cosmetic, a sunscreen cosmetic, a first base makeup cosmetic, and a powder foundation cosmetic (denoted as "+PFD"), a makeup facial image after application of a skin care cosmetic and a second base makeup cosmetic (denoted as "base 2"), a makeup facial image after application of a skin care cosmetic, a second base makeup cosmetic, and a cream foundation cosmetic (denoted as "+cream FD"), and a makeup facial image after application of a skin care cosmetic, a second base makeup cosmetic, a cream foundation cosmetic, and a face powder cosmetic (denoted as "loose PD").
Then, a group of skin patch images was obtained for each of the eight evaluation skin images, including the bare skin face image, and each skin patch image was input into the discriminant model to obtain an index value for the makeup look of each skin patch image. Finally, a representative index value (average value) for the makeup look of each skin patch image was calculated for each of the original evaluation skin images.

FIG. 9 is a graph showing representative index values of the makeup feel obtained by this analysis method for each of eight types of skin images for evaluation that have different cosmetic application patterns.
9, the representative index value of the makeup feel calculated for the bare skin face image is the lowest, and the representative index value of the makeup feel calculated for the face image after the skin care cosmetic has been applied is next lowest. With regard to the representative index values of the makeup feel calculated for the makeup face images with other cosmetic application modes, the representative index values of the makeup feel increase in the order of the makeup face image labeled +Sunscreen, the makeup face image labeled +Base 2, the makeup face image labeled +Base, the makeup face image labeled +PFD, the makeup face image labeled +Cream FD, and the makeup face image labeled +Loose PD.
These results were generally consistent with the visual impression of the made-up face for each cosmetic application mode, and it can be said that this analysis method can detect the makeup feel associated with the application of skin care cosmetics, sunscreen cosmetics, and base makeup cosmetics, and can also appropriately detect the increase in makeup feel caused by the application of multiple cosmetics. In other words, it was demonstrated that the makeup feel index values obtained for each skin patch image using this analysis method can be applied to various cosmetic application modes.

10 is a diagram showing a color map image showing the index value distribution of a group of skin patch images for each of eight types of evaluation skin images with different cosmetic application patterns. Fig. 10 is difficult to understand because it is a binary image due to drawing restrictions, but it shows that the closer to black the image, the higher the makeup index value (the more similar it is to made-up skin), and the closer to white the image, the lower the makeup index value (the more similar it is to natural skin).
10 also shows that the makeup look index values for each skin patch image obtained by this analysis method are applicable to various cosmetic application patterns. Furthermore, the color map image showing the index value distribution of the skin patch images makes it easy to grasp which areas of the face have a high and low makeup look for each small, divided region.

In Example 4, the accuracy of the index values for each skin appearance age range of each skin patch image obtained by this analysis method was verified.
In Example 4, 512 women in various age groups from their 20s to 70s were photographed as subjects, with bare skin images and facial images after application of foundation cosmetics, and a group of skin patch images was obtained from each facial image, resulting in a total of 43,000 skin patch images. 70% of the group of skin patch images were used as a training group of skin patch images for training a discriminant model, and the remaining 30% were used as an evaluation group of skin patch images for this analysis method using the discriminant model.
The discriminant model was a model that output index values for each of six age ranges from 20s to 70s. In this analysis method, the discriminant model was used to obtain index values for each skin appearance age range for each skin patch image included in the group of evaluation skin patch images. In this example, an estimated age for each skin patch was obtained by linear regression using each index value obtained from the discriminant model as an independent variable, and the average estimated age for all skin patches obtained for each subject was determined as the estimated age in this analysis method.

11 is a graph showing the relationship between estimated age and actual age obtained by this analysis method. In Fig. 11, the vertical lines indicate the range of estimated ages obtained by this analysis method for each actual age band, the horizontal lines indicate the range of actual ages of the subjects for each actual age band, and the circles indicate the average actual ages of the subjects for each actual age band and the average estimated age obtained by this analysis method.
As a result, it was confirmed that there was a correlation between the estimated age obtained by this analysis method and the actual age with a correlation coefficient of 0.77, and the standard error of prediction was confirmed to be 10.77.
In other words, it was demonstrated that the index values and estimated ages for each skin appearance age range of each skin patch image obtained by this analysis method can be obtained with high accuracy.

In Example 5, the accuracy of the index values of male skin-likeness or female skin-likeness of each skin patch image obtained by this analysis method was verified.
In Example 5, bare skin facial images were captured for a total of 52 subjects, 26 men and 26 women, and a group of skin patch images was obtained from each facial image, resulting in a total of 1,200 skin patch images. 70% of the group of skin patch images were used as a training group of skin patch images for training a discriminant model, and the remaining 30% were used as an evaluation group of skin patch images for this analysis method using the discriminant model.
The discriminant model was a model that outputs two index values, one for male skin-likeness and one for female skin-likeness. In this analysis method, the index values for male skin-likeness and one for female skin-likeness were obtained for each skin patch image included in the group of evaluation skin patch images, and the gender with the larger index value was determined as the estimated gender for each skin patch image.

As a result, the estimated gender obtained for each skin patch image using this analysis method showed a discrimination accuracy of 97.1%, demonstrating that the index value of the male skin-likeness or female skin-likeness of each skin patch image obtained using this analysis method shows high accuracy.
12 is a diagram showing a color map image showing the index value distribution of the skin patch image group for each of the evaluation skin images of the five evaluators. Although Fig. 12 is difficult to understand due to the constraints of the drawing, which necessitates a binary image, it shows that the closer to black the color, the higher the index value of male skin-likeness (the lower the index value of female skin-likeness), and the closer to white the color, the lower the index value of male skin-likeness (the higher the index value of female skin-likeness).
In this way, the color map image shown in FIG. 12 makes it possible to easily grasp the areas of the entire face that are highly masculine or feminine in appearance and the areas that are less masculine in appearance for each small divided area.

In Example 6, the accuracy of the degree of makeup smearing of each skin patch image obtained by this analysis method was verified.
In Example 6, facial images of 60 women were taken immediately after application of a foundation cosmetic and 6 to 10 hours after application of the foundation cosmetic, and a group of skin patch images was obtained from each facial image, resulting in a total of 5,000 skin patch images. 70% of the group of skin patch images were used as a training group of skin patch images for training a discriminant model, and the remaining 30% were used as an evaluation group of skin patch images for this analysis method using the discriminant model.
The discriminant model was a model that output two index values: one immediately after cosmetic application and one after a predetermined time had elapsed since cosmetic application. In this analysis method, two index values were obtained for each skin patch image included in the group of evaluation skin patch images: one immediately after cosmetic application and one after a predetermined time had elapsed since cosmetic application, and the state with the larger index value (either immediately after cosmetic application or the predetermined time had elapsed since cosmetic application) was determined to be the estimated cosmetic application state for each skin patch image.

As a result, the estimated cosmetic application state obtained for each skin patch image using this analysis method demonstrated a discrimination accuracy of 78.4%. While makeup smearing can occur uniformly across the entire face, it is often the case that it occurs only in parts. Considering this, this discrimination accuracy value (78.4%) can be said to be sufficiently high. In other words, it was demonstrated that the two index values obtained for each skin patch image obtained using this analysis method, immediately after cosmetic application and after a predetermined time has passed since cosmetic application, demonstrate high accuracy. Furthermore, these two index values can be said to indicate the degree of makeup smearing.
13A and 13B are diagrams showing color map images showing the index value distribution of the skin patch image group for each evaluation skin image immediately after application of two types of foundation cosmetics and 4, 8, and 10 hours after application. The same woman was used as the subject, but different foundation cosmetics were used in Fig. 13A and Fig. 13B.
Figure 13 is difficult to understand because it has to be a binary image due to drawing restrictions, but it shows that the closer to black the color, the higher the index value indicating a predetermined time after application (the greater the degree of makeup smearing), and the closer to white the color, the lower the index value indicating a predetermined time after application (the less the degree of makeup smearing).
13A and 13B, it can be seen that the change in the degree of makeup smearing over time is smaller for the foundation cosmetic used in FIG. 13A than for the foundation cosmetic used in FIG. 13B.
In this way, the cosmetic smearing characteristics can be objectively evaluated using the color map image shown in Fig. 13. Furthermore, this color map image makes it easy to grasp the areas of the face that are more likely to smear and the areas that are less likely to smear, divided into small regions.

In Example 7, the accuracy of the index values of the skin moistness of each skin patch image obtained by this analysis method was verified.
In Example 7, facial images of eight women were captured before and after application of a skin care agent, and a group of skin patch images was obtained from each facial image, resulting in a total of 231 skin patch images. 70% of the group of skin patch images were used as a training group of skin patch images for training a discriminant model, and the remaining 30% were used as an evaluation group of skin patch images for this analysis method using the discriminant model.
The discriminant model was a model that outputs two index values, one before application of a skin care agent and one after application of a skin care agent. In this analysis method, two index values, one before application of a skin care agent and one after application of a skin care agent, were obtained for each skin patch image included in the group of skin patch images for evaluation, and the state with the larger index value (before application of a skin care agent or after application of a skin care agent) was determined to be the estimated skin care application state of each skin patch image.

14 is a graph showing the relationship between the number of epochs and the discrimination accuracy of the discriminant model in this learning method. The discrimination accuracy of the discriminant model indicates the accuracy of the estimated skin care application state obtained for each skin patch image, as described above.
In Figure 14, the discrimination accuracy of the discriminant model in response to the input of a teacher skin patch image is shown by a broken line labeled "teacher data," and the discrimination accuracy of the discriminant model in response to the input of an evaluation skin patch image is shown by a broken line labeled "verification data."
As a result, it was confirmed that when the discriminant model was trained using the optimal number of epochs (64), the discrimination accuracy before or after application of a skin care agent was 88.6%. In other words, it was demonstrated that the two index values obtained from each skin patch image before and after application of a skin care agent exhibit high accuracy. These two index values can be called index values of skin moisturization.

In Example 8, the accuracy of the visual skin evaluation score of each skin patch image obtained by this analysis method was verified using a trained model that estimates the visual skin evaluation score as a discriminant model.
First, 341 facial images were prepared, showing the entire bare face of 87 Japanese women aged between 20 and 39. These 341 facial images were obtained by photographing each of the 87 subjects (with some exceptions) up to four times at different times.
Furthermore, 10 judges visually evaluated the skin of 341 facial images, and as evaluation results, a 7-level evaluation score ranging from -3 points to +3 points (average value of the 10 judges) was assigned to each of the 10 evaluation items.Finally, the evaluation scores assigned to each facial image for each of the 10 evaluation items were determined to be the correct answer information.
The evaluation items were 10 items: overall evaluation, whiteness of the skin, yellowness of the skin, moisture, dryness, firmness, gloss, transparency, smoothness of the skin surface, and evenness of the skin texture.
As a result, 341 combinations of face images and 10 visual evaluation scores as correct answer information were obtained. Here, the 10 visual evaluation scores obtained as correct answer information were normalized so that the average and variance for each evaluation item were 0 and 1, respectively.

Next, face detection and skin color detection were performed on the face images, and a group of skin patch images of a predetermined image size, each containing 90% or more of the skin area, was obtained from each face image. In this example, the skin area of the subject in each skin patch image was set to a 17.8 mm square skin area of the subject, and 9,306 skin patch images were obtained from all face images.
Furthermore, the pixel values of each skin patch image were normalized, and for each normalized skin patch image, the 10 visual evaluation scores of the face image from which the skin patch image was derived were assigned as labels (correct answer information).
In this example, 70% of the 341 combinations of the skin patch images thus obtained and the 10 visual evaluation scores assigned as their labels were used as training data, and the remaining 30% were used as verification data.

The discriminant model used in Example 8 (hereinafter referred to as the skin patch model) is a model that inputs a skin patch image and outputs a 10-dimensional vector having estimated values of visual evaluation scores for 10 items as elements. This discriminant model was trained using the training data described above so as to minimize the average value between the root mean square errors (RMSE) of all 10 items, which are elements of the output vector, and the actual visual evaluation scores across evaluation items.
As a comparative example, a comparative discriminant model (hereinafter referred to as the full face model) was also generated, which inputs the above-mentioned face image and outputs a 10-dimensional vector having estimated values of the visual evaluation scores of 10 items as elements.

15 is a graph for comparing the estimation accuracy of a trained model (all-face type discriminant model) that inputs a face image to estimate a visual skin evaluation score and a trained model (skin patch type discriminant model) that inputs a skin patch image to estimate a visual skin evaluation score. For each of the 10 visual evaluation items, Fig. 15 shows the correlation coefficient between the estimated visual evaluation score output by the skin patch type discriminant model or the all-face type discriminant model and the correct answer information (visual evaluation score) of the verification data.
According to Figure 15, the skin patch type discrimination model shows an estimation accuracy with a correlation coefficient exceeding 0.6 for all 10 evaluation items, whereas the all-face type discrimination model as a comparative example shows a significant drop in estimation accuracy for the two evaluation items of sideways glance, namely skin color and yellowish skin.
From the above, the effectiveness of using skin patch images was demonstrated even when using a discriminant model to estimate visual skin assessment scores.

In this example, the estimation accuracy of the above-mentioned skin patch type discrimination model was further verified using a new group of face images obtained by photographing a subject different from the above-mentioned face images in a different shooting environment.
The new face image group includes 269 Japanese women in their 20s to 70s as subjects, and includes two face images of each subject: one with bare skin and one with foundation applied.
On the other hand, since the skin patch type discrimination model is trained using the training data described above, it has not yet been trained on facial images of subjects aged 40 or over or facial images after applying foundation.

A group of skin patch images was obtained from this new group of face images in the same manner as described above, and each skin patch image was input into a skin patch type discrimination model to obtain estimated visual evaluation scores for the 10 items for each skin patch image.
The group of skin patch images was then classified into two skin groups: bare skin and skin after foundation application. Each skin group was further classified into six age groups (20s to 70s) based on the age of the subject, and the average estimated visual evaluation score obtained by the skin patch type discrimination model was calculated for each group.

Fig. 16 is a radar chart showing the correlation between age and the estimated visual evaluation score obtained from the skin patch type discriminant model in Example 8, and the change in the estimated visual evaluation score following the application of a cosmetic product. Fig. 16(a) is a radar chart showing the average value of the estimated visual evaluation score obtained from the skin patch type discriminant model for each of the six age groups in the bare skin group, and Fig. 16(b) is a radar chart showing the average value of the estimated visual evaluation score obtained from the skin patch type discriminant model for each of the six age groups in the skin group after the application of foundation. In Fig. 16, the wider the area surrounded by connecting the points of the estimated values of the visual evaluation scores for each of the 10 evaluation items (hereinafter referred to as the chart area), the higher the estimated value of the visual evaluation score for each evaluation item.
16, it can be seen that for both the bare skin group and the foundation-applied skin group, the chart area for the 20s group is the widest, and the chart area becomes narrower as the age group increases. This means that the skin patch type discrimination model accurately captures the decline in visual skin evaluation scores with age.
Furthermore, when comparing between the same age groups, it can be seen that the chart area shown in Fig. 16(b) is wider than the chart area shown in Fig. 16(a) for all age groups. This means that the skin patch type discrimination model accurately captures the improvement in the skin visual evaluation score that accompanies the application of foundation.
From the above, it was demonstrated that the skin patch type discrimination model of Example 8 can obtain highly accurate estimates of visual evaluation scores even for new groups of face images including face images from untrained ages and face images after untrained foundation application.

10 Information processing device 11 CPU
12 Memory 13 Input/Output I/F
14 Communication unit 15 Display device 16 Input device

Claims (19)

one or more processors that can use a trained discrimination model that has been machine-learned based on a plurality of training data including a plurality of combinations of correct answer information indicating a common state of subject skin in a training skin image and a group of training skin patch images of a predetermined image size extracted from the training skin image;
A step of acquiring an evaluation skin image showing the skin of the person to be evaluated;
acquiring a group of skin patch images of a predetermined image size from the acquired skin images for evaluation;
normalizing pixel values of each of the acquired skin patch images;
an index value acquisition step of inputting each normalized skin patch image into the discriminant model to acquire a skin condition index value for each skin patch image;
Run
the correct answer information indicating a common skin state of the subject skin of the teacher skin image included in the teacher data indicates either bare skin or made-up skin,
In the index value acquisition step, an index value of a bare skin look or a makeup look is acquired for each of the skin patch images as an index value of a skin condition.
Skin image analysis method. one or more processors that can use a trained discrimination model that has been machine-learned based on a plurality of training data including a plurality of combinations of correct answer information indicating a common state of subject skin in a training skin image and a group of training skin patch images of a predetermined image size extracted from the training skin image;
A step of acquiring an evaluation skin image showing the skin of the person to be evaluated;
acquiring a group of skin patch images of a predetermined image size from the acquired skin images for evaluation;
normalizing pixel values of each of the acquired skin patch images;
an index value acquisition step of inputting each normalized skin patch image into the discriminant model to acquire a skin condition index value for each skin patch image;
Run
the correct answer information indicating a common state of the subject skin of the teacher skin image included in the teacher data indicates either a state immediately after makeup application or a state after a predetermined time has elapsed since makeup application,
In the index value acquisition step, a makeup smear degree is acquired for each of the skin patch images as an index value of a skin condition.
Skin image analysis method. one or more processors that can use a trained discrimination model that has been machine-learned based on a plurality of training data including a plurality of combinations of correct answer information indicating a common state of subject skin in a training skin image and a group of training skin patch images of a predetermined image size extracted from the training skin image;
A step of acquiring an evaluation skin image showing the skin of the person to be evaluated;
acquiring a group of skin patch images of a predetermined image size from the acquired skin images for evaluation;
normalizing pixel values of each of the acquired skin patch images;
an index value acquisition step of inputting each normalized skin patch image into the discriminant model to acquire a skin condition index value for each skin patch image;
Run
the correct answer information indicating a common state of the subject skin of the teacher skin image included in the teacher data indicates either a state before or after application of a skin care agent;
In the index value acquisition step, an index value of skin moisturizing feeling is acquired as an index value of skin condition for each of the skin patch images.
Skin image analysis method. one or more processors that can use a trained discrimination model that has been machine-learned based on a plurality of training data including a plurality of combinations of correct answer information indicating common attributes or states of subject skin in a training skin image and a group of training skin patch images of a predetermined image size extracted from the training skin image;
A step of acquiring an evaluation skin image showing the skin of the person to be evaluated;
acquiring a group of skin patch images of a predetermined image size from the acquired skin images for evaluation;
normalizing pixel values of each of the acquired skin patch images;
inputting each normalized skin patch image into the discriminant model to obtain a skin attribute or skin condition index value for each skin patch image;
Run
The skin area shown in the skin patch image is a skin area of the subject that is 1 cm ^{or more} and 20 cm or ^less .
Skin image analysis method. the one or more processors:
The skin image analysis method according to claim 1 , further comprising: generating index value distribution information indicating a distribution of the acquired index values according to the position of the subject skin region of each skin patch image. generating a color map image as the index value distribution information by mapping each of the acquired index values onto the acquired skin image for evaluation according to the position of the subject skin region of each skin patch image;
The skin image analysis method according to claim 5 . the one or more processors:
calculating a representative evaluation value of the skin condition of the subject based on the index value for each of the acquired skin patch images;
The skin image analysis method according to claim 1 , further comprising: the one or more processors:
A step of calculating a representative evaluation value of the skin attributes or skin condition of the subject based on the index value for each of the acquired skin patch images;
The method of claim 4 further comprising: the one or more processors:
acquiring area designation information that designates a desired image area in the acquired skin image for evaluation;
calculating a representative evaluation value of the skin condition of the image region based on the index values acquired for a group of skin patch images corresponding to the image region indicated by the acquired region designation information;
The skin image analysis method according to claim 1 , further comprising: the one or more processors:
acquiring area designation information that designates a desired image area in the acquired skin image for evaluation;
calculating a representative evaluation value of a skin attribute or a skin condition of an image region based on the index values acquired for a group of skin patch images corresponding to the image region indicated by the acquired region designation information;
The method of claim 4 further comprising: The correct answer information indicating a common attribute of subject skin of the teacher skin image included in the teacher data indicates either male or female,
An index value of male skin-likeness or female skin-likeness is acquired for each of the skin patch images as an index value of a skin attribute.
The method for analyzing skin images according to claim 4, 8 or 10 . The correct answer information indicating a common attribute of the subject's skin of the teacher skin image included in the teacher data indicates the actual age or actual age range of the subject of the teacher skin image,
For each of the skin patch images, an index value for each skin appearance age range or an appearance age is acquired as a skin attribute index value.
The method for analyzing skin images according to claim 4, 8 or 10 . The discrimination model is composed of a convolutional neural network including a deep neural network.
The method for analyzing skin images according to any one of claims 1 to 12 . A method for learning a discriminant model for discriminating a skin condition of a skin patch image in response to an input of the skin patch image having a predetermined image size, comprising:
obtaining a plurality of combinations of teacher skin images and correct answer information indicating a common state of subject skin in the teacher skin images;
a step of acquiring a group of teacher skin patch images of a predetermined image size, each containing a skin region at a predetermined ratio or more from the acquired teacher skin images;
normalizing pixel values of each of the acquired teacher skin patch images;
generating a group of teacher data in which the correct answer information corresponding to the teacher skin image from which each teacher skin patch image was obtained is associated with each of the normalized teacher skin patch images;
a step of training the discriminant model using the training data group;
Including,
The correct answer information indicating a common state of subject skin in the teacher skin image indicates either bare skin or made-up skin.
How to train a discriminant model. A method for learning a discriminant model for discriminating a skin condition of a skin patch image in response to an input of the skin patch image having a predetermined image size, comprising:
obtaining a plurality of combinations of teacher skin images and correct answer information indicating a common state of subject skin in the teacher skin images;
a step of acquiring a group of teacher skin patch images of a predetermined image size, each containing a skin region at a predetermined ratio or more from the acquired teacher skin images;
normalizing pixel values of each of the acquired teacher skin patch images;
generating a group of teacher data in which the correct answer information corresponding to the teacher skin image from which each teacher skin patch image was obtained is associated with each of the normalized teacher skin patch images;
a step of training the discriminant model using the training data group;
Including,
The correct answer information indicating a common state of the subject skin of the teacher skin image indicates either a state immediately after applying makeup or a state after a predetermined time has elapsed since applying makeup.
How to train a discriminant model. A method for learning a discriminant model for discriminating a skin condition of a skin patch image in response to an input of the skin patch image having a predetermined image size, comprising:
obtaining a plurality of combinations of teacher skin images and correct answer information indicating a common state of subject skin in the teacher skin images;
a step of acquiring a group of teacher skin patch images of a predetermined image size, each containing a skin region at a predetermined ratio or more, from the acquired teacher skin images;
normalizing pixel values of each of the acquired teacher skin patch images;
generating a group of teacher data in which the correct answer information corresponding to the teacher skin image from which each teacher skin patch image was obtained is associated with each of the normalized teacher skin patch images;
a step of training the discriminant model using the training data group;
Including,
the correct answer information indicating a common state of the subject skin of the teacher skin image indicates either a state before application of a skin care agent or a state after application of a skin care agent;
How to train a discriminant model. A method for learning a discriminant model that discriminates skin attributes or skin conditions of a skin patch image in response to an input of the skin patch image having a predetermined image size, comprising:
obtaining a plurality of combinations of teacher skin images and correct answer information indicating attributes or states common to the subject skin of the teacher skin images;
a step of acquiring a group of teacher skin patch images of a predetermined image size, each containing a skin region at a predetermined ratio or more from the acquired teacher skin images;
normalizing pixel values of each of the acquired teacher skin patch images;
generating a group of teacher data in which the correct answer information corresponding to the teacher skin image from which each teacher skin patch image was obtained is associated with each of the normalized teacher skin patch images;
a step of training the discriminant model using the training data group;
Including,
a skin area of the subject appearing in each of the teacher skin patch images included in the teacher skin patch image group is equal to or greater than 1 ^cm2 and equal to or less than 20 ^cm2 ;
How to train a discriminant model. A skin image analysis device comprising at least the one or more processors and a memory,
A skin image analysis device capable of executing the skin image analysis method according to any one of claims 1 to 13 . A discriminant model learning device capable of executing the discriminant model learning method according to any one of claims 14 to 17 .