JP7759182B2 - Information processing device, program, and method of operating information processing device - Google Patents

Description

This disclosure relates to an information processing device, a program, and an operating method of an information processing device.

There has long been much interest in the predicted age that others perceive based on one's own face, and various evaluation methods have been investigated. For example, methods have been proposed that involve visual evaluation by dedicated engineers and a method that uses AI to estimate skin age from facial images (Patent Document 1).

Special table 2019-512797 publication

It is known that the lower half of the face, particularly the area around the mouth, is likely to have an impact on how others predict one's age. For example, factors thought to influence this include the appearance of wrinkles such as nasolabial folds and marionette lines, noticeable sagging cheeks, and dullness of the red lip area. Techniques have been established that allow quantitative evaluation using visual evaluation by dedicated technicians and image processing technology using digital images, but age-related changes around the mouth are not only diverse but also vary greatly between individuals, making it difficult to objectively define which characteristics are most influential among those common to age-related changes at different ages and those that differ between individuals.

Furthermore, with the current COVID-19 pandemic making mask-wearing a habit, opportunities to see the lower half of the face, especially the mouth, have decreased, but activity around the mouth has dropped significantly, raising concerns that this may accelerate the progression of mouth aging. Therefore, there is a need for technology to identify facial areas that affect age prediction values and color information that contributes to lowering the appearance of age, as well as cosmetic counseling technology based on this information.

Therefore, the present inventors have conducted extensive research and have discovered the following technology.
The information processing device according to the present disclosure includes a storage unit that stores a machine learning model 1 that has performed machine learning to learn associations between characteristic parts of the lower half of a face of a face image that contribute to a predicted age value and the predicted age value using a face image of a person and a predicted age value for the face image as training data, and a machine learning model 2 that has performed machine learning to learn influences of the predicted age value based on color information of the characteristic parts;
The apparatus has a control unit that uses the model to determine and output characteristic parts of an input face image and color information relating to the characteristic parts.

Further, the method of operating an information processing device according to the present disclosure includes a first step of creating a model by machine learning a correspondence between a feature part of a face image that contributes to a predicted age value and the predicted age value, using a face image of a person and a predicted age value for the face image as training data;
a second step of creating a machine learning model that learns the influence level by machine learning using color information of the characteristic part;
and a third step of determining and outputting characteristic parts of the input face image and color information relating to the characteristic parts using the model.

The information processing device and the like disclosed herein can identify areas that contribute significantly to a person's age estimation prediction, and can also indicate the degree of influence of color information in the identified areas, making it possible to calculate an estimated age with objectivity and accuracy. It also makes it possible to provide cosmetic consultations based on this information.

FIG. 1 illustrates an example of the configuration of an information processing system. FIG. 1 illustrates an example of the configuration of an information processing device. FIG. 2 is a diagram illustrating an example of the configuration of a terminal device. FIG. 10 is a flowchart illustrating an example of an operation performed by the information processing device. FIG. 10 is a flowchart illustrating an example of an operation performed by the information processing device. This is an example of a mouth image obtained by dividing an image into 16 parts. This is an example of a mouth image in which an image is divided into 16 parts and some of the parts are masked.

The following describes an embodiment of the present invention.

Figure 1 is a diagram showing an example configuration of an information processing device in one embodiment. The information processing system 1 supports sales staff in, for example, a cosmetics store in providing beauty treatment counseling to customers. The information processing system 1 includes a terminal device 11 and an information processing device 12. The terminal device 11 and the information processing device 12 are connected to each other peer-to-peer via a network 10, or via wired or short-range wireless communication without using the network 10, and transmit and receive information. The network 10 includes the Internet, at least one wide area network (WAN), at least one metropolitan area network (MAN), or any combination thereof. The network 10 may also include at least one wireless network, at least one optical network, or any combination thereof. The wireless network may be, for example, an ad hoc network, a cellular network, a wireless local area network (LAN), a satellite communication network, or a terrestrial microwave network. The terminal device 11 is an information processing terminal device with an imaging function, such as a tablet terminal or smartphone, used by a cosmetics sales staff. The information processing device 12 is, for example, a computer such as a server belonging to a cloud computing system or other computing system.

The information processing device 12 uses the lower half of the facial image (hereinafter referred to as the mouth image) obtained by dividing the facial image obtained by capturing a person's face into upper and lower parts based on the cheekbones, and the person's actual age corresponding to the mouth image as training data, and stores in advance a machine learning model (hereinafter referred to as the age prediction value model) that associates the characteristic parts of the facial image that contribute to age prediction with the age prediction.
In a sales store, a salesperson stationed at the store captures an image of the face of a visiting customer using a terminal device 11. The information processing device 12 receives an input of the customer's facial image from the terminal device 11, acquires a mouth image from the input facial image, calculates characteristic parts of the mouth image and a predicted age value using an age prediction model, and outputs the calculated predicted age value to the terminal device 11. The salesperson uses the terminal device 11 to display or otherwise output to the customer the predicted age value, the image region contributing to the predicted age value, a heat map showing the contribution of the region to an increase or decrease in the predicted age value, or changes in color information in the region that contribute to lowering the predicted age value. Presenting the information to the customer can improve the customer's sense of satisfaction with the predicted age value.

The information processing device 12 also stores recommendation information indicating the cosmetics that should be recommended to the customer. For example, the recommended information can include skin care cosmetics to reduce sagging, makeup cosmetics that even out blemishes with the skin tone, and product group information and effects required to enhance a youthful appearance. The information processing device 12 then selects the cosmetics to recommend to the user based on the recommendation information, and further outputs information indicating the selected cosmetics. The salesperson receives the information output by the information processing device 12 via the terminal device 11 and presents it to the customer. This makes it possible to understand the customer's aging phenomenon and recommend cosmetics that will enhance a youthful appearance.

Figure 2 shows an example configuration of the information processing device 12. The information processing device 12 is, for example, a server computer that functions as a server that implements various functions. The information processing device 12 may be one or more server computers that are connected to each other so that they can communicate information and operate in cooperation with each other. The information processing device 12 has an input/output unit 20, a communication unit 21, a memory unit 22, and a control unit 23.

The input/output unit 20 has an input interface that detects user input and sends the input information to the control unit 23. The input interface may be any input interface, including, for example, physical keys, capacitive keys, a touch screen integrated with a panel display, or various pointing devices. The input/output unit 20 also has an output interface that outputs information generated by the control unit 23 or read from the memory unit 22 to the user. Such an output interface may be, for example, a display that outputs information as an image or video, a speaker that outputs information as audio, or any output interface including a connection interface with an external output device.

The communication unit 21 has a communication module that supports one or more wired or wireless LAN standards for connecting to the network 10. Alternatively, the communication unit 21 may have a communication module that supports one or more wired or wireless standards for connecting peer-to-peer with the terminal device 11. Such a communication module may include a communication module that supports short-range communication such as Bluetooth (registered trademark), AirDrop, IrDA, ZigBee, FeliCa, RFID, etc.

The storage unit 22 includes one or more memories. Each memory included in the storage unit 22 may be, for example, a semiconductor memory, a magnetic memory, or an optical memory, but is not limited to these. Each memory functions, for example, as a main memory device, an auxiliary memory device, or a cache memory. The storage unit 22 stores any information and control/processing programs used in the operation of the information processing device 12. The storage unit 22 may also store application programs that provide various functions and are downloaded via the network 10 or other networks.

The control unit 23 has one or more processors. Each processor may be, but is not limited to, a general-purpose processor or a dedicated processor specialized for a specific process. The control unit 23 comprehensively controls the operation of the information processing device 12 in accordance with the control and processing programs stored in the memory unit 22.

In the information processing device 12, the memory unit 22 stores an age prediction value model 24 created by the control unit 23 using the procedure described below. The memory unit 22 also stores recommendation information 25 including information on cosmetics to be recommended for combinations of characteristic parts and age prediction values associated in the age prediction value model 24, and reference information 26 regarding the criteria for determining age prediction values.

Figure 3 shows an example configuration of the terminal device 11. The terminal device 11 is an information processing terminal device such as a tablet terminal or a smartphone. The terminal device 11 has an input/output unit 30, a communication unit 31, a memory unit 32, and a control unit 33.

The input/output unit 30 has an input interface that detects user input and sends the input information to the control unit 33. Such an input interface may be any input interface including, for example, physical keys, capacitive keys, a touch screen integrated with a panel display, various pointing devices, a microphone that accepts audio input, or a camera that captures captured images or image codes. The input/output unit 30 also has an output interface that outputs information generated by the control unit 33 or read from the memory unit 32 to the user. Such an output interface may be, for example, a display that outputs information as an image or video, a speaker that outputs information as audio, or any output interface including a connection interface with an external output device.

The communication unit 31 includes a communication module compatible with wired or wireless LAN standards, a module compatible with mobile communication standards such as 4G (4th Generation) and 5G (5th Generation), and a communication module compatible with short-range communication standards such as Bluetooth (registered trademark), AirDrop, IrDA, ZigBee, FeliCa, and RFID. The terminal device 11 communicates information with the information processing device 12 via the network 10 or peer-to-peer using the communication unit 31.

The storage unit 32 includes one or more memories. Each memory included in the storage unit 32 may be, for example, a semiconductor memory, a magnetic memory, or an optical memory, but is not limited to these. Each memory functions, for example, as a main memory device, an auxiliary memory device, or a cache memory. The storage unit 32 stores any information used in the operation of the terminal device 11. For example, the storage unit 32 stores control and processing programs, embedded software, application programs that provide various functions and are downloaded via the network 10 or other networks, etc.

The control unit 33 has one or more processors. Each processor may be, but is not limited to, a general-purpose processor or a dedicated processor specialized for a particular process. The control unit 33 operates in accordance with the control and processing programs stored in the memory unit 32, thereby providing overall control over the operation of the terminal device 11.

In the terminal device 11, the control unit 33 responds to operational input by the salesperson to the input/output unit 30 by causing the camera of the input/output unit 30 to capture an image of the customer's face, and sends the captured facial image to the information processing device 12 via the communication unit 31. The control unit 33 also receives information sent from the information processing device 12 via the communication unit 31 and displays it on the display of the input/output unit 30.

Figure 4 is a flowchart showing an example of the operational procedure of the information processing device 12 when the information processing device 12 creates an age prediction value model 24 through machine learning. This flowchart corresponds to the information processing procedure by the control unit 23.

In step S400, the control unit 23 reads the training data and stores it in the memory unit 22. The training data includes facial images obtained by capturing a person's face with a digital camera and the actual age of the person in the image. Hereinafter, the facial images included in the training data are images of the lower half of the face, obtained by dividing an image into upper and lower halves using the individual's cheekbones as a guide (hereinafter referred to as "mouth images"), and are referred to as training facial images. First, the training facial images are obtained, for example, by capturing images of the faces of several hundred Japanese women, roughly uniformly distributed between their 20s and 70s, from the front. Each person's hairstyle is standardized using a hair net, and they are not wearing makeup, and a mouth image is created from images of them with a straight face. For example, facial features may be extracted by using an open source library such as OpenCV, or by specifying the vertices of a bounding box on the device, but these are not limiting.

In step S402, the control unit 23 performs image processing on each training face image as a preliminary step to machine learning. Image processing includes, for example, normalization of the training face image, extraction of facial features from the training face image, random cropping, random horizontal and vertical flips, random rotations, random saturation adjustments, vertical dropout, and the like. When extracting facial features, facial features can be extracted using any open source library to eliminate the influence of features such as hair and neck in the training face image, or facial features can be extracted by specifying the vertices of a bounding box by the operator. In addition, in one example of random cropping, a 512 x 512 pixel training face image is resized to 256 x 256 pixels, and then randomly cropped to 224 x 224 pixels.

In step S404, the control unit 23 creates the predicted age value model 24 through machine learning using training data. The control unit 23 performs machine learning using, for example, a convolutional neural network. For example, the control unit 23 extracts feature parts from the training face images using a convolutional layer and a pooling layer, and compares the extracted feature parts with the predicted age values using a multi-layer neural network, thereby learning the feature parts that contribute to the predicted age values assigned to the training face images and the predicted age values in association with each other, and creates the predicted age value model 24.
Next, the partially masked mouth image shown in Figure 6B is input to the supervised learning model 1, and the output predicted age value and the rate of change from the predicted age value before masking are calculated. This predicted age may be an age range with a certain range. Areas with a high rate of change are then recognized as areas that contribute to age prediction.

Figure 5 is a flowchart showing an example of the operational procedure of the information processing device 12 when the information processing device 12 uses the age prediction value model 24 to predict the age of a customer's facial image. This flowchart corresponds to the information processing procedure performed by the control unit 23.

In step S500, the control unit 23 accepts input of a facial image and customer information. For example, a salesperson at a store uses the terminal device 11 to capture an image of the face of a visiting customer, and inputs the customer's age, gender, and other customer information into the terminal device 11. The terminal device 11 sends the captured image and customer information to the information processing device 12. In the information processing device 12, the control unit 23 accepts the captured image and customer information via the communication unit 21. The facial image and customer information are stored in the memory unit 22. Hereinafter, the facial image accepted by the control unit 23 and used to calculate an age prediction value is referred to as an evaluation facial image.

In step S502, the control unit 23 performs image processing on the evaluation face image as a preliminary step to determining the predicted age value. The image processing includes normalizing the evaluation face image, extracting facial features from the evaluation face image, random cropping, random flipping in the horizontal and vertical directions, random rotation, random saturation adjustment, vertical dropout, etc., and creates a mouth image using OpenCV or the like.

In step S504, the control unit 23 uses the age prediction value model 24 to determine the predicted age value of the evaluation face image. The control unit 23 determines the class of evaluation value corresponding to the maximum value in the final layer output of the age prediction value model 24 as the predicted age value. The evaluation value is assigned a discrete value from 1 to 9 for each age group.

In step S506, the control unit 23 determines the characteristic features that contribute to the predicted age value. The mouth image is divided into multiple analysis regions, and one region from each analysis region is selected and initialized with a brightness value of 1 (hereinafter referred to as masking). An occlusion region is created in the mouth image. By processing the mouth image to a pure white state with a brightness value of 1, all information contained in each region is deleted, allowing the impact of each region on age prediction to be examined. For example, in Figure 6A, the number of occlusion regions is set to 16. However, if the number is small, the output change of the machine learning model will be minimal, and if it is too large, multiple pieces of information will be included in the analysis region, making interpretation difficult. A partially masked mouth image is input, and the output predicted age value and the rate of change compared to the predicted age value before masking are calculated. This predicted age may be a range of ages. Considering the bilateral symmetry of the face, the number of regions is preferably 2n x 2n (n is an arbitrary integer), such as 4, 8, or 16. Regions with a high rate of change are then recognized as characteristic features that contribute to age prediction.
The control unit 23 also obtains color information that contributes to lowering the predicted age value. It performs occlusion processing of the a* value, b* value, and grayscale of the characteristic portion. The a value corresponds to a, which is one of the complementary color dimensions in the L*a*b color space.
By converting the RGB color information for the feature areas to L*a*b color information, setting the a value to 0, and then converting it back to RGB color information, only the color information for the a* value within the area is eliminated, allowing us to examine the impact of the occlusion process, which sets the a* value to 0 for each area, on age prediction. Furthermore, the b* value corresponds to b, one of the complementary color dimensions in the L*a*b color space. By converting the RGB color information for the feature areas to L*a*b color information, setting the b* value to 0, and then converting it back to RGB color information, only the color information for the b* value within the area is eliminated, allowing us to examine the impact of the occlusion process, which sets the b* value to 0 for each area, on age prediction. Furthermore, grayscale is a method of representing color in computers and photographs, and represents the absence of information other than luminosity. Therefore, by processing the image to grayscale, i.e., black and white, for the feature areas, the color information within each area is eliminated, leaving only the shape information, allowing us to examine the impact of the color information for each area on age prediction.

In step S506, the control unit 23 may use the age prediction value model 24 to visualize the feature portions that contribute to the age prediction value of the evaluation facial image. For example, the feature portions may be represented as a heat map. The control unit 23 creates a heat map showing the feature portions via a convolutional neural network. The control unit 23 converts the maximum output value in the final output layer into a probabilistic expression using a numerical value between 0 and 1 using an activation function such as a sigmoid function or a softmax function. The control unit 23 calculates the gradient with respect to the maximum output value of the final output layer, i.e., the partial differential value, for each pixel of the feature map, which consists of 512 channels in the final convolutional layer. The control unit 23 then calculates the average value of the gradient for each pixel and creates an activation map for 512 channels by multiplying each pixel of the feature map by this average value. The number of channels in the feature map shown here is an example, and the number of channels may be other than 512. The control unit 23 then integrates the created activation map across all channels and scales the integrated activation factors to values between 0 and 1 to generate a two-dimensional activation map, i.e., a heat map. The control unit 23 then resizes the heat map to the size of the face image for evaluation to create a heat map image corresponding to the face image for evaluation. The control unit 23 may also add the heat map image to the face image for evaluation and scale it with a gradation value between 0 and 255 to generate a display image in which the heat map is superimposed on the face image for evaluation. By superimposing the heat map on the face image for evaluation, characteristic portions in the face image for display are displayed with a predetermined color distribution that is arbitrarily set. Note that when scaling the size of the heat map, the control unit 23 may eliminate outliers using an arbitrarily set threshold value to remove noise from the heat map.

In step S508, the control unit 23 selects recommended cosmetics based on the age prediction value, characteristic part, and color information. For example, if the characteristic part is the cheek area, cosmetics for skin care to improve sagging of the cheeks are selected as cosmetics corresponding to the characteristic part. For the mouth area, cosmetics for skin care to improve sagging around the lips are selected for skin care of the mouth area. Furthermore, for the chin area, cosmetics for skin care to improve sagging around the chin are selected. Furthermore, if the color information significantly contributes to increasing the age prediction value, skin care cosmetics containing an anti-inflammatory agent that improves inflammation by suppressing localized redness are selected, since localized inflammation is expected when the a* value has a large influence, and whitening cosmetics that improve dullness are selected, since localized dullness is expected when the b* value has a large influence.

According to this embodiment, it is possible to reproduce objective age prediction values with high accuracy. Furthermore, it is possible to visualize the characteristic parts that contributed to the age prediction value through display, which makes it possible to improve the customer's sense of satisfaction with the age prediction value.

In the above description, the information processing device 12 acquires facial images and displays various output information via the terminal device 11. However, the information processing procedures shown as the operation of the information processing device 12 may be shared between the information processing device 12 and the terminal device 11 as appropriate. In this case, the information processing device 12 and the terminal device 11 constitute an "information processing device." For example, the information processing device 12 may execute the machine learning procedure shown in FIG. 4 to create the age prediction value model 24, and the terminal device 11 may execute the procedure shown in FIG. 5 using the age prediction value model 24 of the information processing device 12. Alternatively, for example, the information processing device 12 may operate standalone and acquire facial images and display various output information using the input/output unit 20. Furthermore, the terminal device 11 does not have to be installed in a store; for example, it may be a smartphone used by a customer. In this case, the customer captures an image of their face and sends the facial image to the information processing device 12. The smartphone then receives information from the information processing device 12 and displays it to the customer. This allows, for example, the customer to obtain information about recommended cosmetics without visiting a store.

In addition, in the above explanation, the information processing device 12 generates the age prediction value model 24 once and then predicts the customer's age. However, with the customer's consent, the evaluation images can be imported into the training data, and an experienced evaluator can assign an age prediction value using the information processing device 12 or a terminal device capable of communicating with the information processing device 12, thereby allowing the training data to be updated at any time.

REFERENCE SIGNS LIST 1 Information processing system 10 Network 11 Terminal device 12 Information processing device 20, 30 Input/output unit 21, 31 Communication unit 22, 32 Storage unit 23, 33 Control unit

Claims (6)

a storage unit for storing a model that has been machine-learned to associate a person's lip image with a predicted age value , using the person's lip image and the person's actual age as training data;
a control unit that uses the model to determine and output an input first lip image and a first predicted age value corresponding to the input first lip image ,
the control unit outputs information for superimposing on the analysis region, information indicating a rate of change in the first age predicted value before and after a process for changing a predetermined value in a color space in the analysis region obtained by dividing the first mouth image, and displaying the information;
the storage unit further stores recommendation information indicating a cosmetic product to be recommended for a combination of the analysis region and the age predicted value associated in the model;
The control unit selects cosmetics to be recommended for a combination of the analysis area having a predetermined change rate and the age prediction value based on the recommendation information, and further outputs information indicating the selected cosmetics. In claim 1,
In the process of changing the predetermined value, one or more of a decrease in the a* value, a decrease in the b* value, and setting the grayscale value to a predetermined value is executed. In claim 1 or 2 ,
In the information processing device, the information indicating the rate of change is displayed in a manner superimposed on the analysis region as a heat map. In any one of claims 1 to 3 ,
The information processing device, wherein the subdivided locations are 2n x 2n (n is any integer) locations. A program that, when executed by a computer, causes the computer to operate as the information processing device according to any one of claims 1 to 4 . a first step of creating a model by machine learning the correspondence between a person's lip image and a predicted age value using a person's lip image and the person's actual age as training data;
a second step of calculating and outputting the input first lip image and a first predicted age value corresponding to the first lip image using the model;
a third step of outputting information for superimposing on the analysis region the information indicating a rate of change in the first predicted age value before and after a process for changing a predetermined value in a color space in the analysis region obtained by dividing the first mouth image; and
a fourth step of referring to recommendation information indicating cosmetics to be recommended for a combination of the analysis area and the predicted age value associated in the model, and selecting cosmetics to be recommended for the combination of the analysis area and the predicted age value having a predetermined change rate ;
a fifth step of further outputting information indicating the selected cosmetic product;
A method for operating an information processing device having the above-mentioned components.