JP7740873B2 - Information processing device, program, and method of operating information processing device - Google Patents

Description

This disclosure relates to an information processing device, a program, and an operating method of an information processing device.

In the sale of cosmetics, there is a demand for customizing beauty treatments to meet the diverse needs of customers. For example, when a customer wishes to improve their skin condition, a salesperson will provide counseling by asking about the customer's age, skin type, skin concerns, etc. using a questionnaire, and measuring the customer's physical characteristics using equipment, and then recommend products suited to the customer's skin type. Patent Document 1, for example, proposes a technology to support this process: an information processing device that estimates a customer's skin age from a facial image taken of the customer's face.

Special table 2019-512797 publication

To further increase customer satisfaction, it is desirable to objectively evaluate the overall impression that a customer's appearance makes on others and provide beauty treatments to improve that impression. While skin type can be quantitatively measured using conventional methods, such as measuring the physical characteristics of the skin or estimating skin age through image recognition of those physical characteristics, it is only one of the factors that influence impression. Many of the factors that influence impression, such as youthfulness and liveliness, are difficult to measure quantitatively and are grasped intuitively. Furthermore, objective evaluation of these non-quantitative, sensory factors relies on the personal skills of an experienced beauty technician. Therefore, to objectively evaluate impression evaluations, it is necessary to ensure highly accurate reproducibility of impression evaluations by experienced evaluators.

In light of the above, the following describes an information processing device and the like that can reproduce impression evaluations by experienced evaluators with high accuracy and support the sale of cosmetics.

The information processing device disclosed herein has a storage unit that stores a machine-learned model that uses a person's facial image and an impression evaluation of the facial image as training data to associate the characteristic parts of the facial image that contribute to the impression evaluation with the impression evaluation, and a control unit that uses the model to determine and output the characteristic parts of an input facial image and the impression evaluation that corresponds to the characteristic parts.

Furthermore, the operating method of the information processing device disclosed herein includes a first step of using a person's facial image and an impression evaluation of the facial image as training data to perform machine learning to associate the characteristic parts of the facial image that contribute to the impression evaluation with the impression evaluation, thereby creating a model; and a second step of using the model to determine and output the characteristic parts of an input facial image and the impression evaluation corresponding to the characteristic parts.

The information processing device and the like disclosed herein can support the sales of cosmetics by reproducing with high accuracy impression evaluations by experienced evaluators.

FIG. 1 illustrates an example of the configuration of an information processing system. FIG. 1 illustrates an example of the configuration of an information processing device. FIG. 2 is a diagram illustrating an example of the configuration of a terminal device. FIG. 10 is a flowchart illustrating an example of an operation performed by the information processing device. FIG. 10 is a flowchart illustrating an example of an operation performed by the information processing device. FIG. 10 is a flowchart illustrating an example of an operation performed by the information processing device. FIG. 10 is a diagram illustrating an example of information to which the information processing system refers. FIG. 10 is a diagram illustrating an example of information to which the information processing system refers. FIG. 10 is a diagram illustrating processing of a face image. FIG. 10 is a diagram illustrating processing of a face image. FIG. 10 is a diagram illustrating processing of a face image. FIG. 10 is a diagram illustrating processing of a face image. FIG. 10 is a diagram illustrating information on recommended cosmetics.

The following describes an embodiment of the present invention.

Figure 1 is a diagram showing an example configuration of an information processing device in one embodiment. The information processing system 1 supports sales staff in, for example, a cosmetics store in providing beauty treatment counseling to customers. The information processing system 1 includes a terminal device 11 and an information processing device 12. The terminal device 11 and the information processing device 12 are connected to each other peer-to-peer via a network 10, or via wired or short-range wireless communication without using the network 10, and transmit and receive information. The network 10 includes the Internet, at least one wide area network (WAN), at least one metropolitan area network (MAN), or any combination thereof. The network 10 may also include at least one wireless network, at least one optical network, or any combination thereof. The wireless network may be, for example, an ad hoc network, a cellular network, a wireless local area network (LAN), a satellite communication network, or a terrestrial microwave network. The terminal device 11 is an information processing terminal device with an imaging function, such as a tablet terminal or smartphone, used by a cosmetics sales staff. The information processing device 12 is, for example, a computer such as a server belonging to a cloud computing system or other computing system.

The information processing device 12 stores a model (hereinafter referred to as the impression evaluation model) that has been machine-learned in advance to associate impression evaluations with facial image features that contribute to impression evaluations, using training data consisting of facial images obtained by capturing a person's face and impression evaluations of the facial images by experienced evaluators. The impression evaluations are, for example, evaluations of the degree of satisfaction of elements such as friendliness, youthfulness, cuteness, cheerfulness, sociability, and liveliness. The impression evaluation model obtained by machine learning of such training data reflects the impression evaluations by the experienced evaluators. At a sales store, a salesperson employed at the store captures images of customers' faces using a terminal device 11. The information processing device 12 receives input of the customer's facial image from the terminal device 11, determines the feature features of the input facial image and the impression evaluations corresponding to those feature features using the impression evaluation model, and outputs the determined impression evaluations to the terminal device 11. The salesperson then displays or otherwise outputs the impression evaluations and feature features on the terminal device 11 and presents them to the customer. By using an impression evaluation model when evaluating the impression of a customer's facial image, the information processing device 12 can reproduce with high accuracy impression evaluations made by experienced evaluators. Furthermore, by presenting characteristic features that contribute to the impression evaluation, it is possible to improve the customer's sense of satisfaction with the impression evaluation.

In addition, the information processing device 12 further stores recommendation information indicating cosmetics that should be recommended for combinations of characteristic parts associated in the impression evaluation model and the impression evaluation. The recommendation information includes, for example, cosmetics for wrinkle-reducing skin care, cosmetics for eyebrows to tone the eyes, cosmetics for skin care to reduce sagging, effects required to improve impression, and product group information. The information processing device 12 then selects cosmetics that should be recommended for the combination of characteristic parts and impression evaluation determined using the impression evaluation model based on the recommendation information, and further outputs information indicating the selected cosmetics. The salesperson receives the information output by the information processing device 12 on the terminal device 11 and presents it to the customer. This makes it possible to recommend cosmetics to customers that reflect the impression evaluations of experienced evaluators.

Figure 2 shows an example configuration of the information processing device 12. The information processing device 12 is, for example, a server computer that functions as a server that implements various functions. The information processing device 12 may be one or more server computers that are connected to each other so that they can communicate information and operate in cooperation with each other. The information processing device 12 has an input/output unit 20, a communication unit 21, a memory unit 22, and a control unit 23.

The input/output unit 20 has an input interface that detects user input and sends the input information to the control unit 23. The input interface may be any input interface, including, for example, physical keys, capacitive keys, a touch screen integrated with a panel display, or various pointing devices. The input/output unit 20 also has an output interface that outputs information generated by the control unit 23 or read from the memory unit 22 to the user. Such an output interface may be, for example, a display that outputs information as an image or video, a speaker that outputs information as audio, or any output interface including a connection interface with an external output device.

The communication unit 21 has a communication module that supports one or more wired or wireless LAN standards for connecting to the network 10. Alternatively, the communication unit 21 may have a communication module that supports one or more wired or wireless standards for connecting peer-to-peer with the terminal device 11. Such a communication module may include a communication module that supports short-range communication such as Bluetooth, AirDrop, IrDA, ZigBee, FeliCa, RFID, etc.

The storage unit 22 includes one or more memories. Each memory included in the storage unit 22 may be, for example, a semiconductor memory, a magnetic memory, or an optical memory, but is not limited to these. Each memory functions, for example, as a main memory device, an auxiliary memory device, or a cache memory. The storage unit 22 stores any information and control/processing programs used in the operation of the information processing device 12. The storage unit 22 may also store application programs that provide various functions and are downloaded via the network 10 or other networks.

The control unit 23 has one or more processors. Each processor may be, but is not limited to, a general-purpose processor or a dedicated processor specialized for a specific process. The control unit 23 comprehensively controls the operation of the information processing device 12 in accordance with the control and processing programs stored in the memory unit 22.

In the information processing device 12, the memory unit 22 stores an impression evaluation model 24 created by the control unit 23 using the procedure described below. The memory unit 22 also stores recommendation information 25 including information on cosmetics to be recommended for combinations of characteristic parts and impression evaluations associated in the impression evaluation model 24, and standard information 26 regarding the criteria for impression evaluation.

Figure 3 shows an example configuration of the terminal device 11. The terminal device 11 is an information processing terminal device such as a tablet terminal or a smartphone. The terminal device 11 has an input/output unit 30, a communication unit 31, a memory unit 32, and a control unit 33.

The input/output unit 30 has an input interface that detects user input and sends the input information to the control unit 33. Such an input interface may be any input interface including, for example, physical keys, capacitive keys, a touch screen integrated with a panel display, various pointing devices, a microphone that accepts audio input, or a camera that captures captured images or image codes. The input/output unit 30 also has an output interface that outputs information generated by the control unit 33 or read from the memory unit 32 to the user. Such an output interface may be, for example, a display that outputs information as an image or video, a speaker that outputs information as audio, or any output interface including a connection interface with an external output device.

The communication unit 31 includes a communication module compatible with wired or wireless LAN standards, a module compatible with mobile communication standards such as 4G (4th Generation) and 5G (5th Generation), and a communication module compatible with short-range communication such as Bluetooth, AirDrop, IrDA, ZigBee, FeliCa, and RFID. The terminal device 11 communicates information with the information processing device 12 via the network 10 or peer-to-peer using the communication unit 31.

The storage unit 32 includes one or more memories. Each memory included in the storage unit 32 may be, for example, a semiconductor memory, a magnetic memory, or an optical memory, but is not limited to these. Each memory functions, for example, as a main memory device, an auxiliary memory device, or a cache memory. The storage unit 32 stores any information used in the operation of the terminal device 11. For example, the storage unit 32 stores control and processing programs, embedded software, application programs that provide various functions and are downloaded via the network 10 or other networks, etc.

The control unit 33 has one or more processors. Each processor may be, but is not limited to, a general-purpose processor or a dedicated processor specialized for a particular process. The control unit 33 operates in accordance with the control and processing programs stored in the memory unit 32, thereby providing overall control over the operation of the terminal device 11.

In the terminal device 11, the control unit 33 responds to operational input by the salesperson to the input/output unit 30 by causing the camera of the input/output unit 30 to capture an image of the customer's face, and sends the captured facial image to the information processing device 12 via the communication unit 31. The control unit 33 also receives information sent from the information processing device 12 via the communication unit 31 and displays it on the display of the input/output unit 30.

Figure 4 is a flowchart showing an example of the operational procedure of the information processing device 12 when the information processing device 12 creates the impression evaluation model 24 through machine learning. This flowchart corresponds to the information processing procedure by the control unit 23.

In step S400, the control unit 23 reads the training data and stores it in the memory unit 22. The training data includes facial images obtained by capturing images of people's faces with a digital camera and impression evaluations assigned to the facial images by experienced evaluators. Hereinafter, the facial images included in the training data are referred to as training facial images. Training facial images are obtained, for example, by capturing images of the faces of several hundred Japanese women, roughly uniformly distributed between their 20s and 70s, from the front. Each person is photographed wearing a hair net to maintain a uniform hairstyle, without makeup, and with a straight face. The impression evaluation is a discrete value (hereinafter referred to as evaluation value) assigned to each training facial image for each given element. The evaluation value is assigned by the experienced evaluator's visual evaluation of the presented facial image, eliminating the influence of the facial expression, personality, and evaluator's preferences of the person in the training facial image. The elements include, for example, friendliness, youthfulness, cuteness, cheerfulness, sociability, and liveliness. The higher the degree to which each element is satisfied, the better the evaluation. The better the evaluation, the lower the evaluation value, which is assigned as a discrete value on a seven-level scale, for example from 1 to 7. The dynamic range of the discrete values may be set arbitrarily according to the range within which the evaluator can classify. Here, learning of evaluation values using a classification model is used as an example, but a regression model may also be used.

In step S402, the control unit 23 performs image processing on each training face image as a preliminary step to machine learning. Image processing includes, for example, normalization of the training face image, extraction of facial features from the training face image, random cropping, random horizontal and vertical flips, random rotations, random saturation adjustments, vertical dropout, and the like. When extracting facial features, facial features can be extracted using any open source library to eliminate the influence of features such as hair and neck in the training face image, or facial features can be extracted by specifying the vertices of a bounding box by the operator. In addition, in one example of random cropping, a 512 x 512 pixel training face image is resized to 256 x 256 pixels, and then randomly cropped to 224 x 224 pixels.

In step S404, the control unit 23 creates the impression assessment model 24 through machine learning using training data. The control unit 23 performs machine learning using, for example, a convolutional neural network. For example, the control unit 23 extracts feature parts from the training face image using a convolutional layer and a pooling layer, and compares the extracted feature parts with impression assessments using a multi-layer neural network, thereby learning to associate the feature parts that contribute to the impression assessment assigned to the training face image with the impression assessments, and creates the impression assessment model 24.

Figure 5 is a flowchart showing an example of the operational procedure of the information processing device 12 when the information processing device 12 performs an impression evaluation of a customer's facial image using the impression evaluation model 24. This flowchart corresponds to the information processing procedure by the control unit 23.

In step S500, the control unit 23 accepts input of a facial image and customer information. For example, a salesperson at a store uses the terminal device 11 to capture an image of the face of a visiting customer, and inputs the customer's age, gender, and other customer information into the terminal device 11. The terminal device 11 sends the captured image and customer information to the information processing device 12. In the information processing device 12, the control unit 23 accepts the captured image and customer information via the communication unit 21. The facial image and customer information are stored in the memory unit 22. Hereinafter, the facial image that the control unit 23 accepts as input and requests an impression evaluation is referred to as an evaluation facial image.

In step S502, the control unit 23 performs image processing on the evaluation face image as a preliminary step for obtaining an impression evaluation. The image processing includes normalization of the evaluation face image, extraction of facial features from the evaluation face image, random cropping, random flipping in the horizontal and vertical directions, random rotation, random saturation adjustment, vertical dropout, etc.

In step S504, the control unit 23 uses the impression assessment model 24 to determine the impression assessment of the evaluation face image. The control unit 23 determines, as the impression assessment, the class of the assessment value corresponding to the maximum value in the final layer output of the impression assessment model 24. The assessment value is assigned as a discrete value from 1 to 7.

In step S506, the control unit 23 uses the impression evaluation model 24 to determine feature parts that contribute to the impression evaluation of the evaluation face image. The feature parts that contribute to the impression evaluation are represented, for example, as a heat map. The control unit 23 creates a heat map showing the feature parts via a convolutional neural network. The control unit 23 converts the maximum output value in the final output layer into a probabilistic expression using a numerical value between 0 and 1 using an activation function such as a sigmoid function or a softmax function. The control unit 23 calculates the gradient with respect to the maximum output value of the final output layer, i.e., the partial differential value, for each pixel of the feature map, which consists of 512 channels in the final convolutional layer. The control unit 23 then calculates the average value of the gradient for each pixel and creates an activation map for 512 channels by multiplying each pixel of the feature map by this average value. The number of channels in the feature map shown here is an example, and the number of channels may be other than 512. The control unit 23 then integrates the created activation map across all channels and scales the integrated activation factors to values between 0 and 1 to generate a two-dimensional activation map, i.e., a heat map. The control unit 23 then resizes the heat map to the size of the face image for evaluation to create a heat map image corresponding to the face image for evaluation. The control unit 23 may also add the heat map image to the face image for evaluation and scale it with a gradation value between 0 and 255 to generate a display image in which the heat map is superimposed on the face image for evaluation. By superimposing the heat map on the face image for evaluation, characteristic portions in the face image for display are displayed with a predetermined color distribution that is arbitrarily set. Note that when scaling the size of the heat map, the control unit 23 may eliminate outliers using an arbitrarily set threshold value to remove noise from the heat map.

In step S508, the control unit 23 selects recommended cosmetics based on the impression evaluation and characteristic parts. Since a smaller evaluation value corresponds to a better impression evaluation, characteristic parts that contribute to a relatively low evaluation value are considered to indicate parts that contribute to a good impression, and characteristic parts that contribute to a relatively high evaluation value are considered to indicate parts that contribute to a poor impression. Therefore, the control unit 23 identifies parts of the face that contribute to a poor impression based on the impression evaluation and characteristic parts, and selects recommended cosmetics to improve the impression evaluation according to the characteristics of those parts. The control unit 23 then outputs information about the recommended cosmetics along with the impression evaluation and characteristic parts.

Here, a specific example of step S508 will be explained using Figures 6 to 9. Figure 6 is a flowchart showing the subroutine of step S508. Here, using the degree of liveliness as an example of an impression element, the procedure is shown when the impression evaluation for liveliness is expressed as a seven-level evaluation value from 1 to 7.

In step S600, the control unit 23 determines whether the liveliness evaluation value is equal to or greater than a reference value of 3. If the evaluation value is equal to or greater than 3 (Yes), the control unit 23 proceeds to step S602. On the other hand, if the evaluation value is less than 3 (No), the control unit 23 proceeds to step S604 and determines that the liveliness evaluation value belongs to the "sufficient liveliness group." Note that the reference value can be adjusted as appropriate and may be other than 3.

In step S602, the control unit 23 determines whether the liveliness evaluation value is higher than the average for the customer's age group. The control unit 23 compares the customer's age included in the customer information received in step S500 with the judgment criteria included in the reference information 26 stored in the memory unit 22. For example, the reference information 26 stores the average evaluation values for each age group of people who were subjects of the training face images as judgment criteria for impression evaluation, and this is used for comparison with the evaluation value.

Figure 7A shows a schematic representation of reference information 26. It shows the average liveliness evaluation score for each age group from teens to 70s. In addition to liveliness, reference information 26 also includes age-group average evaluation scores for each impression element. Note that the criteria included in reference information 26 may be something other than the average for each age group, such as the median. Furthermore, as shown in the distribution of evaluation scores in the age group direction in Figure 7B, the same evaluation value tends to span multiple age groups. In other words, compared to features that have a strong correlation with a person's age, such as skin age, elements that have a low correlation with age tend to contribute significantly to impression evaluation.

Returning to Figure 6, if the evaluation value is higher than the age average (Yes in step S602), that is, if the impression evaluation is not better than the age average, the control unit 23 proceeds to step S606 and determines that the liveliness evaluation value belongs to the "liveliness 'needs improvement' group." If the evaluation value is lower than the age average (No in step S602), that is, if the impression evaluation is at or better than the age average, the control unit 23 proceeds to step S608 and determines that the liveliness evaluation value belongs to the "liveliness 'slightly lacking' group."

If the assignment of the evaluation value is determined in step S604, S606, or S608, the control unit 23 calculates the feature parts that contributed to the evaluation value and their contribution levels in step S610. For example, the control unit 23 divides the evaluation image into any number of regions. For example, FIG. 8A shows an example of four regions in the evaluation face image 80: a forehead region 80A, an eye region 80B, a cheek region 80C, and a mouth region 80D. For each region, the control unit 23 calculates the sum of the activity elements in the heat map for that region (e.g., the number of pixels) as the feature activity level for that region. Here, a feature activity value ranging from 0 to 1.0 is calculated as the contribution level. The control unit 23 then generates a display image that displays the feature parts as color-coded regions in the heat map and the contribution levels as the area of the color-coded regions, and outputs the display image. The display image is sent to the terminal device 11 by the communication unit 21 and displayed on the terminal device 11.

Figures 8B to 8D are schematic examples of display images generated by the control unit 23 of the information processing device 12 and displayed on the terminal device 11. Figures 8B, 8C, and 8D show examples for liveliness evaluation values of "2," "3," and "6," respectively. Here, the average evaluation value for each age group is assumed to be between "3" and "6." The example in Figure 8B corresponds to the "sufficient liveliness group," the example in Figure 8C corresponds to the "slightly insufficient liveliness group," and the example in Figure 8D corresponds to the "needs improvement liveliness group." Figure 8B shows a wide range of characteristic features 82 in the cheek region 80C of the evaluation face image 80. This indicates that the features in the cheek region 80C contribute significantly to a good liveliness (evaluation value "2"). Furthermore, Figure 8C shows characteristic features 83 in the forehead region 80A and mouth region 80D of the evaluation face image 80. That is, it is shown that the feature portions 83 in the forehead region 80A and mouth region 80D make a high contribution to a moderate liveliness (evaluation value "3"). Furthermore, in FIG. 8D, feature portions 84 are shown in the forehead region 80A, cheek region 80C, and mouth region 80D in the evaluation face image 80. That is, it is shown that the feature portions 84 in the forehead region 80A, cheek region 80C, and mouth region 80D make a high contribution to a poor liveliness (evaluation value "6").

Returning to Figure 6, in steps S612, S614 and S616, the control unit 23 identifies areas of the face that contribute to a poor impression based on the evaluation, characteristic parts, and the contribution of the characteristic parts, and selects recommended cosmetics to improve the impression evaluation based on the characteristics of those parts. If the evaluation value of the liveliness belongs to the "sufficient liveliness group" (step S604), the control unit 23 determines in step S612 the region with the highest contribution of the characteristic part as the "good region." For example, in the example shown in FIG. 8B, the cheek region 80C is determined to be the good region. On the other hand, if the evaluation value of the liveliness belongs to the "slightly insufficient liveliness group" (step S608), the control unit 23 determines in step S614 the region with a high contribution of the characteristic part as the "good region," and the region with a low contribution of the characteristic part as the "slightly insufficient region." For example, in the example shown in FIG. 8C, the mouth region 80D is determined to be the "good region," and the forehead regions 80B and 80C are determined to be the "slightly insufficient regions." ". For example, the control unit 23 determines whether the area is a "good area" or a "slightly deficient area" depending on the feature activity of the feature part. For example, a feature activity of 0 to 0.3 can be determined as a "slightly deficient area," and a feature activity of 0.7 to 1.0 can be determined as a "good area." There may be three or more types of determinations depending on feature activity, and the numerical range used as the basis for determination can be set arbitrarily. Then, if the liveliness evaluation value belongs to the "liveliness 'improvement required' group" (step S606), the control unit 23 determines in step S616 the area with the highest contribution of the feature part as an "area requiring improvement." For example, in the example shown in FIG. 8D, the forehead area 80A is determined to be an area requiring improvement.

In step S618, the control unit 23 selects recommended cosmetics for the areas requiring improvement and the areas that are slightly lacking by referring to the recommendation information 25, and displays information about the selected recommended cosmetics.

Figure 9 schematically shows an example of recommendation information 25, including information on recommended cosmetics for improving liveliness. The recommendation information 25 has information for each impression element, as shown in Figure 9. The control unit 23 references the recommendation information 25 to select recommended cosmetics for a "slightly deficient area" (evaluation value of 3.0 or more and less than the age average) or an "area requiring improvement" (evaluation value of the age average or more and 7.0 or less). For example, if the forehead area 80A is determined to be a "slightly deficient area," cosmetics for skin care to improve wrinkles between the eyebrows are selected for skin care for the forehead area 80A. Furthermore, if the forehead area 80A is determined to be an "area requiring improvement," cosmetics for highlighting the temples are selected for makeup for the forehead area 80A. Furthermore, if the eye area 80B is determined to be a "slightly deficient area," cosmetics for eye makeup that emphasizes the eyes are selected for makeup for the eye area 80B. Furthermore, if the eye area 80B is determined to be an "area requiring improvement," eyebrow cosmetics to shape the eyes are selected for makeup application to the eye area 80B. Furthermore, if the cheek area 80C is determined to be an "area where there is a slight shortage," cheek color cosmetics are selected for makeup application to the cheek area 80C. Furthermore, if the cheek area 80C is determined to be an "area where there is a slight shortage," cheek skin care cosmetics to improve sagging are selected for skin care of the cheek area 80C. Furthermore, if the mouth area 80D is determined to be an "area where there is a slight shortage," skin care cosmetics to improve sagging around the lips are selected for skin care of the mouth area 80D. Furthermore, if the mouth area 80D is determined to be an "area where there is a shortage," skin care cosmetics to improve sagging around the chin are selected for skin care of the mouth area 80D. Information on the cosmetics selected in this manner is sent to the terminal device 11 by the communication unit 21 and displayed on the terminal device 11.

For example, the terminal device 11 displays a lively feeling assessment result, such as sufficient, slightly lacking, or requiring improvement, along with a display image showing the characteristic parts that contributed to the assessment result, and further displays information about recommended cosmetics for skin care or makeup. The information about recommended cosmetics may be the type of cosmetics or the specific name of the cosmetics.

Figures 6 to 9 have been used to explain the impression evaluation of liveliness as an example, but the same procedure as above can also be performed for other impression elements, such as friendliness, youthfulness, cuteness, cheerfulness, and sociability.

According to this embodiment, the objective impression evaluation of an experienced evaluator can be reproduced with high accuracy even for non-quantitative impression elements. Furthermore, the characteristic parts that contributed to the impression evaluation can be visualized by display, making it possible to improve the customer's sense of satisfaction with the impression evaluation. Furthermore, the information processing device 12 that reproduces the impression evaluation of an experienced evaluator can be used to train inexperienced evaluators in impression evaluation.

In the above description, the information processing device 12 acquires facial images and displays various output information via the terminal device 11. However, the information processing procedures shown as the operation of the information processing device 12 may be shared between the information processing device 12 and the terminal device 11 as appropriate. In this case, the information processing device 12 and the terminal device 11 constitute an "information processing device." For example, the information processing device 12 may execute the machine learning procedure shown in FIG. 4 to create the impression evaluation model 24, and the terminal device 11 may execute the procedure shown in FIG. 5 using the impression evaluation model 24 of the information processing device 12. Alternatively, for example, the information processing device 12 may operate standalone and acquire facial images and display various output information using the input/output unit 20. Furthermore, the terminal device 11 does not have to be installed in a store; for example, it may be a smartphone used by a customer. In this case, the customer captures an image of their face and sends the facial image to the information processing device 12. The smartphone then receives information from the information processing device 12 and displays it to the customer. This allows, for example, the customer to obtain information about recommended cosmetics without visiting a store.

In addition, in the above explanation, the information processing device 12 generates the impression evaluation model 24 once and then performs an impression evaluation on the customer. However, with the customer's consent, the evaluation captured images can be imported into the training data, and an experienced evaluator can provide an impression evaluation using the information processing device 12 or a terminal device capable of communicating with the information processing device 12, thereby updating the training data at any time.

While the present invention has been described based on various drawings and examples, it should be noted that those skilled in the art would easily be able to make various modifications and alterations based on this disclosure. Therefore, it should be noted that these modifications and alterations are included within the scope of the present invention. For example, the functions included in each means, step, etc. may be rearranged so as not to result in logical inconsistencies, and multiple means, steps, etc. may be combined into one or divided into separate means, steps, etc.

REFERENCE SIGNS LIST 1 Information processing system 10 Network 11 Terminal device 12 Information processing device 20, 30 Input/output unit 21, 31 Communication unit 22, 32 Storage unit 23, 33 Control unit

Claims (7)

a storage unit for storing a model obtained by machine learning a correspondence between a facial image of a person and an impression evaluation of the facial image, the characteristic portion of the facial image contributing to the impression evaluation, and the impression evaluation, using the facial image and the impression evaluation as training data;
a control unit that uses the model to determine characteristic parts of an input face image and impression evaluations corresponding to the characteristic parts in accordance with the proportions of the characteristic parts for each area included in the face image , and outputs information on the impression evaluations and an image showing the characteristic parts contributing to the impression evaluation in a manner corresponding to the impression evaluations together with the input captured image;
An information processing device having the above. In claim 1,
the storage unit further stores recommendation information indicating a cosmetic product to be recommended for a combination of the characteristic portion associated in the model and the impression evaluation;
the control unit selects, based on the recommendation information, a cosmetic product to be recommended for the combination of the characteristic portion and the impression evaluation obtained using the model, and further outputs information indicating the selected cosmetic product.
Information processing device. In claim 1 or 2,
In the training data, the same impression evaluation is associated with face images of a plurality of people of different ages.
Information processing device. When executed by a computer, the computer is caused to operate as the information processing device according to any one of claims 1 to 3.
program. A method for operating an information processing device, comprising:
a first step of using a person's face image and an impression evaluation of the face image as training data to machine-learn the correspondence between the characteristic parts of the face image that contribute to the impression evaluation and the impression evaluation, and creating a model;
a second step of using the model to determine characteristic parts of an input face image and impression evaluations corresponding to the characteristic parts in accordance with the proportions of the characteristic parts in each area included in the face image , and outputting information on the impression evaluations and an image showing the characteristic parts contributing to the impression evaluation in a manner corresponding to the impression evaluations together with the input captured image;
A method for operating an information processing device having the above-mentioned components. In claim 5,
a third step of referring to recommendation information indicating cosmetics to be recommended for the combination of the feature portion and the impression evaluation associated in the model, and selecting cosmetics to be recommended for the combination of the feature portion and the impression evaluation obtained using the model;
a fourth step of further outputting information indicating the selected cosmetic product;
A method for operating an information processing device further comprising: In claim 5 or 6,
In the training data, the same impression evaluation is associated with face images of a plurality of people of different ages.
A method for operating an information processing device.