JP7817125B2 - Information processing device, information processing method, and information processing program - Google Patents

Description

The present invention relates to an information processing device, an information processing method, and an information processing program.

Technology for compositing a user's facial image with a hair image and presenting the result to the user is known. For example, Patent Document 1 proposes a technology for performing image processing using hair simulation to change the hairstyle and hair color, and displaying the results of this image processing.

JP 2013-178789 A

However, the above-mentioned conventional technology is limited to image processing using hair simulation, and does not allow users to know the treatment details required to achieve the selected hairstyle.

The present application has been made in consideration of the above, and aims to provide an information processing device, information processing method, and information processing program that can estimate the treatment details required to achieve a hairstyle selected by the user.

The information processing device according to the present application includes an image processing unit and an estimation unit. The image processing unit generates a composite image by combining a user's facial image, which is an image of the user's face, with a hair image selected by the user. The estimation unit estimates the treatment details from the image of the user before the treatment and the composite image, using a learning model that has learned the relationship between the hair images before and after the treatment and the treatment details.

According to one aspect of the embodiment, it is possible to estimate the treatment details required to achieve the hairstyle selected by the user.

FIG. 1 is a diagram illustrating an example of information processing according to an embodiment. FIG. 2 is a diagram illustrating an example of the configuration of an information processing system according to the embodiment. FIG. 3 is a diagram illustrating an example of the configuration of an information providing device according to an embodiment. FIG. 4 is a diagram illustrating an example of a user information table stored in the user information storage unit of the information providing device according to the embodiment. FIG. 5 is a diagram illustrating an example of a posted information table stored in the posted information storage unit of the information providing device according to the embodiment. FIG. 6 is a diagram illustrating an example of a model information table stored in the model information storage unit of the information providing device according to the embodiment. FIG. 7 is a diagram illustrating an example of a content table stored in the content storage unit of the information providing device according to the embodiment. FIG. 8 is a diagram illustrating an example of the configuration of a terminal device according to the embodiment. FIG. 9 is a diagram illustrating an example of the configuration of the reception unit of the terminal device according to the embodiment. FIG. 10 is a diagram illustrating an example of the configuration of an image processing unit of the terminal device according to the embodiment. FIG. 11 is a diagram illustrating an example of a reservation setting screen displayed on the display unit by the output unit of the information providing device according to the embodiment. FIG. 12 is a diagram illustrating an example of a keyword specification screen displayed on the display unit by the output unit of the information providing device according to the embodiment. FIG. 13 is a diagram for explaining a process of acquiring a similar hair image from a specified keyword by the processing unit of the information providing device according to the embodiment. FIG. 14 is a diagram showing another example of the reservation setting screen displayed on the display unit by the output unit of the information providing device according to the embodiment. FIG. 15 is a diagram showing yet another example of a reservation setting screen displayed on the display unit by the output unit of the information providing device according to the embodiment. FIG. 16 is a diagram illustrating an example of a composite image change screen displayed on the display unit by the output unit of the information providing device according to the embodiment. FIG. 17 is a diagram illustrating a process of estimating a treatment content from a user image and a composite image by an estimation unit of an information providing device according to an embodiment. FIG. 18 is a diagram illustrating an example of a face shape selection screen displayed on the display unit by the output unit of the information providing device according to the embodiment. FIG. 19 is a flowchart illustrating an example of information processing by the processing unit of the terminal device according to the embodiment. FIG. 20 is a flowchart illustrating an example of a reservation process by the processing unit of the terminal device according to the embodiment. FIG. 21 is a hardware configuration diagram illustrating an example of a computer that realizes the functions of the information providing device and the terminal device according to the embodiment.

Below, detailed explanations will be given of the information processing device, information processing method, and information processing program (hereinafter referred to as "embodiments") according to the present application, with reference to the drawings. Note that the information processing device, information processing method, and information processing program according to the present application are not limited to these embodiments. Furthermore, the embodiments can be combined as appropriate to the extent that the processing content is not contradictory. Furthermore, the same components in the following embodiments will be assigned the same reference numerals, and redundant explanations will be omitted.

[1. An example of information processing]
First, an example of information processing according to the embodiment will be described with reference to Fig. 1. Fig. 1 is a diagram showing an example of information processing according to the embodiment. Note that, although the following description will be given of a case where the information processing device according to the embodiment is the terminal device 4 shown in Fig. 1, the information processing device may be the information providing device 1 shown in Fig. 1, or may be a configuration in which the terminal device 4 and the information providing device 1 are combined.

The information providing device 1 according to the embodiment shown in FIG. 1 is an information processing device that provides services such as a social networking service (SNS). For example, the information providing device 1 accepts posted information posted by a poster. The information providing device 1 distributes the posted information accepted from the poster to a terminal device 4 of a user U who will be a viewer.

The poster may be, for example, a hair salon, a hair practitioner, or a person receiving a treatment, and may include user U. A hair salon is a beauty salon (beauty parlor) or a barber shop (hairdressing parlor). A hair practitioner is a practitioner who styles hair at a hair salon, and is also called a hair stylist. A person receiving a treatment is a user who has received a hair treatment by a hair practitioner.

The posted information posted by the poster includes, for example, a posted image, which is a poster-captured image obtained by capturing an image of the poster, and information indicating posted text, which is text posted by the poster. The posted image includes, for example, a captured image of the poster's head, including the poster's hair and face.

In addition, the posted text may include, for example, the poster's comments on the posted image. The poster's comments on the posted image may include, for example, evaluation information indicating the poster's evaluation of the posted image. Hereinafter, the poster's comments on the posted image may be referred to as the poster's comments.

The information providing device 1 also accepts evaluation information indicating the evaluations of other users (an example of a viewer) of the posted images included in the posted information, as well as comments from other users on the posted information. The information providing device 1 then distributes the posted information, including the received evaluation information and comments from other users, to the viewer's device. The viewer's device is, for example, the terminal device 4 of user U or a device of another user. Hereinafter, comments from other users may be referred to as viewer comments.

Evaluations of posted information may be, for example, positive evaluations (e.g., "good") or negative evaluations (e.g., "bad"), but may also include neutral evaluations (e.g., "average"). The information providing device 1 can also distribute information such as the number of times the posted information has been added to a timeline by other users as evaluations of the posted information to the terminal device 4 of the user U who will be viewing the information.

The information providing device 1 transmits information on multiple types of learning models to the terminal device 4 (step S1). The multiple types of learning models include a first learning model, a second learning model, a third learning model, and a fourth learning model.

First, we will explain the first learning model. The first learning model is a model that learns the relationship between keywords and hair images, and is a learning model that takes one or more keywords as input and hair images as output. The first learning model is generated by machine learning using learning data that includes multiple combinations of keywords and hair images.

The first learning model may be a learning model that inputs one or more keywords and outputs hair image identification information. The hair image identification information is information that identifies a hair image.

The information providing device 1 extracts, for example, one or more keywords contained in the poster's comments or text, and generates data containing multiple combinations of the one or more keywords and the posted image that was the subject of the comment as the learning data described above.

Furthermore, if the information providing device 1 is unable to extract keywords from the poster's comments or poster's text, it extracts one or more keywords contained in the viewer's comments and generates data containing multiple combinations of the one or more keywords and the posted image that was the subject of the comment as the learning data described above.

The information providing device 1 can also extract one or more keywords included in both the poster's comments or poster's text and the viewer's comments, and generate data containing multiple combinations of the one or more keywords and the posted images that were the subject of the comments as the learning data described above.

The learning data may also be data containing multiple combinations of posted images and keywords assigned by the worker to each posted image. Furthermore, instead of content such as posted information, the learning data may be images included in content such as web pages, or images provided by a third party. Posted information is an example of posted content.

For example, the information providing device 1 can acquire content containing text and images of people's faces and hair via a network, and extract keywords from the text contained in the content. In this case, the information providing device 1 generates data as learning data that contains multiple combinations of keywords extracted from the text contained in the content and images contained in the content.

Next, we will explain the second learning model. The second learning model is a learning model that learns the relationship between facial images and information indicating facial features, and takes images containing facial images as input and outputs information indicating facial features.

The second learning model is generated by machine learning using learning data that includes multiple combinations of images containing facial images and information indicating facial features. The information indicating facial features is, for example, information indicating the type of face shape, but it may also be information indicating the characteristics of each part of the face, such as the size, shape, and position of one or more of the eyebrows, eyes, nose, mouth, and ears.

Furthermore, the information indicating facial features may be information indicating the positions of the eyebrows, eyes, nose, mouth, and ears on the face, or information indicating the positions and sizes of the eyebrows, eyes, nose, mouth, and ears on the face.

Next, we will explain the third learning model. The third learning model is a model that learns the relationship between information indicating facial features and hairstyles that are evaluated as suiting that face. It is a learning model that takes information indicating facial features as input and outputs information indicating hairstyles that are estimated to suit that face.

The third learning model is generated by machine learning using learning data that includes multiple combinations of information indicating facial features, hair images, and information indicating evaluations of combinations of information indicating facial features and hair images.

The information providing device 1 collects multiple pieces of posted information, each of which includes a posted image, viewer comments, and rating information. The facial images include a facial image and a hair image, and the rating information indicates the ratings of other users who are viewers of the posted image, but may also include information indicating ratings included in the poster's comments.

The information providing device 1 extracts facial features indicated by facial images included in each piece of collected posted information. The information providing device 1 then generates data including hair images included in the posted images of the posted information, information indicating the extracted facial features, and evaluation information as the learning data described above.

In the third learning model, the information indicating facial features is, for example, information indicating the type of face shape, but it may also be information indicating the features of each part of the face, such as the size, shape, and position of one or more of the eyebrows, eyes, nose, mouth, and ears. Furthermore, the information indicating facial features may also be information indicating the positions of the eyebrows, eyes, nose, mouth, and ears, or information indicating the positions and sizes of the eyebrows, eyes, nose, mouth, and ears.

Next, we will explain the fourth learning model. The fourth learning model is a learning model that learns the relationship between hair images before and after a treatment and the treatment details. It is a learning model that inputs hair images before and after a treatment and outputs information indicating the estimated treatment details. The hair images before and after a treatment include hair images before the hair treatment and hair images after the hair treatment.

The details of the treatment include, for example, whether or not to cut, the amount of cut, whether or not to bleach, the number of times to bleach, whether or not to color, the type of color, whether or not to perm, the type of perm, whether or not to straighten the hair, etc. Information indicating the details of the treatment includes score information for each of whether or not to cut, the amount of cut, whether or not to bleach, the number of times to bleach, whether or not to color, the type of color, whether or not to perm, the type of perm, and whether or not to straighten the hair.

In addition, the information indicating the treatment details may include score information for each combination of whether or not a cut was performed, the amount of cut, whether or not bleaching was performed, the number of times bleaching was performed, whether or not coloring was performed, the type of coloring, whether or not perming was performed, the type of perming, and whether or not hair straightening was performed.

The fourth learning model is generated by machine learning using learning data that includes multiple combinations of hair images before and after a treatment and information indicating the treatment details. The information providing device 1 collects treatment information that includes hair images before and after a treatment and information indicating the treatment details, for example, from devices such as a hair salon or hair stylist. Then, based on the collected treatment information, the information providing device 1 generates learning data that includes multiple combinations of hair images before and after a treatment and information indicating the treatment details.

Any type of model can be used for each of the above-mentioned learning models. For example, each learning model is a model generated by machine learning, such as a deep neural network (DNN), a gradient boosting decision tree (GBDT), or a support vector machine (SVM). A DNN is, for example, a convolutional neural network (CNN) or a recurrent neural network (RNN). An RNN may also be a long short-term memory (LSTM). Each learning model may also be a model realized by combining multiple models, such as a model combining a CNN and an RNN.

Next, the terminal device 4 receives information on multiple types of learning models transmitted from the information providing device 1 and stores the received information on multiple types of learning models in an internal memory unit (step S2).

A user U of terminal device 4 may consider what hairstyle they want before making a reservation for a hair treatment at a hair salon. They may consider hairstyles from, for example, information found through a web search or information published in a magazine.

For example, user U can display hairstyle-related images obtained by keyword search or web search on terminal device 4. Hairstyle-related images are images that allow user U to check hairstyles and include at least a hair image and a facial image.

Now, we will explain keyword search. The terminal device 4 displays multiple keywords on the display unit (step S3). These multiple keywords are predetermined keywords, but they may also be multiple keywords randomly extracted from the multiple keywords or multiple keywords that satisfy predetermined conditions.

The multiple keywords that satisfy the predetermined conditions are, for example, a predetermined number of keywords that are ranked highest in order of frequency of selection by user U. Keywords that are candidates for extraction are, for example, keywords that are to be input into the first learning model.

When one or more keywords are specified by the user U, the terminal device 4 accepts the one or more keywords specified by the user U and identifies hair images corresponding to the one or more keywords using the first learning model (step S4).

In step S4, the terminal device 4 inputs one or more keywords specified by the user U into the first learning model and, based on the information output from the first learning model, identifies hair images corresponding to the one or more keywords specified by the user U.

For example, when the first learning model outputs a hair image, the terminal device 4 identifies the hair image output from the first learning model as a hair image corresponding to one or more keywords specified by the user U.

Furthermore, when the first learning model outputs hair image identification information, the terminal device 4 acquires the hair image identification information output from the first learning model and acquires the hair image identified by the acquired hair image identification information from the internal storage unit or the information providing device 1. In this case, too, the terminal device 4 identifies the acquired hair image as a hair image corresponding to one or more keywords designated by the user U.

Next, the terminal device 4 acquires multiple similar hair images, which are images that include hair images similar to the hair image identified in step S4, and displays the acquired similar hair images on the display unit (step S5). In step S5, the terminal device 4 sends a similar image search request including the hair image identified in step S4 to a similar image search server or information providing device 1 (not shown).

The similar image search server or information providing device 1 searches for multiple similar hair images based on the hair image included in the similar image search request from the terminal device 4, and transmits one or more similar hair images obtained from the search to the terminal device 4. The terminal device 4 acquires the multiple similar hair images transmitted from the similar image search server or information providing device 1 in response to the similar image search request.

In this way, the terminal device 4 displays multiple similar hair images on the display unit based on one or more keywords specified by the user U. This allows the user U to check the hair images they are considering simply by specifying a keyword.

In the above example, the user U specifies one or more keywords from among the multiple keywords presented by the terminal device 4, but the specification of one or more keywords by the user U is not limited to this example. For example, the user U can specify one or more keywords by operating the terminal device 4 to input one or more keywords into the terminal device 4, or can specify one or more keywords by inputting one or more keywords into the terminal device 4 by voice.

In addition to or instead of a keyword search, user U can also consider hairstyles based on user U's facial features. Below, we will explain how to consider hairstyles based on user U's facial features.

The terminal device 4 determines the facial features of the user U based on the user-captured image specified by the user U (step S6). The user-captured image is an image of the head, including the user U's hair and face, captured by an imaging unit (not shown) of the terminal device 4 or an imaging unit of an unillustrated device, and is an example of a user image. The terminal device 4 acquires the user-captured image based on, for example, an operation by the user U.

For example, user U can operate terminal device 4 to capture an image of user U's head, including their hair and face, using the imaging unit of terminal device 4, and have terminal device 4 acquire multiple user-captured images. User U can operate terminal device 4 to specify a desired user-captured image from the multiple user-captured images acquired by terminal device 4.

In step S6, the terminal device 4 determines the facial features of user U shown in the user-captured image, for example, from among multiple predetermined types of facial features. For example, the terminal device 4 inputs the user-captured image into the second learning model described above, and determines the facial features of user U based on the information indicating the facial features output from the second learning model.

In addition, the terminal device 4 may have information indicating each of multiple types of facial features. In this case, the terminal device 4 determines the facial features of the user U using pattern matching technology between the information indicating each type of facial feature and the information indicating the facial features of the user U shown in the user-captured image.

The terminal device 4 can also display facial feature line drawings, which are line drawings showing multiple types of facial features (e.g., face shapes), on the display unit, and allow the user U to select the facial feature line drawing that most closely matches their own face shape from among these multiple types of facial feature line drawings.

Next, the terminal device 4 uses the third learning model described above to identify a hairstyle that is estimated to suit the face of the user U, based on the facial features of the user U determined in step S6 (step S7). As described above, the third learning model is a model that learns the relationship between information indicating facial features and hairstyles that are evaluated as suiting that face.

In step S7, the terminal device 4 inputs information indicating the facial features of user U determined in step S6 into a third learning model, and identifies a hairstyle that is estimated to suit user U's face based on the hairstyle information output from the third learning model. The hairstyle information is a hair image, and the terminal device 4 identifies the hair image output from the third learning model as the hairstyle that is estimated to suit user U's face.

Next, the terminal device 4 generates a composite image by combining the user face image, which is a facial image of the user U, with the hair image selected by the user U, and displays the generated composite image on the display unit (step S8). The user face image is, for example, the above-mentioned user-captured image or an image included in the user-captured image.

The hair image selected by user U is, for example, one or more images selected by user U from the images of the hairstyle identified in step S8 or multiple similar hair images obtained by keyword search.

The terminal device 4 accepts the user U's selection of one or more hair images from among the specific hair image, which is an image of the hairstyle identified in step S8, or from among multiple similar hair images obtained by keyword search. The terminal device 4 can also accept the user U's selection of one or more hair images from among multiple hair images included in multiple web pages viewed by the user U, for example.

When two or more hair images are selected by the user U, the terminal device 4 generates a composite image by combining the two or more hair images to create a composite hair image with the user's facial image.

The hair image selected by user U may be, for example, a captured image taken by an imaging unit (not shown) in terminal device 4. Such a captured image may be, for example, a photograph of a person's face and hair featured in a magazine. In this case, terminal device 4 generates a composite image by, for example, replacing the facial image of the person included in the captured image selected by user U from the captured image taken by terminal device 4 with the facial image of user U included in the user facial image.

Furthermore, in captured images taken from photographs published in magazines, the faces of people included in the captured images may not be facing forward. In this case, the terminal device 4 generates a composite image by projectively transforming the captured image so that the faces of the people are facing forward, and replacing the facial images included in the captured image with the user's facial image. Furthermore, when multiple captured images are selected by the user U, the terminal device 4 generates a composite image by replacing the facial images included in the composite captured image obtained by combining the multiple captured images with the user's facial image.

Furthermore, when the hair image selected by the user U from among multiple hair images is changed based on the user U's operation, the terminal device 4 generates a modified composite image that is a composite image combined with the hair image whose selection was changed by the user U's operation, and displays the generated modified composite image on the display unit.

For example, each time a hair image selected by user U from among multiple hair images is changed based on a first change operation by user U, terminal device 4 generates a corresponding changed composite image and displays the generated changed composite image on the display unit. The first change operation is, for example, a scroll operation in the left/right direction or a swipe operation in the left/right direction.

The terminal device 4 can also change at least one of the hair length and color of the hair image included in the composite image based on a second change operation by the user U. For example, the terminal device 4 changes at least one of the hair length and color of the hair image included in the composite image each time the second change operation is performed. The first change operation is, for example, a vertical scroll operation or a vertical swipe operation.

Furthermore, when the terminal device 4 receives hair image editing selected by the user U, it can also generate an edited hair image by applying the received editing to the hair image. Editing the hair image is, for example, editing to change at least one of the hair length and color of the hair in the hair image. In this case, in step S8, the terminal device 4 generates a composite image by combining the user's face image with the edited hair image, and displays the generated composite image on the display unit.

Next, the terminal device 4 uses the fourth learning model described above to estimate the treatment details from the pre-treatment image of the user U and the composite image generated in step S8 (step S9). The pre-treatment image of the user U is, for example, the user-captured image described above. The treatment details estimated in step S9 are the treatment details required to achieve the hairstyle shown in the composite image generated in step S8, and may be referred to below as estimated treatment details.

In step S9, when the terminal device 4 inputs the pre-treatment image of user U and the composite image generated in step S8 into the fourth learning model, it determines the estimated treatment content based on the information indicating the treatment content output from the fourth learning model.

The terminal device 4 also determines the treatment cost, which is the cost required for the treatment based on the estimated treatment content determined (step S10). For example, the terminal device 4 can determine the treatment cost for each hair salon based on treatment cost information for each hair salon that indicates the relationship between the treatment content and the treatment cost. The terminal device 4 can obtain treatment cost information for each hair salon from, for example, the information providing device 1.

Furthermore, when the terminal device 4 receives a treatment specification from the user U, it compares the user-specified treatment specification, which is the accepted treatment specification, with the estimated treatment specification and generates comparison information indicating the comparison results (step S11). The comparison information includes, for example, information indicating the difference between the user-specified treatment specification and the estimated treatment specification. For example, if the estimated treatment specification includes the user-specified treatment specification plus one more bleaching, the terminal device 4 generates comparison information including information indicating that the number of bleachings is one more than the user-specified treatment specification.

Next, the terminal device 4 displays treatment-related information on the display unit, including information indicating the treatment details estimated in step S9, treatment cost information which is information on the treatment costs determined in step S10, and comparison information generated in step S11 (step S12). This allows the user U to understand the treatment details and treatment costs estimated to be necessary to achieve the hairstyle shown in the selected hair image, and the terminal device 4 can improve convenience for the user U.

Next, the terminal device 4 receives an acceptance/rejection response indicating whether the user U accepts or rejects the treatment content estimated in step S9 (step S13). If the treatment content estimated in step S9 is acceptable, the user U operates the terminal device 4 to respond indicating acceptance; if the treatment content estimated in step S9 is not acceptable, the user U operates the terminal device 4 to respond indicating rejection.

Next, if the received permission/denial response indicates permission, the terminal device 4 outputs a reservation request including a specific image and treatment content information to the hair salon professional (step S14). The specific image includes, for example, the composite image generated in step S8 or a hair image selected by the user U, and a user-captured image including a hair image of the user U before the treatment. The treatment content information includes at least one of information indicating the treatment content specified and accepted by the user U and information indicating the treatment content estimated in step S9.

The above-mentioned reservation request is output to the hair practitioner's store device 5 via the information providing device 1, for example, but may also be output to the hair practitioner's store device 5 via SNS or a mail server (not shown). The hair practitioner's store device 5 receives the reservation request output from the terminal device 4 and displays the specific image and treatment content information included in the received reservation request. This allows the hair practitioner to check the specific image and treatment content information included in the reservation request.

The hairdresser can operate the in-store device 5 to modify the specific image included in the reservation request and the treatment details indicated in the treatment detail information. In this case, the in-store device 5 outputs modification information to the user U, including a modified specific image, which is the specific image after modification, and modified treatment detail information, which is information indicating the treatment details after modification. The modification information is output to the terminal device 4, for example, via the information providing device 1, but may also be output to the terminal device 4 via SNS or a mail server (not shown).

The hairdresser can operate the store device 5 to input a comment into the store device 5 regarding whether or not the hairdresser can perform the hairstyle shown in the composite image included in the reservation request. In this case, the store device 5 outputs a hairdresser comment, which is a comment regarding whether or not the hairdresser can perform the hairstyle shown in the composite image, to the user U. The comment is output to the terminal device 4, for example, via the information providing device 1, but may also be output to the terminal device 4 via SNS or a mail server (not shown).

Furthermore, if there is no need to modify the specific image included in the reservation request or the treatment details indicated in the treatment details information included in the reservation request, the hairdresser operates the store device 5 to cause the store device 5 to output reservation acceptance information to user U indicating that the reservation has been accepted.

When the terminal device 4 acquires the correction information and practitioner comments from the in-store device 5, it displays the acquired correction information and practitioner comments on the display unit (step S15). This allows the user U to check the correction suggestions and comments made by the hair practitioner, facilitating communication with the hair practitioner.

As described above, terminal device 4, which is an example of an information processing device, generates a composite image by combining a user face image, which is a facial image of user U, with a hair image selected by user U, and outputs the generated composite image to the hairdresser. This allows user U to present the composite image, which combines the desired hair image with the user face image, to the hairdresser, and allows user U to specifically communicate the desired hairstyle to the hairdresser, compared to vaguely communicating the desired hairstyle to the hairdresser. Therefore, terminal device 4 can improve convenience for user U.

Furthermore, terminal device 4, which is an example of an information processing device, generates a composite image by combining a user face image, which is a facial image of user U, with a hair image selected by user U, and estimates the treatment content from the image of user U before the treatment and the composite image using a learning model that has learned the relationship between the hair images before and after the treatment and the treatment content. This allows user U to, for example, understand the treatment content to achieve the desired hairstyle, and can specifically communicate the treatment content to the hairdresser. Therefore, terminal device 4 can improve convenience for user U.

Furthermore, terminal device 4, which is an example of an information processing device, accepts the designation of one or more keywords by user U, and identifies a hair image from the one or more keywords designated by the reception unit using a learning model that inputs the one or more keywords and outputs a hair image. Terminal device 4 then acquires similar hair images, which are images that include hair images similar to the identified hair image. This allows user U to easily collect hair images that correspond to a desired hairstyle simply by designating a keyword. Therefore, terminal device 4 can improve convenience for user U.

Furthermore, terminal device 4, which is an example of an information processing device, identifies a hairstyle that is estimated to suit user U's face using a learning model that has learned the relationship between information indicating facial features and hairstyles that are evaluated as suiting the face, and acquires posted images that include a specific hair image that is an image of the identified hairstyle. Then, based on the posted image acquired by the acquisition unit and the facial image of user U, terminal device 4 generates a composite image that combines the image of the hairstyle identified by the identification unit with the facial image of user U. This allows user U to easily acquire an image of themselves wearing the hairstyle that is estimated to suit user U's face. Therefore, terminal device 4 can improve convenience for user U.

In addition, while the above example describes an example in which various processes are performed by the terminal device 4, some or all of the above-mentioned processes of the terminal device 4 may be performed primarily by the information providing device 1. For example, the information providing device 1 may provide information to the terminal device 4 generated based on information input from the terminal device 4 via an interface such as an API (Application Programming Interface), and the terminal device 4 may display the above-mentioned information based on the information provided by the information providing device 1. Also, while FIG. 1 shows a case in which the terminal device 4 and the information providing device 1 are separate devices, the terminal device 4 and the information providing device 1 may be integrated as an information processing device.

The following provides a detailed explanation of the configuration of the information processing system that includes the information providing device 1, terminal device 4, and store device 5 that perform this processing.

2. Configuration of the information processing system
2 is a diagram illustrating an example of a configuration of an information processing system 100 according to an embodiment. As illustrated in FIG. 2, the information processing system 100 according to the embodiment includes an information providing device 1, a plurality of terminal devices 4, and a plurality of in-store devices 5.

Each of the multiple terminal devices 4 is used by a different user U. Each of the multiple in-store devices 5 is installed, for example, in a different hair salon and used by a hair professional at each different hair salon.

The information providing device 1, terminal device 4, and store device 5 are connected to each other via a network N, either wired or wirelessly, so that they can communicate with each other. Note that the information processing system 100 shown in FIG. 2 may include multiple information providing devices 1. The network N is, for example, a local area network (LAN) or a wide area network (WAN) such as the Internet.

The information providing device 1 is an information processing device that provides various types of information. The information providing device 1 provides online services such as SNS sites, web search sites, and content distribution sites. The SNS sites accept posted information sent from terminal devices 4 and store devices 5, and transmit the posted information to the terminal devices 4 and store devices 5 based on requests from users U, hair practitioners, etc. The information providing device 1 may also be an information processing device that acquires information such as posted information from the above-mentioned SNS sites, etc.

Content distribution sites are sites that provide content distribution services, and include video and music distribution sites, map sites, route search sites, route guidance sites, line information sites, traffic information sites, and weather forecast sites. Information providing device 1 can also provide online services through various online sites, such as electronic payment sites, online game sites, online banking sites, and accommodation and ticket reservation sites.

Each of the terminal device 4 and the in-store device 5 may be, for example, a desktop PC (Personal Computer), a notebook PC, a tablet terminal, a smartphone, a mobile phone, or a PDA (Personal Digital Assistant). Note that each of the terminal device 4 and the in-store device 5 is not limited to the above examples and may be, for example, a smart watch or a wearable device.

In addition, each terminal device 4 can connect to the network N via a wireless communication network such as LTE (Long Term Evolution), 4G (4th Generation), or 5G (5th Generation: fifth generation mobile communication system), or via short-range wireless communication such as Bluetooth (registered trademark) or wireless LAN, and communicate with the information providing device 1 and each store device 5.

In addition, each store device 5 can connect to the network N via a wireless communication network such as LTE, 4G, or 5G, or via short-range wireless communication such as Bluetooth or wireless LAN, and communicate with the information providing device 1 and each terminal device 4.

3. Configuration of information providing device 1
Next, a configuration example of the information providing device 1 will be described with reference to Fig. 3. Fig. 3 is a diagram showing a configuration example of the information providing device 1 according to an embodiment. As shown in Fig. 3, the information providing device 1 includes a communication unit 10, a storage unit 11, and a processing unit 12.

3.1. Communication Unit 10
The communication unit 10 is realized by, for example, a network interface card (NIC). The communication unit 10 is connected to a network N by wire or wirelessly, and transmits and receives information to and from the terminal devices 4, the store devices 5, and the like via the network N.

[3.2. Storage unit 11]
The storage unit 11 is realized by, for example, a semiconductor memory element such as a random access memory (RAM) or a flash memory, or a storage device such as a hard disk or an optical disk. The storage unit 11 stores a user information storage unit 20, a posted information storage unit 21, a model information storage unit 22, and a content storage unit 23.

3.2.1. User Information Storage Unit 20
The user information storage unit 20 stores information about the user U. Fig. 4 is a diagram showing an example of a user information table stored in the user information storage unit 20 of the information providing device 1 according to the embodiment.

As shown in Figure 4, the user information table stored in the user information storage unit 20 includes items such as "User ID (IDentifier)," "Attribute Information," and "History Information." "User ID" is identification information that identifies user U.

"Attribute information" is information about user attributes, which are attributes of user U corresponding to "user ID," and includes information on psychographic attributes and demographic attributes. Demographic attributes include, for example, gender, age, place of residence, and occupation, while psychographic attributes include interests such as travel, clothing, cars, and religion, lifestyle, thoughts, and ideological tendencies.

"History information" is history information that includes information such as the service usage history of user U corresponding to the "user ID," and includes, for example, user U's posting history information, user U's search history information, and user U's browsing history information. User U's posting history information includes information posted by user U on the SNS site, comments by user U on information posted by other users U, and information indicating user U's evaluation of information posted by other users U.

User U's search history information is, for example, information about the search history of information posted on SNS sites, information about search history on various websites, etc. User U's browsing history information is, for example, information about the user U's browsing history of information posted on SNS sites, information about browsing history on various websites, etc.

3.2.2. Posted information storage unit 21
The posted information storage unit 21 stores various pieces of information posted by each user U. Fig. 5 is a diagram showing an example of a posted information table stored in the posted information storage unit 21 of the information providing device 1 according to the embodiment. As shown in Fig. 5, the posted information table stored in the posted information storage unit 21 includes items such as "post ID,""posting date and time,""postinguser,""postedtext,""postedimage,""ratinginformation,""number of views," and "comments."

"Post ID" is identification information that identifies posted information. Post information is an example of posted content. "Posting date and time" is information that indicates the posting date and time of the posted information corresponding to the "Posting ID." "Posting user" is information that indicates the user U who posted the posted information corresponding to the "Posting ID," and in the example shown in Figure 5, it is the user ID. "Posting text" is information that indicates the text included in the posted information corresponding to the "Posting ID," and is expressed as a string of characters, but may also include stamps, etc.

A "posted image" is a posted image included in the posted information corresponding to a "post ID." "Rating information" is rating information indicating other users U's ratings of the posted information corresponding to a "post ID," but may also include information indicating the poster's rating of the posted information corresponding to a "post ID." "Number of views" is information indicating the number of times the posted information corresponding to a "post ID" has been viewed.

A "comment" is a comment from another user U on the posted information corresponding to the "post ID" and is represented, for example, as a string of characters, but may also include stamps, etc. Note that a "comment" may also include a comment from the posting user on the posted information corresponding to the "post ID."

The evaluation information shown in Figure 5 is information indicating a positive evaluation (e.g., "good") or a negative evaluation (e.g., "bad"), but may also include information indicating an intermediate evaluation (e.g., "average").

3.2.3. Model information storage unit 22
The model information storage unit 22 stores information on various learning models. Fig. 6 is a diagram showing an example of a model information table stored in the model information storage unit 22 of the information providing device 1 according to the embodiment.

As shown in FIG. 6, the model information table stored in the model information storage unit 22 includes items such as "Model ID" and "Model Information." "Model ID" is identification information that identifies a learning model. "Model Information" is information about the learning model that corresponds to the "Model ID." The learning model information is information such as the parameters of the learning model, but is not limited to such examples and may be information for using the learning model on the terminal device 4.

Learning models whose information is included in the model information table include, for example, the first learning model, second learning model, third learning model, and fourth learning model described above. The first learning model is a model that learns the relationship between keywords and hair images, and is a learning model that takes one or more keywords as input and outputs a hair image or hair image-specific information.

The second learning model is a model that has learned the relationship between facial images and facial features; it is a learning model that takes images containing facial images as input and outputs information indicating the facial features. The third learning model is a model that has learned the relationship between information indicating facial features and hairstyles that are evaluated as suiting the face; it is a learning model that takes information indicating facial features as input and outputs information indicating hairstyles that are estimated to suit the face.

The fourth learning model is a learning model that learns the relationship between hair images before and after a treatment and the treatment details, and is a learning model that inputs hair images before and after a treatment and outputs information indicating the estimated treatment details. Note that the learning models whose information is included in the model information table are not limited to the examples described above.

Any type of model can be used for each of the above-mentioned learning models. For example, each learning model is a model generated by machine learning, such as DNN, GBDT, or SVM. A DNN is, for example, a CNN or RNN. An RNN may also be, for example, an LSTM. Each learning model may also be a model realized by combining multiple models, such as a model that combines a CNN and an RNN.

3.2.4. Content storage unit 23
The content storage unit 23 stores information about content. Fig. 7 is a diagram showing an example of a content table stored in the content storage unit 23 of the information providing device 1 according to the embodiment. The content table shown in Fig. 7 includes items such as "content ID" and "content."

A "content ID" is an identifier that identifies the content. A "content" is information about the content associated with the "content ID." Specifically, information about the content is, for example, information about the content's contents or information about the content's address.

In the example shown in Figure 7, the content with content ID "C1" is content CNT1, the content with content ID "C2" is content CNT2, and the content with content ID "C3" is content CNT3. Note that the content storage unit 23 is not limited to the above and may store various types of information depending on the purpose.

3.3. Processing Unit 12
The processing unit 12 is a controller, and is realized by a processor such as a CPU (Central Processing Unit) or an MPU (Micro Processing Unit) executing various programs (corresponding to examples of information processing programs) stored in a storage device inside the information providing device 1 using RAM or the like as a working area.

The processing unit 12 is also a controller and may be realized by an integrated circuit such as an ASIC (Application Specific Integrated Circuit), FPGA (Field Programmable Gate Array), or GPGPU (General Purpose Graphics Processing Unit).

As shown in FIG. 3, the processing unit 12 has an acquisition unit 30, a learning unit 31, and a provision unit 32, and realizes or executes the information processing functions and actions described below. Note that the internal configuration of the processing unit 12 is not limited to the configuration shown in FIG. 3, and may be any other configuration that performs the information processing described below.

[3.3.1. Acquisition unit 30]
The acquisition unit 30 acquires various types of information. The acquisition unit 30 acquires various types of information from the storage unit 11. The acquisition unit 30 acquires various types of information from the user information storage unit 20, the posted information storage unit 21, the model information storage unit 22, the content storage unit 23, etc.

The acquisition unit 30 also acquires information about the user U (e.g., attribute information and behavioral history information about the user U) from, for example, the terminal device 4, the store device 5, or an external device, and stores the acquired information about the user U in the user information storage unit 20. The acquisition unit 30 also acquires various types of content from, for example, an external device, and stores the acquired content in the content storage unit 23.

The acquisition unit 30 also acquires posted information from devices such as the terminal device 4 or the store device 5 via the communication unit 10. The acquisition unit 30 adds the acquired posted information to a posted information table stored in the posted information storage unit 21.

The acquisition unit 30 also acquires an acquisition request for posted information from the terminal device 4 via the communication unit 10. For example, the acquisition unit 30 notifies the provision unit 32 of information related to the acquired acquisition request.

The acquisition unit 30 also acquires various information such as reservation requests from the terminal device 4 via the communication unit 10, stores the acquired information such as reservation requests in the storage unit 11, and notifies the provision unit 32 of the acquired information such as reservation requests. The acquisition unit 30 also acquires various information such as correction information from the store device 5 via the communication unit 10, and notifies the provision unit 32 of the acquired information such as correction information.

Furthermore, the acquisition unit 30 acquires learning data for generating the above-mentioned learning model and notifies the learning unit 31 of the acquired learning data. For example, the acquisition unit 30 acquires information stored in the memory unit 11 from the memory unit 11 and notifies the learning unit 31 of the acquired information as learning data.

The learning data for the first learning model is, for example, learning data that includes multiple combinations of keywords and hair images. The learning data for the second learning model is, for example, learning data that includes multiple combinations of images that include facial images and information that indicates facial features.

Learning data for the third learning model is, for example, learning data that includes multiple combinations of information indicating facial features, a hair image, and information indicating an evaluation of the combination of the information indicating facial features and the hair image. The evaluation of the combination of the information indicating facial features and the hair image is, for example, information indicating a positive evaluation or information indicating a negative evaluation, and is used as a label. Learning data for the fourth learning model is, for example, learning data that includes multiple combinations of hair images before and after a treatment and information indicating the treatment details.

The acquisition unit 30 can also generate the above-mentioned learning data based on information stored in the storage unit 11. The acquisition unit 30 can also acquire and generate learning data other than the above-mentioned data.

3.3.2. Learning Unit 31
The learning unit 31 generates various learning models through machine learning using the various learning data notified by the acquisition unit 30, and stores information about the various learning models generated in the posted information storage unit 21.

For example, the learning unit 31 generates a first learning model using the learning data for the first learning model, and generates a second learning model using the learning data for the second learning model. Furthermore, the learning unit 31 generates a third learning model using the learning data for the third learning model, and generates a fourth learning model using the learning data for the fourth learning model.

Note that the learning unit 31 can also generate learning models other than the first learning model, the second learning model, the third learning model, and the fourth learning model.

[3.3.3. Providing Department 32]
The providing unit 32 starts the communication unit 10 and provides various contents to the terminal device 4 of the user U and the store device 5 of the hair practitioner. For example, the providing unit 32 generates viewing content including information posted on SNS based on a request from the terminal device 4 or the store device 5, and provides the viewing content to the terminal device 4 or the store device 5 via the communication unit 10 and the network N.

The providing unit 32 also provides various types of information notified by the acquiring unit 30 to the terminal device 4 or the store device 5, etc., via the communication unit 10 and the network N, etc. For example, the providing unit 32 provides information from the terminal device 4 notified by the acquiring unit 30 to the store device 5, etc., via the communication unit 10 and the network N, etc., or provides information from the store device 5 notified by the acquiring unit 30 to the terminal device 4, etc., via the communication unit 10 and the network N, etc.

In addition, the providing unit 32 provides information on the various learning models generated by the learning unit 31 to the terminal device 4 via the communication unit 10, the network N, etc. For example, the providing unit 32 provides the first learning model, the second learning model, the third learning model, and the fourth learning model generated by the learning unit 31 to the terminal device 4 via the communication unit 10, the network N, etc.

The providing unit 32 also provides application programs (hereinafter referred to as apps) used by the terminal device 4 to the terminal device 4 via the communication unit 10, the network N, etc. Such apps are, for example, installed on the terminal device 4 and perform the above-mentioned processes on the terminal device 4. For example, the functions of such apps enable the terminal device 4 to generate the above-mentioned composite images and obtain various types of information using the above-mentioned learning models.

[4. Terminal Device 4]
8 is a diagram illustrating an example of the configuration of the terminal device 4 according to the embodiment. As illustrated in FIG. 8, the terminal device 4 according to the embodiment includes a communication unit 40, a display unit 41, an operation unit 42, an imaging unit 43, a sensor unit 44, a storage unit 45, and a processing unit 46.

4.1. Communication Unit 40
The communication unit 40 is realized by, for example, a NIC. The communication unit 40 is connected to the network N by wire or wirelessly, and transmits and receives information to and from the information providing device 1, the store device 5, and the like via the network N. Communication with the store device 5 is performed, for example, via the information providing device 1, an SNS, or a mail server (not shown), but communication with the store device 5 may also be performed directly without going through these.

[4.2. Display section 41]
The display unit 41 is, for example, an LCD (Liquid Crystal Display) or an organic EL (Electro Luminescence) display.

[4.3. Operation unit 42]
The operation unit 42 includes, for example, a keyboard including keys for inputting letters, numbers, and spaces, an enter key, arrow keys, etc., a mouse, a power button, etc. If the display unit 41 is a touch panel compatible display, the operation unit 42 includes a touch panel.

4.4. Imaging unit 43
The imaging unit 43 is an image sensor (camera) that captures an image of a subject. For example, the imaging unit 43 is a complementary metal oxide semiconductor (CMOS) image sensor or a charge-coupled device (CCD) image sensor. The imaging unit 43 is not limited to a built-in camera, and may be a wireless camera that can communicate with the terminal device 4 or an external camera such as a web camera.

4.5. Sensor unit 44
The sensor unit 44 includes a position detection unit, a gyro sensor, etc. The position detection unit detects, for example, the position of the terminal device 4, which is the current position of the user U, and outputs information indicating the detected current position of the user U to the processing unit 46.

The position detection unit receives multiple positioning signals transmitted from multiple positioning satellites in the Global Navigation Satellite System (GNSS) and detects the current position of the user U based on the multiple received positioning signals. The gyro sensor is a sensor that detects the attitude, such as the tilt and rotation, of the terminal device 4.

[4.6. Storage unit 45]
The storage unit 45 is realized by, for example, a semiconductor memory element such as a random access memory (RAM) or a flash memory, or a storage device such as a hard disk or an optical disk.

The memory unit 45 stores, for example, information transmitted from the information providing device 1, the store device 5, etc. and acquired by the processing unit 46 via the network N and the communication unit 40, captured images that are images captured by the imaging unit 43, and detection information that is information detected by the sensor unit 44. For example, the memory unit 45 stores information about apps and various learning models transmitted from the information providing device 1.

4.7. Processing Unit 46
The processing unit 46 is a controller, and is realized by, for example, a CPU or an MPU executing various programs (corresponding to an example of an information processing program) stored in a storage device inside the terminal device 4 using the RAM as a work area. The processing unit 46 may be realized in part or in whole by an integrated circuit such as an ASIC or an FPGA.

The processing unit 46 includes a reception unit 50, an acquisition unit 51, an identification unit 52, an image processing unit 53, an estimation unit 54, a determination unit 55, a comparison unit 56, and an output unit 57. The processing unit 46 functions as a functional unit including the reception unit 50, acquisition unit 51, identification unit 52, image processing unit 53, estimation unit 54, determination unit 55, comparison unit 56, and output unit 57, for example, by executing the above-mentioned app on the OS (Operating System).

Note that multiple functions, including the reception unit 50, acquisition unit 51, identification unit 52, image processing unit 53, estimation unit 54, determination unit 55, comparison unit 56, and output unit 57, may be pre-integrated in the processing unit 46.

4.7.1. Reception unit 50
The reception unit 50 receives various requests and information. The reception unit 50 receives requests and information transmitted from the terminal device 4 and the in-store device 5.

For example, the reception unit 50 receives correction information transmitted from the in-store device 5. The correction information is information indicating correction suggestions made by the hair practitioner. The reception unit 50 also receives reservation acceptance information transmitted from the in-store device 5. The reservation acceptance information includes information indicating that a reservation has been accepted, such as the treatment content, in the reservation request.

The reception unit 50 also receives various operations by the user U using the operation unit 42, etc. For example, the reception unit 50 receives various operations such as the user U selecting a hair image, the user U editing a hair image, the user U specifying the treatment details, the user U approving or not approving the estimated treatment details, and the user U specifying the treatment details.

The reception unit 50 also receives an imaging operation from the user U. When the reception unit 50 receives an imaging operation from the user U, it operates the imaging unit 43 to cause the imaging unit 43 to perform the imaging operation.

The reception unit 50 also receives a first change operation of the hair image by the user U, a second change operation of the hair image by the user U, etc. The first change operation is an operation for changing the composite image by changing the hair image used to generate the composite image. The second change operation is an operation for changing at least one of the hair length and color of the hair image.

Figure 9 is a diagram showing an example of the configuration of the reception unit 50 of the terminal device 4 according to the embodiment. As shown in Figure 9, the reception unit 50 includes a selection reception unit 60, an edit reception unit 61, a designation reception unit 62, a revision proposal reception unit 63, and an approval/denial reception unit 64.

The selection receiving unit 60 receives various selections by the user U. For example, the selection receiving unit 60 receives a hair image selected by the user U from a plurality of types of hair images. The user U can perform a hair image selection operation by operating the operation unit 42, and the selection receiving unit 60 receives the hair image selection based on the hair image selection operation.

The hair image selected by the user U is, for example, one or more of an image of a hairstyle identified by the identification unit 52, a plurality of similar hair images acquired by the acquisition unit 51, and a hair image included in a captured image captured by the imaging unit 43.

Note that the hair image selected by user U is not limited to the above-mentioned example. For example, the hair image selected by user U may include a hair image selected by the processing unit 46 when an automatic hair image selection mode by the processing unit 46 is selected by an operation of user U.

The edit receiving unit 61 accepts edits to the hair image. For example, the edit receiving unit 61 accepts a selection of at least one of the hair length and color of the hair in the hair image. The hair in the hair image is the hair shown in the hair image, and the user U can edit the hair image by operating the operation unit 42, and the edit receiving unit 61 accepts edits to the hair image based on the edit operations for the hair image.

The specification acceptance unit 62 accepts various specifications from the user U. For example, the specification acceptance unit 62 accepts the specification of treatment content by the user U. The user U can specify the treatment content by operating the operation unit 42, and the specification acceptance unit 62 accepts the specification of the treatment content based on the specification operation of the treatment content.

In addition, the specification receiving unit 62 receives a specification of a user-captured image, which is an image captured by the imaging unit 43 and includes a hair image, based on an operation by the user U on the operation unit 42.

The designation receiving unit 62 also receives the user U's approval or denial of the estimated treatment content. The user U can perform an approval or denial operation for the estimated treatment content by operating the operation unit 42, and the designation receiving unit 62 receives approval or denial of the treatment content based on the approval or denial operation.

The specification acceptance unit 62 also accepts the specification of one or more keywords by the user U. For example, the specification acceptance unit 62 accepts the specification by the user U of one or more keywords from among a plurality of keywords. The user U can specify one or more keywords by operating the operation unit 42, and the specification acceptance unit 62 accepts the specification of one or more keywords based on the specification operation of one or more keywords.

The revision proposal receiving unit 63 receives revision proposals made by the hair practitioner to the estimated treatment content. For example, the revision proposal receiving unit 63 receives revision proposal information, which is information indicating revision proposals made by the hair practitioner to the estimated treatment content, from the in-store device 5 as revision proposals made by the hair practitioner to the estimated treatment content.

The revision proposal receiving unit 63 also receives revision proposals from the hairdresser for the hairstyle displayed in the composite image. For example, the revision proposal receiving unit 63 receives revision proposal information, which is information indicating revision proposals from the hairdresser, from the in-store device 5 as revision proposals for the hairstyle from the hairdresser.

The permission acceptance unit 64 accepts the user U's permission or denial of the estimated treatment content, which is the treatment content estimated by the estimation unit 54. The user U can operate the operation unit 42 to perform a permission operation indicating permission for the estimated treatment content or a denial operation indicating denial of the estimated treatment content, and the permission acceptance unit 64 accepts the user U's permission or denial of the estimated treatment content based on the permission operation or denial operation.

[4.7.2. Acquisition unit 51]
The acquisition unit 51 acquires various types of information. For example, the acquisition unit 51 acquires various types of information from the storage unit 45. The acquisition unit 51 acquires information on the learning model stored in the storage unit 45.

The acquisition unit 51 also acquires captured images, which are images captured by the imaging unit 43, and stores the acquired captured images in the storage unit 45. The acquisition unit 51 also acquires detection information, which is information detected by the sensor unit 44, and stores the acquired detection information in the storage unit 45.

The acquisition unit 51 acquires information about apps and learning models transmitted from the information providing device 1 via the network N and the communication unit 40, and stores the acquired information about apps and learning models in the storage unit 45. The acquisition unit 51 acquires multiple pieces of posted information, each of which includes a specific hair image, from the information providing device 1, and stores the acquired pieces of posted information in the storage unit 45.

The acquisition unit 51 also acquires practitioner comments and stores the acquired comments in the storage unit 45. The practitioner comments are comments from the hair practitioner as to whether or not the hairstyle shown in the composite image can be performed.

The acquisition unit 51 also acquires information corresponding to an operation on the operation unit 42 by the user U. For example, the acquisition unit 51 acquires, based on the operation by the user U, an image captured by the imaging unit 43 and including a hair image from the storage unit 45.

The acquisition unit 51 also acquires multiple similar hair images, which are images that include hair images similar to the specific hair image, and stores the multiple acquired similar hair images in the storage unit 45. The specific hair image is an image of a hairstyle identified by the identification unit 52. For example, the acquisition unit 51 transmits a similar image search request including the specific hair image to a search server or information providing device 1 (not shown), and acquires multiple similar hair images transmitted from the search server or information providing device 1 in response to the similar image search request.

Furthermore, the acquisition unit 51 transmits a search request including information indicating the hairstyle identified by the identification unit 52 to a search server or information providing device 1 (not shown), and acquires multiple identified hair images transmitted from the search server or information providing device 1 in response to the search request.

The acquisition unit 51 also acquires treatment cost information for each hair salon, which indicates the relationship between treatment content and treatment cost, from, for example, the information providing device 1 or the store device 5.

[4.7.3. Specification part 52]
The identification unit 52 uses the first learning model to identify a hair image from one or more keywords whose designation has been accepted by the acceptance unit 50. As described above, the first learning model is a model that has learned the relationship between keywords and hair images, and receives one or more keywords as input and outputs a hair image or hair image identification information.

When the first learning model outputs a hair image, the identification unit 52 identifies the hair image output from the first learning model as a hair image corresponding to one or more keywords specified by the user U.

Furthermore, when the first learning model outputs hair image identification information instead of a hair image, the identification unit 52 inputs one or more keywords specified by the user U into the first learning model and obtains the hair image identification information output from the first learning model. The hair image identification information is information that identifies a hair image, such as a score for each hair image. For example, the identification unit 52 identifies the hair image with the highest score among the scores for each hair image as the hair image corresponding to the one or more keywords specified by the user U.

The identification unit 52 acquires the hair image identified by the image identification information from the storage unit 45 or the information providing device 1, and identifies the acquired hair image as a hair image corresponding to one or more keywords specified by the user U.

The identification unit 52 also uses the third learning model to identify a hairstyle that is estimated to suit the face of the user U. For example, the identification unit 52 uses the third learning model to identify a hairstyle that is estimated to suit the face of the user U from the facial features of the user U specified by the user U or the facial features of the user U determined by the determination unit 55.

As described above, the third learning model is a model that learns the relationship between information indicating facial features and hairstyles that are evaluated as suiting that face, and is a model that takes information indicating facial features as input and outputs information indicating hairstyles that are estimated to suit that face.

The identification unit 52 inputs information indicating the facial features of the user U into the third learning model, and identifies a hairstyle that is estimated to suit the user U's face based on the hairstyle information output from the third learning model. The hairstyle information output from the third learning model is, for example, a score for each hair image, and the identification unit 52 identifies the hairstyle indicated by the hair image with the highest score among the scores for each hair image as the hairstyle that is estimated to suit the user U's face. The hairstyle information output from the third learning model may also be a hair image. The information indicating the facial features of the user U is, for example, information indicating the facial features of the user U specified by the user U or information indicating the facial features of the user U determined by the determination unit 55.

Facial features are, for example, information indicating the type of face shape, but may also be information indicating the characteristics of each part of the face, such as the size, shape, and position of one or more of the eyebrows, eyes, nose, mouth, and ears. For example, information indicating facial features may be information indicating the position of each of the eyebrows, eyes, nose, mouth, and ears, or information indicating the position and size of each of the eyebrows, eyes, nose, mouth, and ears.

4.7.4. Image Processing Unit 53
The image processing unit 53 performs various types of image processing. For example, the image processing unit 53 generates a composite image by combining a user face image, which is a facial image of the user U, with a hair image selected by the user U. Hereinafter, the hair image selected by the user U may be referred to as a selected hair image.

The selected hair image is a hair image whose selection has been accepted by the selection accepting unit 60, but is not limited to this example. For example, the selected hair image may include a hair image selected by the processing unit 46 when an automatic hair image selection mode by the processing unit 46 is selected by an operation of the user U. The user face image is, for example, a user-captured image whose specification has been accepted by the specification accepting unit 62, or a face image of the user U included in a user-captured image.

For example, the image processing unit 53 generates a composite image by combining the selected hair image with the user's face image. The image processing unit 53 generates a composite image by replacing the hair image of user U included in the user's face image with the selected hair image.

The image processing unit 53, for example, extracts the feature amounts of the user's face image and the feature amounts of the selected hair image. The image processing unit 53 generates a composite image by replacing the feature amounts of the hair image included in the user's face image, among the feature amounts of the user's face image, with the feature amounts of the selected hair image.

The image processing unit 53 can also generate a composite image by deleting the user U's hair image included in the user face image and superimposing the selected hair image on the user face image from which the user U's hair image has been deleted.

In addition, when generating a composite image by combining a selected image, which is an image including a selected hair image, with a user image, which is an image including a user face image, the image processing unit 53 can also replace the user face image included in the user image with the selected hair image.

Furthermore, when two or more hair images are selected by the selection receiving unit 60, the image processing unit 53 generates a composite hair image by combining the two or more selected hair images. Then, the image processing unit 53 generates a composite image by combining the generated composite hair image with the user's facial image.

For example, the image processing unit 53 extracts feature amounts from each of two or more selected hair images and generates a composite hair image by combining the extracted feature amounts of the two or more selected hair images. For example, the image processing unit 53 generates a composite hair image by combining the feature amounts of the two or more selected hair images using weighted addition, but the combining method is not limited to this example. For example, the image processing unit 53 has a synthesis model that is a learning model for synthesizing two or more images and is generated by machine learning, and by inputting two or more selected hair images into this synthesis model, a composite hair image output from the synthesis model can be obtained.

When the image processing unit 53 generates a composite image by combining a selected image, which is an image including a selected hair image, with a user image (e.g., a user-captured image), which is an image including a user's face image, it can also replace the face image of a person included in the selected image with the user's face image included in the user image.

For example, suppose the selected image is a selected captured image captured by the imaging unit 43. In this case, the image processing unit 53 generates a composite image by replacing the facial image of a person included in the selected captured image with the user's facial image included in the user image.

When the image processing unit 53 generates a composite image by combining a selected image, which is an image including a selected hair image, with a user image, which is an image including a user face image, it can also replace the user face image included in the user image with the face image of a person included in the selected image.

For example, suppose the selected image is a selected captured image captured by the imaging unit 43. In this case, the image processing unit 53 generates a composite image by replacing the user's face image in the user image with the face image of a person included in the selected captured image.

Furthermore, when there are multiple selected images, the image processing unit 53 combines the multiple selected images to generate a composite selected image. For example, the image processing unit 53 uses a specific selected image as a reference to combine the multiple selected images to generate a composite selected image. The specific selected image is, for example, the initially selected selected image or the selected image specified by the user U.

For example, the image processing unit 53 extracts features of the hair image of each of the multiple selected images, synthesizes the extracted features of the multiple hair images to generate a composite hair image, and replaces the hair image of a specific selected image with the composite hair image to generate a composite selected image. Also, the image processing unit 53 has a synthesis model that is a learning model for synthesizing multiple images and is generated by machine learning, and can obtain a composite hair image output from the synthesis model by inputting the multiple selected images into the synthesis model.

In addition, when the processing reception unit 61 receives a request to process a hair image included in the composite image, the image processing unit 53 generates a user face image including a processed hair image obtained by processing the hair image received by the processing reception unit 61.

For example, when the processing reception unit 61 receives a request to edit a hair image, the image processing unit 53 generates an edited hair image by processing the hair image received by the processing reception unit 61. The image processing unit 53 then generates a composite image by combining the edited hair image with the user's facial image.

The image processing unit 53 generates an edited hair image by, for example, extracting feature amounts from the hair image and adjusting the extracted feature amounts, but is not limited to this example. Furthermore, when the edit receiving unit 61 receives an instruction to edit the hair image in the composite image, the image processing unit 53 can also extract feature amounts from the hair image in the composite image and edit the hair image in the composite image by adjusting the extracted feature amounts.

The image processing unit 53 can also perform projective transformation of the selected image. Figure 10 is a diagram showing an example of the configuration of the image processing unit 53 of the terminal device 4 according to the embodiment. As shown in Figure 10, the image processing unit 53 includes a transformation unit 65 and a synthesis unit 66. The transformation unit 65 performs projective transformation of the selected image. The synthesis unit 66 generates a synthesized image by replacing the facial image of a person included in the selected image or the facial image of a person included in the selected image projectively transformed by the transformation unit 65 with the user's facial image.

The transformation unit 65 can also perform projective transformation on a user image or a user face image in place of the selected image. In this case, the synthesis unit 66 generates a composite image by replacing the face image of a person included in the selected image or the face image of a person included in the selected image projectively transformed by the transformation unit 65 with the face image of user U included in the user image projectively transformed by the transformation unit 65 or the user face image projectively transformed by the transformation unit 65.

The synthesis unit 66 can also generate a composite image by replacing the facial image of user U included in the user image projectively transformed by the transformation unit 65 with the facial image of a person included in the selected image or the facial image of a person included in the selected image projectively transformed by the transformation unit 65.

[4.7.5. Estimation unit 54]
The estimation unit 54 estimates the treatment content from the pre-treatment image and the composite image of the user U. For example, the estimation unit 54 uses the fourth learning model to estimate the treatment content from the pre-treatment image and the composite image of the user U.

The fourth learning model is a learning model that learns the relationship between hair images before and after a treatment and the treatment details. It takes hair images before and after a treatment as input and outputs information indicating the estimated treatment details.

The image of user U before the treatment is, for example, the user-captured image described above. The treatment details estimated by the estimation unit 54 are the estimated treatment details described above, and are the treatment details required to achieve the hairstyle shown in the composite image generated by the image processing unit 53.

When the estimation unit 54 inputs the image of the user U before the treatment and the composite image generated by the image processing unit 53 into the fourth learning model, it determines the estimated treatment content based on the information indicating the treatment content output from the fourth learning model.

The information indicating the treatment content output from the fourth learning model includes, for example, score information for each of the following: whether or not a haircut was performed, the amount of haircut, whether or not bleaching was performed, the number of times bleaching was performed, whether or not coloring was performed, the type of coloring, whether or not perming was performed, the type of perming, and whether or not hair straightening was performed. The estimation unit 54 determines the estimated treatment content based on each score output from the fourth learning model.

For example, the estimation unit 54 includes a cut in the estimated treatment details when the score for whether or not a cut was performed is equal to or greater than a threshold, and does not include a cut in the estimated treatment details when the score for whether or not a cut was performed is less than the threshold. Furthermore, the estimation unit 54, for example, includes in the estimated treatment details the amount of cut that is proportional to the score for the amount of cut. Furthermore, the estimation unit 54, for example, includes in the estimated treatment details colors with scores equal to or greater than a threshold among the scores for each type of color.

Furthermore, the information indicating the treatment content from the fourth learning model may be information indicating the treatment content including the treatment items and the treatment order. In this case, the fourth learning model is configured to output scores for multiple treatment methods, each including the treatment items and the treatment order, and the treatment content indicated by the treatment method with the highest score is determined as the estimated treatment content.

[4.7.6. Judgment unit 55]
The determination unit 55 determines the treatment cost, which is the cost required for the treatment according to the estimated treatment content, based on the estimated treatment content determined by the estimation unit 54.

For example, the determination unit 55 has treatment cost information for each hair salon that indicates the relationship between the treatment content and the treatment cost, and determines the treatment cost for each hair salon based on this treatment cost information. The treatment cost information for each hair salon is information acquired by the acquisition unit 51 as described above, but is not limited to this example.

Furthermore, the determination unit 55 determines, for example, the facial features of the user U shown in the user-captured image. For example, the determination unit 55 inputs the user-captured image into the second learning model described above, and determines the facial features of the user U based on the information indicating the facial features output from the second learning model.

The information indicating the facial features output from the second learning model is, for example, a score for each facial feature. In this case, the determination unit 55 determines the facial feature with the highest score among the scores for each facial feature as the facial feature of user U.

Facial feature information output from the second learning model may also be a score for each feature (e.g., eyebrows, eyes, nose, mouth, ears). In this case, the determination unit 55 determines the facial feature including the feature with the highest score among the scores for each feature (combination of position, size, and shape) as the facial feature of user U.

Facial feature information output from the second learning model may also include scores for each feature, such as the size and shape of each feature (e.g., eyebrows, eyes, nose, mouth, ears). In this case, the determination unit 55 determines the facial feature that includes the feature with the highest score among the scores for each feature, such as the size and shape, as the facial feature of user U.

The determination unit 55 may also have information indicating each of multiple types of facial features. In this case, the determination unit 55 determines the facial features of user U using pattern matching technology between each type of facial feature and the facial features of user U shown in the user-captured image.

The determination unit 55 can also display facial feature line drawings, which are line drawings showing multiple types of facial features (e.g., face shapes), on the display unit, and allow the user U to select the facial feature line drawing that most closely matches their own face shape from among these multiple types of facial feature line drawings.

4.7.7. Comparison Unit 56
The comparison unit 56 compares the user-specified treatment content, which is the treatment content specified and accepted by the specification acceptance unit 62, with the estimated treatment content, which is the treatment content estimated by the estimation unit 54.

The comparison unit 56, for example, compares the user-specified treatment content with the estimated treatment content and generates comparison information indicating the comparison results. The comparison information includes, for example, information indicating the difference between the user-specified treatment content and the estimated treatment content.

For example, if the estimated treatment content includes one more bleach than the user-specified treatment content, the comparison unit 56 generates comparison information including information indicating that the number of bleaches is one more than the user-specified treatment content.

In addition, if the estimated treatment content or the user-specified treatment content includes a treatment order, and if part of the treatment order is different, the comparison unit 56 generates comparison information including information indicating the different treatment order.

4.7.8. Output Unit 57
The output unit 57 outputs, as output information, information indicating the identification result by the identification unit 52, information indicating the result of image processing by the image processing unit 53, information indicating the estimation result by the estimation unit 54, information indicating the judgment result by the judgment unit 55, and information indicating the comparison result by the comparison unit 56.

The output unit 57 can also output, as output information, information indicating the reception result by the reception unit 50, information indicating the acquisition result by the acquisition unit 51, and the like. The output unit 57 can, for example, output the output information to the display unit 41 to display the output information on the display unit 41, or output the output information to the hair practitioner via the communication unit 40 and the network N, for example. Output to the hair practitioner is performed, for example, via the information providing device 1, SNS, or a mail server (not shown), but may also be output directly to the in-store device 5.

The output unit 57 outputs information on a reservation setting screen for reserving a hair treatment at a hair salon as output information. The reservation setting screen is a GUI (Graphical User Interface) screen for setting a user image, hair image, hair salon, reservation date and time, etc., and will be described in detail later.

The output unit 57 outputs the reservation request including the treatment-related information to the hair salon practitioner, for example, by sending the reservation request including the treatment-related information to the hair salon practitioner.

The treatment-related information includes, for example, a user image including a hair image of the user U before the treatment, a composite image generated by the image processing unit 53, information indicating the treatment content specified and accepted by the specification accepting unit 62, and information indicating the treatment content estimated by the estimating unit 54.

The treatment-related information may include a selected hair image, which is a hair image whose selection has been accepted by the selection accepting unit 60, instead of or in addition to the composite image generated by the image processing unit 53.

The output unit 57 also outputs multiple keywords that can be specified by the user U. For example, the output unit 57 outputs output information including multiple keywords to the display unit 41, thereby displaying the multiple keywords on the display unit 41 so that the user U can specify them.

The output unit 57 also outputs the practitioner comments acquired by the acquisition unit 51. For example, the output unit 57 outputs the practitioner comments acquired by the acquisition unit 51 to the display unit 41, thereby causing the practitioner comments to be displayed on the display unit 41.

The output unit 57 also outputs revision proposal information, which is information indicating revision proposals made by the hair practitioner and accepted by the revision proposal acceptance unit 63. The revision proposal information is, for example, information indicating revision proposals for the estimated treatment content or information indicating revision proposals for the hairstyle shown in the composite image. For example, the output unit 57 outputs the revision proposal information accepted by the revision proposal acceptance unit 63 to the display unit 41, thereby displaying the revision proposals made by the hair practitioner on the display unit 41.

In addition, if the permission acceptance unit 64 accepts permission, the output unit 57 outputs the above-mentioned reservation request to the hair practitioner, and if the permission acceptance unit 64 accepts denial, the output unit 57 does not output the above-mentioned reservation request to the hair practitioner.

When displaying output information on the display unit 41, the output unit 57 converts the output information into information for display by, for example, rendering the output information, and outputs the converted output information to the display unit 41. If the display unit 41 has a rendering function, the output unit 57 outputs the output information to the display unit 41 as is, and displays the output information on the display unit 41.

Figure 11 is a diagram showing an example of a reservation setting screen displayed on the display unit 41 by the output unit 57 of the information providing device 1 according to the embodiment. As shown in Figure 11, the reservation setting screen 70 includes a user image selection area 71, a hair image selection area 72, a hair salon selection area 73, and a reservation date and time setting area 74.

The appointment setting screen 70 is displayed on the display unit 41 by the output unit 57, for example, when the operation by the user U on the operation unit 42 is a specific appointment setting operation. Although not shown in FIG. 11, the appointment setting screen 70 also includes, for example, a treatment content input area, a treatment content estimation button, and an appointment button.

The user image selection area 71 includes a start shooting button 711 and a folder button 712. The start shooting button 711 is a button for driving an imaging application program (hereinafter sometimes referred to as an imaging app) that captures images with the imaging unit 43. The user U can operate the start shooting button 711 to display the imaging app screen on the display unit 41 and operate the imaging app by operating the operation unit 42. By operating the imaging app, the user U can capture an image of an area including their own face. As a result, the acquisition unit 51 acquires a user image.

The folder button 712 is a button for opening a specific folder (for example, a photo folder containing images captured by the imaging unit 43). By operating the folder button 712, the user U can display multiple captured images contained in the specific folder on the display unit 41. By operating the operation unit 42, the user U can select a user image from the multiple captured images contained in the specific folder. This causes the acquisition unit 51 to acquire the user image.

The hair image selection area 72 includes a keyword setting button 721, a start shooting button 722, and a folder button 723. The keyword setting button 721 is a button for displaying a keyword specification screen on the display unit 41, which allows the user U to specify one or more desired keywords from among multiple keywords.

The start shooting button 722 is a button for driving the imaging app. The user U can operate the start shooting button 722 to display the imaging app screen on the display unit 41 and operate the imaging app by operating the operation unit 42. By operating the imaging app, the user U can capture an image of an area including a desired hair image (for example, an area including an image of a hair model featured in a hair catalog or fashion magazine). This causes the acquisition unit 51 to acquire the selected image.

The folder button 723 is a button for opening a specific folder (for example, a photo folder containing images captured by the imaging unit 43). By operating the folder button 723, the user U can display multiple captured images contained in the specific folder on the display unit 41. By operating the operation unit 42, the user U can select a desired image from the multiple captured images contained in the specific folder. This causes the acquisition unit 51 to acquire the selected image.

The hair salon selection area 73 includes a current location vicinity search button 731 and a keyword search button 732. The current location vicinity search button 731 is a button for displaying selectable hair salons located near the current location (e.g., an area within a predetermined distance from the current location) on the display unit 41. By operating the operation unit 42, the user U can select the desired hair salon from the hair salons displayed on the display unit 41.

The keyword search button 732 is a button for displaying a keyword search screen on the display unit 41 for performing a keyword search. By operating the operation unit 42, the user U can select the desired hair salon from the hair salons searched for on the keyword search screen and displayed on the display unit 41.

The reservation date and time setting area 74 is an area for setting the reservation date and time. By operating the operation unit 42, the user U can set the date and time at which they wish to receive the hair treatment in the reservation date and time setting area 74.

Figure 12 is a diagram showing an example of a keyword specification screen displayed on the display unit 41 by the output unit 57 of the information providing device 1 according to the embodiment. As shown in Figure 12, the keyword specification screen 80 includes a group of keyword specification buttons 81, 82, 83, and 84, and a search button 85.

The keyword specification button group 81 includes multiple buttons for specifying keywords corresponding to hair length. The keyword specification button group 81 shown in FIG. 12 includes a button for specifying the keyword "short," a button for specifying the keyword "medium," a button for specifying the keyword "long," and so on.

The keyword specification button group 82 includes multiple buttons for specifying keywords corresponding to hair color. The keyword specification button group 82 shown in FIG. 12 includes a button for specifying the keyword "brown," a button for specifying the keyword "beige," a button for specifying the keyword "pink," a button for specifying the keyword "black," and so on.

The keyword specification button group 83 includes multiple buttons for specifying keywords corresponding to atmospheres. The keyword specification button group 83 shown in FIG. 12 includes a button for specifying the keyword "cute," a button for specifying the keyword "cool," a button for specifying the keyword "pop," a button for specifying the keyword "natural," and so on.

The keyword specification button group 84 includes multiple buttons for specifying keywords corresponding to TPO (Time, Place, Occasion). The keyword specification button group 84 shown in FIG. 12 includes a button for specifying the keyword "Harajuku," a button for specifying the keyword "Ginza," a button for specifying the keyword "Shinjuku," a button for specifying the keyword "Roppongi," and so on.

The multiple keywords indicated by the keyword specification buttons 81, 82, 83, and 84 are multiple predetermined keywords, but may also be multiple keywords randomly extracted from the multiple keywords or multiple keywords that satisfy predetermined conditions. Multiple keywords that satisfy predetermined conditions are, for example, a predetermined number of keywords that are ranked highest in order of frequency of selection by user U. Keywords that are candidates for extraction are, for example, keywords that are to be input into the first learning model.

On the keyword specification screen 80, the user U can specify hair length, hair color, mood, time, occasion, etc. by operating the operation unit 42 and pressing a button corresponding to the desired keyword. Then, the user U operates the operation unit 42 to select the search button 85, which causes the processing unit 46 to execute a process of acquiring similar hair images from the specified keyword. Note that the multiple keywords included in the keyword specification screen 80 are not limited to the examples described above.

Figure 13 is a diagram illustrating the process of acquiring similar hair images from specified keywords by the processing unit 46 of the information providing device 1 according to the embodiment. In the example shown in Figure 13, the keywords specified by user U are "long," "pink," "cute," and "Harajuku."

In this case, the reception unit 50 receives the keywords "long hair," "pink," "cute," and "Harajuku" specified by the user U. The identification unit 52 uses the first learning model to identify a hair image from the keywords "long hair," "pink," "cute," and "Harajuku" specified and accepted by the reception unit 50.

Then, the acquisition unit 51 sends a similar image search request including the specific hair image to a search server or information providing device 1 (not shown), and acquires multiple similar hair images sent from the search server or information providing device 1 in response to the similar image search request.

Figure 14 is a diagram showing another example of a reservation setting screen displayed on the display unit 41 by the output unit 57 of the information providing device 1 according to an embodiment. In the reservation setting screen 70 shown in Figure 14, a user image selected by user U is set in a user image selection area 71, and multiple similar hair images corresponding to one or more keywords specified by user U are set in a hair image selection area 72.

By operating the operation unit 42, the user U can select one or more similar hair images from the multiple similar hair images set in the hair image selection area 72 as selected images.

Figure 15 is a diagram showing yet another example of a reservation setting screen displayed on the display unit 41 by the output unit 57 of the information providing device 1 according to an embodiment. The reservation setting screen 70 shown in Figure 15 is displayed on the display unit 41 when the user U selects one of the multiple similar hair images shown in Figure 14 by operating the operation unit 42 on the reservation setting screen 70 shown in Figure 14.

The reception unit 50 receives the selection of a similar hair image by the user U as a selected image. The image processing unit 53 generates a composite image by combining the similar hair image selected and received by the reception unit 50 with the user image selected by the user U, and places the generated composite image on the reservation setting screen 70 as shown in FIG. 15. This allows the user U to check the composite image displayed on the display unit 41 and more realistically confirm whether the hairstyle shown in the selected hair image suits them.

In addition, by operating the operation unit 42, the user U can select a composite image on the reservation setting screen 70, thereby displaying the composite image on the display unit 41 while changing the selected image used to generate the composite image.

Figure 16 is a diagram showing an example of a composite image change screen displayed on the display unit 41 by the output unit 57 of the information providing device 1 according to the embodiment. The composite image change screen 90 shown in Figure 16 includes a length selection button 91, a color selection button 92, and a composite image display area 93.

The length selection button 91 is a button for changing the target to be changed by swiping (or scrolling) up or down to the hair length. The color selection button 92 is a button for changing the target to be changed by swiping (or scrolling) up or down to the hair color. Swiping (or scrolling) up or down is an example of a second change operation for the hair image.

In the example shown in FIG. 16, the length selection button 91 is selected, and the user U can change the hair length of the composite image by operating the operation unit 42 to swipe (or scroll) up or down. When the reception unit 50 receives a swipe (or scroll) up or down, the image processing unit 53 changes the hair length of the composite image and displays the composite image with the changed hair length on the display unit 41.

Furthermore, the user U can change the hair color of the composite image by operating the operation unit 42 to select the color selection button 92, and then operating the operation unit 42 to swipe (or scroll) up or down. When the reception unit 50 receives an up or down swipe, the image processing unit 53 changes the hair color of the composite image and displays the composite image with the changed hair color on the display unit 41.

Furthermore, user U can display a composite image in which the selected image has been changed by swiping (or scrolling) left or right on the display unit 41. Swiping (or scrolling) left or right is an example of a first change operation for the hair image. When a left or right swipe (or scroll) is accepted by the accepting unit 50, the image processing unit 53 changes the selected image to generate a composite image, and displays the composite image in which the selected image has been changed on the display unit 41.

As described above, the appointment setting screen 70 includes a treatment content input area (not shown). By operating the operation unit 42, the user U can input the treatment content that the user U has specified or desired into the treatment content input area.

As described above, the reservation setting screen 70 also includes a treatment content estimation button and a reservation button (not shown). After the user U decides on the composite image and inputs the treatment content specified or desired by the user U in the treatment content input area, the user U selects a hair salon and sets the reservation date and time. The user U then operates the operation unit 42 to select the reservation button, which causes a reservation request to be sent from the output unit 57 to the hair practitioner. This causes the reservation request to be output to the hair practitioner.

The reception unit 50 receives the selection of the treatment content estimation button by the user U. When the reception unit 50 receives the selection of the treatment content estimation button, the estimation unit 54 estimates the treatment content from the user image and composite image displayed on the appointment setting screen 70. The user image is an example of an image of the user U before the treatment. When the user image and composite image are input into the fourth learning model, the estimation unit 54 determines the estimated treatment content based on information indicating the treatment content output from the fourth learning model.

Figure 17 is a diagram illustrating the process of estimating treatment details from a user image and a composite image by the estimation unit 54 of the information providing device 1 according to the embodiment. In the example shown in Figure 17, the estimation unit 54 determines "cut, bleach once, color bleach twice, and perm once" as the estimated treatment details from the user image and the composite image.

When the reception unit 50 accepts the selection of the reservation button, the output unit 57 outputs to the hair practitioner a reservation request including the user image, the composite image, information indicating the treatment content entered in the treatment content input area, information indicating the estimated treatment content, and information indicating the reservation date and time. Furthermore, when the reception unit 50 accepts the selection of a specific button (not shown), the output unit 57 causes the display unit 41 to display, for example, a screen indicating the end of the operation. The selection of the reservation button by the user U is an example of a response indicating the user U's consent to the estimated treatment content, and the selection of a specific button by the user U is an example of a response indicating the user U's refusal to accept the estimated treatment content.

In addition, the output unit 57 can output a face shape selection screen to the display unit 41 in response to an operation by the user U on the operation unit 42, allowing the user U to select their face shape, and display the face shape selection screen on the display unit 41.

Figure 18 is a diagram showing an example of a face shape selection screen displayed on the display unit 41 by the output unit 57 of the information providing device 1 according to the embodiment. As shown in Figure 18, the face shape selection screen 95 includes a plurality of face shapes. The user U can operate the operation unit 42 to select the face shape that is closest to the user U's face shape from among the face shapes.

The reception unit 50 receives a face shape selection by the user U from among multiple face shapes included on the face shape selection screen 95. The identification unit 52 regards the face shape selected by the user U as the facial features of the user U designated by the user U, and identifies a hairstyle estimated to suit the user U's face using a third learning model from the facial features of the user U.

The identification unit 52 inputs information indicating the face shape selected by the user U into a third learning model, and identifies a hairstyle that is estimated to suit the user U's face based on the hairstyle information output from the third learning model.

5. Processing Procedure
Next, a procedure of information processing by the processing unit 46 of the terminal device 4 according to the embodiment will be described. Fig. 19 is a flowchart showing an example of information processing by the processing unit 46 of the terminal device 4 according to the embodiment.

As shown in FIG. 19, the processing unit 46 of the terminal device 4 determines whether or not learning model information has been acquired from the information providing device 1 (step S20). If the processing unit 46 determines that learning model information has been acquired (step S20: Yes), it stores the acquired learning model information in the memory unit 45 (step S21).

When the processing of step S21 is completed, or when it is determined that learning model information has not been acquired (step S20: No), the processing unit 46 determines whether or not a reservation setting operation has been performed by the user U (step S22). When it is determined that a reservation setting operation has been performed (step S22: Yes), the processing unit 46 executes reservation processing (step S23). The reservation processing of step S23 is the processing of steps S40 to S49 shown in FIG. 20, and will be described in detail later.

When the processing of step S23 is completed or when it is determined that no reservation setting operation has been performed (step S22: No), the processing unit 46 determines whether or not there are any revision suggestions from the hair practitioner (step S24). When it determines that there are revision suggestions (step S24: Yes), the processing unit 46 causes the display unit 41 to display the revision suggestions from the hair practitioner (step S25).

When the processing of step S25 is completed or when it is determined that there are no revision suggestions (step S24: No), the processing unit 46 determines whether there are any therapist comments (step S26). When it is determined that there are any therapist comments (step S26: Yes), the processing unit 46 causes the display unit 41 to display the therapist comments (step S27).

When the processing of step S27 ends or when it is determined that there are no therapist comments (step S26: No), the processing unit 46 determines whether an image synthesis operation has been performed (step S28). When it is determined that an image synthesis operation has been performed (step S28: Yes), the processing unit 46 performs synthesis processing (step S29). The synthesis processing of step S29 is the same as steps S40 to S47 shown in FIG. 20.

When the processing of step S29 is completed, or when it is determined that no image synthesis operation has been performed (step S28: No), the processing unit 46 determines whether the operation end time has arrived (step S30). The processing unit 46 determines that the operation end time has arrived, for example, when the power to the information providing device 1 has been turned off, or when it is determined that an end operation has been performed by operating an operation unit (not shown) of the information providing device 1.

If the processing unit 46 determines that the operation end timing has not yet arrived (step S30: No), it proceeds to step S20. If the processing unit 46 determines that the operation end timing has arrived (step S30: Yes), it terminates the processing shown in FIG. 19.

Figure 20 is a flowchart showing an example of reservation processing by the processing unit 46 of the terminal device 4 according to the embodiment. As shown in Figure 20, the processing unit 46 accepts the selection of a user image and acquires the user image for which the selection has been accepted (step S40).

Next, the processing unit 46 accepts the specification of one or more keywords (step S41). The processing unit 46 identifies a hair image from the one or more keywords accepted in step S41 (step S42). The processing unit 46 then acquires a similar hair image, which is an image similar to the hair image identified in step S42 (step S43).

Next, the processing unit 46 combines the similar hair image selected from the similar hair images acquired in step S42 with the user's face image (step S44). The processing unit 46 then displays the combined image created in step S44 on the display unit 41 (step S45).

Next, the processing unit 46 determines whether a change operation has been performed (step S46). The change operation is, for example, the first hair image change operation or the second hair image change operation described above. If the processing unit 46 determines that a change operation has been performed (step S46: Yes), it performs image processing according to the change operation to change the composite image (step S47), and then proceeds to step S46.

If the processing unit 46 determines that no change operation has been performed (step S46: No), it estimates the treatment content based on the pre-treatment image and the composite image (step S48). The processing unit 46 determines that no change operation has been performed, for example, when the user U selects the reservation button.

Then, the processing unit 46 outputs a reservation request including the pre-treatment image, the composite image, the estimated treatment details, and the specified treatment details to the hair treatment professional (step S49). The specified treatment details are the treatment details input by the user U. When the processing of step S49 is completed, the processing unit 46 ends the processing shown in FIG. 20.

6. Modifications
In the above-described example, the processing unit 46 displays on the display unit 41 a composite image in which the selected image is changed by user operation in the left-right direction, and a composite image in which the hair length or color, etc. is changed by user operation in the up-down direction, but is not limited to such an example.

For example, the processing unit 46 can cause the display unit 41 to display a composite image in which the selected image has been changed by a user operation in the up or down direction, and can cause the display unit 41 to display a composite image in which the hair length or color, etc. has been changed by a user operation in the left or right direction.

The processing unit 46 can also display on the display unit 41 a composite image of the selected image and the hair length or color, etc., changed by a user operation in a diagonal direction. Note that the user operation is a swipe operation or a scroll operation, but is not limited to these examples and may be, for example, a tap operation or a click operation, or a pinch-in operation or a pinch-out operation.

Furthermore, the hairstyle information output from the third learning model is not limited to hair images. For example, the hairstyle information output from the third learning model may be hair image identification information, which is information for identifying a hair image. Information for identifying a hair image is, for example, identification information for the hair image or keywords for identifying the hair image.

In this case, the third learning model is generated using learning data that includes multiple combinations of information indicating facial features, hair image identification information, and information indicating an evaluation of the combination of information indicating facial features and hair image identification information.

Furthermore, each learning model, including the first learning model, second learning model, and third learning model, may be a learning model for each user U. For example, by using hair images included in the posted images viewed by user U as the hair images used to generate each of the first learning model and the third learning model, each of the first learning model and the third learning model can be generated for each user U.

Furthermore, by using a hair image included in the posted images viewed by user U as the facial image used to generate the second learning model, a second learning model can be generated for each user U.

In cooperation with the terminal device 4, the information providing device 1 can execute some or all of the functions of the acquisition unit 51, identification unit 52, image processing unit 53, estimation unit 54, determination unit 55, comparison unit 56, and output unit 57, and can function as part of an information processing device. Note that hereinafter, a configuration including the terminal device 4 and some or all of the information providing device 1 described above may be referred to as an information processing device.

7. Hardware Configuration
Each of the information providing device 1 and the terminal device 4 according to the embodiment described above is realized by a computer 200 having a configuration as shown in Fig. 21. Fig. 21 is a hardware configuration diagram showing an example of the computer 200 that realizes each of the functions of the information providing device 1 and the terminal device 4 according to the embodiment. The computer 200 has a CPU 201, a RAM 202, a ROM (Read Only Memory) 203, an HDD (Hard Disk Drive) 204, a communication interface (I/F) 205, an input/output interface (I/F) 206, and a media interface (I/F) 207.

The CPU 201 operates based on programs stored in the ROM 203 or HDD 204, and controls each component. The ROM 203 stores a boot program executed by the CPU 201 when the computer 200 starts up, as well as programs that depend on the computer 200's hardware.

HDD 204 stores programs executed by CPU 201 and data used by such programs. Communication interface 205 receives data from other devices via network N (see Figure 2) and sends it to CPU 201, and transmits data generated by CPU 201 to other devices via network N.

The CPU 201 controls output devices such as displays and printers, and input devices such as keyboards and mice, via the input/output interface 206. The CPU 201 acquires data from the input devices via the input/output interface 206. The CPU 201 also outputs data generated via the input/output interface 206 to the output devices.

Media interface 207 reads programs or data stored on recording medium 208 and provides them to CPU 201 via RAM 202. CPU 201 loads the programs from recording medium 208 onto RAM 202 via media interface 207 and executes the loaded programs. Recording medium 208 may be, for example, an optical recording medium such as a DVD (Digital Versatile Disc) or a PD (Phase Change Rewritable Disc), a magneto-optical recording medium such as an MO (Magneto-Optical disk), a tape medium, a magnetic recording medium, or a semiconductor memory.

For example, when computer 200 functions as information providing device 1 or terminal device 4 according to the embodiment, CPU 201 of computer 200 executes programs loaded onto RAM 202 to realize the functions of processing unit 12 and processing unit 46. In addition, data stored in memory unit 11 or memory unit 45 is stored in HDD 204. CPU 201 of computer 200 reads and executes these programs from recording medium 208, but as another example, these programs may be acquired from another device via network N.

[8. Other]
Furthermore, among the processes described in the above embodiments, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically using known methods. In addition, the information including the processing procedures, specific names, various data, and parameters shown in the above documents and drawings can be changed as desired unless otherwise specified. For example, the various information shown in each drawing is not limited to the information shown in the drawings.

Furthermore, the components of each device shown in the figure are functional concepts and do not necessarily have to be physically configured as shown. In other words, the specific form of distribution and integration of each device is not limited to that shown, and all or part of them can be functionally or physically distributed and integrated in any unit depending on various loads, usage conditions, etc.

For example, the information providing device 1 described above may be implemented using multiple server computers, and depending on the function, the configuration can be flexibly changed, such as by calling an external platform using an API or network computing.

Furthermore, the above-described embodiments and variations can be combined as appropriate to the extent that the processing content is not contradictory.

9. Effects
As described above, the information processing device according to the embodiment includes an image processing unit 53 and an estimation unit 54. The image processing unit 53 generates a composite image by combining a user face image, which is a facial image of the user U, with a hair image selected by the user U. The estimation unit 54 estimates the treatment content from the image of the user U before the treatment and the composite image using a learning model that has learned the relationship between the hair images before and after the treatment and the treatment content. This allows, for example, the user U to understand the treatment content required to achieve a desired hairstyle and to specifically communicate the treatment content required to achieve the desired hairstyle to the hairdresser. Therefore, the information processing device can improve the convenience of the user U.

In addition, the learning model is generated by machine learning using learning data that includes multiple combinations of post-treatment hair images and information indicating the treatment details. This allows the information processing device to accurately estimate the treatment details.

The information processing device also includes an output unit 57 that outputs treatment content information including information indicating the treatment content estimated by the estimation unit 54. This allows the information processing device to improve convenience for the user U.

The information processing device also includes a reception unit 50 that receives consent from the user U for the treatment details estimated by the estimation unit 54, and an output unit 57 that, when consent is received by the reception unit 50, outputs a reservation request to the hair salon professional that includes information indicating the treatment details estimated by the estimation unit 54. This allows the information processing device to improve convenience for the user U.

In addition, the output unit 57 outputs a reservation request including the composite image or the hair image selected by the user U to the hair salon professional. This allows the information processing device to improve convenience for the user U.

The information processing device also includes an output unit 57 that outputs information indicating the treatment details estimated by the estimation unit 54 to the hair practitioner, and a revision proposal receiving unit 63 that receives the hair practitioner's revision proposal for the treatment details estimated by the estimation unit 54. The output unit 57 outputs information indicating the revised treatment details received by the revision proposal receiving unit 63. This allows the information processing device to improve convenience for the user U.

The information processing device also includes a determination unit 55 that determines the treatment cost based on the treatment content estimated by the estimation unit 54. This allows the information processing device to improve convenience for user U.

The information processing device also includes a reception unit 50 that receives a treatment specification from the user U, a comparison unit 56 that compares the treatment specification received by the reception unit 50 with the treatment specification estimated by the estimation unit 54, and an output unit 57 that outputs the comparison result from the comparison unit 56. This enables the information processing device to improve convenience for the user U.

The above describes in detail the embodiments of the present application based on the drawings, but this is merely an example, and the present invention can be implemented in other forms that incorporate various modifications and improvements based on the knowledge of those skilled in the art, including the aspects described in the Disclosure of the Invention section.

Furthermore, the "section, module, unit" mentioned above can be read as "means" or "circuit." For example, an acquisition unit can be read as an acquisition means or an acquisition circuit.

REFERENCE SIGNS LIST 1 Information providing device 4 Terminal device 5 Store device 10, 40 Communication unit 11, 45 Storage unit 12, 46 Processing unit 20 User information storage unit 21 Posted information storage unit 22 Model information storage unit 23 Content storage unit 30, 51 Acquisition unit 31 Learning unit 32 Providing unit 41 Display unit 42 Operation unit 43 Imaging unit 44 Sensor unit 50 Receiving unit 52 Identification unit 53 Image processing unit 54 Estimation unit 55 Determination unit 56 Comparison unit 57 Output unit 60 Selection receiving unit 61 Processing receiving unit 62 Designation receiving unit 63 Revision proposal receiving unit 64 Approval acceptance receiving unit 65 Conversion unit 66 Synthesis unit 100 Information processing system

Claims (10)

an image processing unit that generates a composite image by combining a user face image, which is a face image of a user, with a hair image selected by the user;
an estimation unit that estimates a treatment content from the image of the user before the treatment and the composite image using a learning model that has learned the relationship between hair images before and after the treatment and the treatment content. The learning model is
The information processing device according to claim 1 , wherein the information is generated by machine learning using learning data including a plurality of combinations of post-treatment hair images and information indicating treatment details. The information processing device according to claim 1 , further comprising an output unit that outputs treatment content information including information indicating the treatment content estimated by the estimation unit. a reception unit that receives consent from the user for the treatment content estimated by the estimation unit;
3. The information processing device according to claim 1, further comprising: an output unit that, when the acceptance unit accepts the permission, outputs a reservation request including information indicating the treatment content estimated by the estimation unit to a hair salon professional. The output unit
The information processing device according to claim 4, wherein the reservation request including the composite image or the hair image selected by the user is output to the hair salon. an output unit that outputs information indicating the treatment content estimated by the estimation unit to a hair treatment practitioner;
a correction proposal receiving unit that receives a correction proposal from the hair treatment professional for the treatment content estimated by the estimation unit,
The output unit
The information processing device according to claim 1 or 2, further comprising: outputting information indicating the treatment content after the corrections received by the correction proposal receiving unit. The information processing device according to claim 1 or 2, further comprising a determination unit that determines a treatment cost based on the treatment content estimated by the estimation unit. a reception unit that receives a designation of a treatment content by the user;
a comparison unit that compares the treatment content designated by the reception unit with the treatment content estimated by the estimation unit;
The information processing device according to claim 1 , further comprising: an output unit that outputs a comparison result obtained by the comparison unit. 1. A computer-implemented information processing method, comprising:
an image processing step of generating a composite image by combining a user face image, which is a face image of a user, with a hair image selected by the user;
an estimation step of estimating treatment details from the user's pre-treatment image and the composite image using a learning model that has learned the relationship between hair images before and after the treatment and the treatment details. an image processing procedure for generating a composite image by combining a user face image, which is a face image of a user, with a hair image selected by the user;
an information processing program that causes a computer to execute an estimation procedure for estimating treatment details from the user's pre-treatment image and the composite image using a learning model that has learned the relationship between hair images before and after the treatment and the treatment details.