JP7591086B2 - GENERATION APPARATUS, GENERATION METHOD, AND GENERATION PROGRAM - Google Patents

Description

The present invention relates to a generation device, a generation method, and a generation program.

Conventionally, techniques for generating images using AI are known. For example, there is a technique for generating an image that combines features extracted from an image (Patent Document 1 below), a technique for generating an image that has been modified to have a target feature (Patent Document 2 below), and a technique for generating an image that corresponds to text that describes the features of a target image (Patent Document 3 below).

In addition, there are known techniques for generating images from text, such as a technique for generating more fashionable images by editing images (Non-Patent Document 1 below), a technique for generating images of selected clothing being tried on (Non-Patent Document 2 below), a technique for generating an image by providing an image to be changed and a description of the image to be generated, and editing the image to resemble the description (Non-Patent Document 3 below), and a technique for making it possible to recall the target of change by having a model that generates images from a description perform additional learning of the image to be changed (Non-Patent Document 4 below).

There is also a known technology that uses Visual-Semantic Embedding (VSE) technology to interpret fashion terms. For example, there is a known technology (see Non-Patent Document 5 below) that quantifies abstract terms specific to fashion together with a full-body image, making it possible to interpret each term from various angles.

JP 2020-112907 A JP 2022-157424 A JP 2022-172173 A

WL. Hsiao, I. Katsman, CY. Wu, D. Parikh, K. Grauman“Fashion++: Minimal Edits for Outfit Improvement”, [online], [Retrieved February 6, 2023], Internet <https://arxiv.org/abs/1904.09261> X. Han, Z. Wu, Z. Wu, R. Yu, L.S. Davis “VITON: An Image-based Virtual Try-on Network”, [online], [searched on February 6, 2020], Internet < https://arxiv.org/abs/1711.08447＞ B. Kawar, S. Zada, O. Lang, O. Tov, H. Chang, T. Dekel, I. Mosseri, M. Irani “Imagic: Text-Based Real Image Editing with Diffusion Models”, [online], [ Searched on February 6, 2020], Internet <https://arxiv.org/abs/2210.09276> N. Ruiz, Y. Li, V. Jampani, Y. Pritch, M. Rubinstein, K. Aberman “DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation”, [online], [Reiwa 5 [Retrieved February 6, 2017], Internet <https://arxiv.org/abs/2208.12242> R. Shimizu, Y. Saito, M. Matsutani, M. Goto“Fashion intelligence system: An outfit interpretation utilizing images and rich abstract tags”, [online], [Retrieved February 6, 2023], Internet <https://www.sciencedirect.com/science/article/pii/S0957417422021856?via%3Dihub>

However, conventional technology has been unable to provide images that properly interpret and reflect the abstract terms specific to fashion.

This application was made in light of the above, and aims to provide images that appropriately interpret and reflect abstract terms specific to fashion.

The generating device according to the present application is characterized by having an acquisition unit that acquires information regarding the similarity between a predetermined keyword and the differences between a plurality of training images in a distributed representation space into which the training images and the keywords are projected, and a generating unit that trains a model to learn the relationship between the plurality of training images and the predetermined keyword based on the information regarding the similarity acquired by the acquisition unit.

According to one aspect of the embodiment, it is possible to provide images that appropriately interpret and reflect abstract terms specific to fashion.

FIG. 1 is a diagram illustrating an example of the configuration of an information processing system according to an embodiment. FIG. 2 is a diagram (1) showing an example of information processing according to the embodiment. FIG. 3 is a diagram (2) showing an example of information processing according to the embodiment. FIG. 4 is a diagram (3) showing an example of information processing according to the embodiment. FIG. 5 is a diagram illustrating an example of the configuration of a user terminal according to the embodiment. FIG. 6 is a diagram illustrating an example of the configuration of a generating device according to an embodiment. FIG. 7 is a diagram illustrating an example of a user information storage unit according to the embodiment. FIG. 8 is a diagram illustrating an example of an image information storage unit according to the embodiment. FIG. 9 is a flowchart illustrating an example of information processing according to the embodiment. FIG. 10 is a flowchart illustrating an example of information processing according to the embodiment. FIG. 11 is a flowchart illustrating an example of information processing according to the embodiment. FIG. 12 is a hardware configuration diagram illustrating an example of a computer that realizes the functions of the generation device.

Below, a detailed description will be given of a form for implementing the generating device, generating method, and generating program of the present application (hereinafter, referred to as "embodiments") with reference to the drawings. Note that the generating device, generating method, and generating program of the present application are not limited to these embodiments. Furthermore, the same parts in the following embodiments are given the same reference numerals, and duplicated descriptions will be omitted.

(Embodiment)
In the fashion industry, there are many abstract and ambiguous terms that are specific to the field, such as "casual,""formal," and "adult casual." This ambiguity is thought to prevent users from taking a deep interest in fashion. For example, it may be a barrier for users to try new fashion. Therefore, if such terms can be quantified, it is thought that the difficulty in users' choices and actions caused by the abstract terms specific to the fashion field can be reduced, which may promote further development of the fashion industry as a whole. For example, it is possible to prepare appropriate answers for users who want to know what is "street" about this fashion, how "street" it is, and how to make it more "street."

Research has been conducted to use VSE technology to interpret fashion terms. For example, a technology (Non-Patent Document 5 above) is known that quantifies fashion-specific abstract terms such as "casual," "adult casual," and "formal" along with full-body images, making it possible to interpret each term from various angles. However, while this technology can select and suggest suitable outfits as search results from existing outfits available in a specific online shopping mall, it cannot suggest items that are not available in the specific online shopping mall, or new ways or angles of wearing the same items.

Research is also underway into the use of image generation technologies such as GAN (Generative Adversarial Network), diffusion models, and VAE (Variational Autoencoder) to interpret what is fashionable. For example, a technology is known that modifies an original fashion image to generate a more fashionable image (see Non-Patent Document 1 above). However, this technology was unable to generate fashion images that correspond to ambiguous descriptions such as "casual," "formal," "adult casual," and "wedding."

In recent years, research into the interpretation of fashion terms has been recognized as a completely new research field that differs from traditional research areas, and research using VSE technology to interpret fashion terms is becoming increasingly important.

This application was made in light of the above, and aims to provide images that appropriately interpret and reflect abstract terms specific to fashion.

1. Configuration of the information processing system
A generating system 1 shown in Fig. 1 will be described. As shown in Fig. 1, the generating system 1 includes a user terminal 10 and a generating device 100. The user terminal 10 and the generating device 100 are connected to each other via a predetermined communication network (network N) so as to be able to communicate with each other by wire or wirelessly. Fig. 1 is a diagram showing an example of the configuration of the generating system 1 according to an embodiment.

The user terminal 10 is an information processing device used by a user. The user terminal 10 is realized, for example, by a smartphone, a tablet terminal, a notebook PC (Personal Computer), a desktop PC, a mobile phone, a PDA (Personal Digital Assistant), or the like. The user terminal 10 also displays information distributed by the generating device 100 or a server device that provides a specified service, using a web browser or an application. Note that the example shown in FIG. 2 shows a case where the user terminal 10 is a smartphone.

The generating device 100 is an information processing device that accepts fashion images composed of a combination of multiple clothes (also called fashion items, including footwear (also called shoes), headwear (such as caps and hats), and accessories (also called accessories)) input by a contributor or user, and fashion terms that serve as keywords for image generation, and projects them into a distributed representation space that has been learned in advance. For example, this can be realized by a server device, a cloud system, or the like. In addition, images and keywords (including tags, etc.) are projected into the distributed representation space according to the embodiment, and the closeness between images, images and keywords, and keywords and keywords are projected. The closer they are, the closer they are in the distributed representation space. In other words, the similarity between images, images and keywords, and keywords and keywords can be measured as a distance in the distributed representation space. Note that the closeness in the distributed representation space according to the embodiment may be closeness based on Euclidean distance or closeness based on cosine distance.

For example, the generating device 100 provides an electronic commerce service that provides (searches, sells, etc.) clothing. The generating device 100 also provides a coordination service that accepts submissions of content (images, videos, articles, etc.) showing clothing coordination from users and provides (searches, distributes, etc.) the content to other users.

The generating device 100 may also have a function as a web server that provides a website related to the service. The generating device 100 may also be a device that distributes, to the user terminal 10, information to be displayed in applications related to various services installed in the user terminal 10. The generating device 100 may also be a device that distributes the application data itself.

The generating device 100 may also function as a distribution device that distributes control information to the user terminal 10. Here, the control information is described, for example, in a script language such as JavaScript (registered trademark) or a style sheet language such as CSS (Cascading Style Sheets). Note that the application itself distributed from the generating device 100 may be regarded as control information.

2. An example of information processing
Next, an example of information processing implemented by the generating device and the like according to the embodiment will be described with reference to FIGS.

[2-1. An example of an image generation model]
Here, the generating device 100 executes a process of generating a new image based on an image and a keyword (for example, a keyword including an ambiguous fashion term). For example, the generating device 100 generates a new image using a model that has been trained to generate an image corresponding to an input keyword from an input image using an image generation method using a diffusion model such as stable diffusion. An example of such a model will be described below. Note that the generating device 100 can employ any model as long as it can generate an image with content corresponding to a keyword.

For example, a learning device (which may be the generation device 100) that learns a model acquires a pair of a certain learning image and text that explains the learning image as learning data (also called teacher data). In such a case, the learning device prepares multiple noise images in which noise is added to the learning image in stages. Then, when the noise image and the text to be annotated are input to the model, the learning device learns to restore an image with one level less noise. By repeatedly executing such processing, the learning device can obtain a model that can generate an image whose content corresponds to the input text from any noise image.

As a derivative of this model, it is possible to train a model that converts an input image into an image that corresponds to text. For example, the training device acquires a first training image, which is an image of a user wearing a black coat, and a second training image, which is an image of a user wearing a red coat. Then, when the first training image and text such as "red coat" are input, the training device trains the model to output the second training image, thereby training a model that converts the content of the input image into an image whose content corresponds to the text.

Here, it is conceivable that the above-mentioned model could be used to realize an e-commerce service related to wearable items (fashion). Furthermore, when training such a model, not only images related to fashion are required, but also text corresponding to the image content and text showing the relationship between images. Thus, by executing the following process, the generating device 100 can provide an e-commerce service using the above-mentioned model (a model that converts input images according to text), extract and use data used to train the model, evaluate the model, etc.

For example, the generating device 100 extracts two images from the VSE space and identifies the difference between the vectors of the images and similar keywords. More intuitively, when images A and B are projected onto the VSE space and image A is projected in the direction of the keyword "adult casual" as viewed from the position of image B, the generating device 100 determines that the image A is an image in which image B is "adult casual". The generating device 100 then trains the model so that when image B and the keyword "adult casual" are input to the model, the model outputs image A. By repeating such processing, the generating device 100 can realize training of a model that outputs an image in which the wearable item photographed in any input image is converted into wearable items with the atmosphere of any input keyword.

[2-2. Regarding the first information processing]
Hereinafter, a first information process implemented by the generating device and the like according to the embodiment will be described with reference to Fig. 2. Fig. 2 is a diagram illustrating an example of the information process according to the embodiment.

User U1 is a user who has a desire to know examples of adult casual coordination (note that these may be examples of coordination that do not actually exist). User U1, for example, specifies image G1 and desires an image of a coordination that makes the coordination of image G1 more "adult casual." User U1 also specifies (for example, inputs) the keyword "adult casual" to perform a product search. The generating device 100 acquires image G1 (step S101). Image G1 may be, for example, an image of user U1's own coordination that combines items owned by user U1 (for example, an image posted on a specific site or a snapshot of user U1), or an image of a coordination on a specific site that user U1 has registered as a favorite (for example, a snapshot of a model wearing the clothing).

The generating device 100 also acquires the keyword "adult casual" specified by the user U1 (step S102). The generating device 100 then generates fashion images G11 to G15 from image G1 based on the acquired keyword "adult casual" (step S103). For example, the generating device 100 generates fashion images by modifying part of the fashion in image G1 to make it more "adult casual."

When the image G1 is a coordinated image of the user, the generating device 100 generates a fashion image in which the user U1 wears the coordinated outfit generated according to the keyword "adult casual" by inputting the coordinated image and the keyword "adult casual" (an example of style conversion instruction information in natural language) into the model. This allows the generating device 100 to realize assistance in designing a coordinated outfit based on the items the user owns and suited to the time, place, and occasion (at this time, a completely new image that does not actually exist may be generated). When the image G1 is a coordinated image on a specific site, the generating device 100 generates a fashion image in which the model wears the coordinated outfit generated according to the keyword "adult casual" by inputting the coordinated image and the keyword "adult casual" into the model. This allows the generating device 100 to promote the user's desire to buy by showing a policy of incorporating favorite items into the user's preferred style (at this time, a completely new image that does not actually exist may be generated). The generating device 100 can also change the outfit to one suitable for the destination or scene based on the items the user owns or the outfit that the user likes.

In addition, if user U1 wishes to "change only the upper body to make it adult casual," the generating device 100 may accept a designation of the conversion part from user U1. Then, the generating device 100 may generate a fashion image by changing only the conversion part designated by user U1. If image G1 is the user's own coordinated outfit image, for example, the generating device 100 inputs the coordinated outfit image, the keyword "adult casual," and the conversion part designation of the "upper body" into the model, and generates a fashion image in which user U1 wears an outfit generated according to the keyword "adult casual" and the conversion part designation of the "upper body." In this way, the generating device 100 can realize assistance in coming up with outfits that suit the time, place, and occasion based on items owned by the user. Furthermore, if image G1 is a coordinated image on a specific site, for example, generating device 100 inputs the coordinated image, the keyword "adult casual", and the designated conversion part of the "upper body" to the model, and generates a fashion image in which the model wears the coordinated image generated according to the keyword "adult casual" and the designated conversion part of the "upper body". In this way, generating device 100 can encourage the user's purchasing motivation by showing the user a policy of incorporating favorite items into the user's preferred style. Furthermore, generating device 100 can make it possible to specify which specific parts the user wants to change, making it easier for the user to understand how to wear items with a preferred atmosphere based on items the user already owns or favorite clothes.

In addition to the request that the user U1 "changes only the upper body to make it adult casual," the user U1 may also have a request to "search for similar items (purchase items included in the adult casual coordination, search for items included in the adult casual coordination among items owned by the user U1)." In this case, the generating device 100 may perform processing to display actual items similar to the generated items. For example, the generating device 100 may input an image G1 (which may be the user's own coordinated image or a coordinated image on a specified site), a keyword "adult casual," and a conversion part designation of the "upper body" to the model, thereby generating a fashion image in which the user U1 or a wearing model wears a coordination generated according to the keyword "adult casual" and the conversion part designation of the "upper body," and may provide a link to product details of similar items on a specified site. In this way, the generating device 100 can contribute to increasing traffic and sales by inducing purchases to the specified site. In addition, the generating device 100 can display items similar to items included in the generated coordination, so that a service can be provided to lead to the purchase of items that actually exist. After the user is shown how to wear the item with items that they actually own (if the model is the user, the user can see how the item will look after actually wearing it), it is highly likely that the item will be selected if the user likes it. It is also highly likely that the item will lead to a purchase, which may have a positive impact on sales. In addition, the generating device 100 may input, for example, an image G1 (which may be the user's own coordinated image or a coordinated image on a specified site), the keyword "adult casual", and the conversion part designation of the "upper body" to the model, to generate a fashion image of the coordinate generated according to the keyword "adult casual" and the conversion part designation of the "upper body" worn by the user U1 or the model wearing the item, and provide a link to the product details of similar items owned. In this way, the generating device 100 can contribute to increasing traffic by guiding the user to a specified service (such as a service that pre-registers and manages items owned by the user U1).

The details of the fashion image generation process will be described later. In FIG. 2, images G11 to G15 are generated as generated images. Here, the generating device 100 may provide the generated fashion image to the user U1. In FIG. 2, the generating device 100 performs a product search using the generated fashion image (step S104). Specifically, the generating device 100 searches for a similar image similar to the generated fashion image (a similar image similar to the entire fashion image or a similar image similar to a part of the fashion image (for example, an upper body item part)). For example, the generating device 100 searches for a similar image from the product details of a product handled in a specified online shopping mall. At this time, the generating device 100 may provide the generated fashion image to the user U1, accept the selection of the fashion image, and search for a similar image similar to the selected fashion image. Then, the generating device 100 provides the search result to the user U1 (step S105). The generating device 100 can also estimate how close the specified keyword is. For this reason, the generating device 100 may provide user U1 with information indicating how "adult casual" each fashion image is (for example, "Fashion image G11: "adult casual" rating of 90 points, fashion image G12: "adult casual" rating of 80 points, ...") along with each generated fashion image. Furthermore, the generating device 100 may provide user U1 with each generated fashion image in order of proximity to "adult casual". For example, if fashion image G11 has an "adult casual" rating of 90 points and fashion image G12 has an "adult casual" rating of 80 points, the generating device 100 may provide the fashion images in the order of fashion image G11, fashion image G12.

[2-3. Regarding the second information processing]
Hereinafter, the second information processing realized by the generation device etc. according to the embodiment will be described with reference to Fig. 3. Fig. 3 is a diagram showing an example of the information processing according to the embodiment. In the first information processing, a case was shown in which a user specifies a keyword, and a fashion image is generated based on the specified keyword, but in the second information processing, tags are prepared in advance, and a user selects a tag from the prepared tags, and a fashion image is generated based on the selected tag. Therefore, in the second information processing, it becomes possible to generate a fashion image by selecting a tag instead of a user inputting a keyword.

User U1 is a user who has a desire to know "examples of military-style outfits suitable for a wedding after-party (note that these may be examples of outfits that do not actually exist)." User U1 specifies (e.g., selects) a tag and wishes to generate an outfit image and search for products from the tag. The generation device 100 acquires tag information T1 (step S201). Tag information T1 includes information indicating that user U1 has specified the tags "female," "30s," "military," and "wedding after-party" from among the tags that can be specified.

Based on the acquired tag information T1, the generating device 100 also identifies tags of "female", "30s", "military", and "wedding after-party" specified by the user U1 (step S202). The generating device 100 then generates a fashion image based on the identified tags of "female", "30s", "military", and "wedding after-party" (step S203). For example, the generating device 100 generates a fashion image so as to satisfy all tags of "female", "30s", "military", and "wedding after-party". Note that, as with the first information processing, the details of the fashion image generation processing will be described later. In FIG. 3, images G21 to G25 are generated as generated images. Here, the generating device 100 may provide the generated fashion image to the user U1. In FIG. 3, the generating device 100 performs a product search using the generated fashion image (step S204). Specifically, the generating device 100 searches for similar images similar to the generated fashion image. For example, the generating device 100 searches for similar images among the product details of products handled in a specific online shopping mall. At this time, the generating device 100 may provide the generated fashion image to the user U1, accept the selection of a fashion image, and search for similar images similar to the selected fashion image. The generating device 100 then provides the search results to the user U1 (step S205).

Note that in the second information processing, unlike the first information processing, the user U1 does not specify an image that serves as a base for image generation. However, as in the first information processing, the user U1 may specify an image that serves as a base for image generation. Then, the generation device 100 may generate an image based on the specified image and the specified tag.

[2-4. Regarding the third information processing]
Hereinafter, the third information processing realized by the generating device and the like according to the embodiment will be described with reference to Fig. 4. Fig. 4 is a diagram illustrating an example of the information processing according to the embodiment.

The generating device 100 provides the user U1 with the content C1, which allows the user U1 to select whether or not to allow the display of the generated image when generating a fashion image (step S301). The timing of providing the content C1 to the user U1 may be any time and is not particularly limited. The user U1 selects whether or not to allow the display of the generated image from the content C1 (step S302). When the generating device 100 acquires the selection result from the user U1 (step S303), it performs a display process of the generated image. Specifically, the generating device 100 provides the generated image to the user U1 (step S304) and displays it on the user terminal 10 (step S305). In FIG. 4, if the user U1 does not allow the display of the generated image, a coordinated image using only items that actually exist is displayed in the region R1, and if the user U1 allows the display of the generated image, a coordinated image that may include items that do not actually exist is displayed in the region R1. In addition, when the generating device 100 receives a request to change the background or person in the content C1, it may change the background or person specified by the user U1. This allows the generating device 100 to display the generated image on the service with the user's consent.

[3. Image Generation Processing]
The details of the fashion image generation process will be described below. The fashion image generation process will be described using five approaches as examples, but the present invention is not limited to these examples. Among the five approaches, the first and second approaches use the VSE space, and the third, fourth, and fifth approaches do not use the VSE space. Each approach will be described.

[3-1. Regarding the first approach]
In the above-mentioned non-patent document 5, a technique is known in which a learning image and a keyword are projected onto a distributed representation space using a VSE technique for interpreting fashion terms. The generating device 100 may generate fashion images using a model that has learned the relationship between a plurality of learning images and a predetermined keyword based on the similarity between the difference (vector) of a plurality of learning images in the distributed representation space into which the learning image and the keyword are projected and the predetermined keyword. In approach 1, the generating device 100 generates fashion images using a model (a model prepared without using a VSE space) that is individually prepared in advance. Then, the generating device 100 trains the model by evaluating the generation result using this model using the VSE space. For example, when image A and a keyword are input to the model and image B is output, the generating device 100 projects image A, image B, and the keyword onto the VSE space. Then, the generating device 100 judges whether the model has been correctly learned by evaluating how close the difference between the vector of image A and the vector of image B is to the vector of the keyword. The generating device 100 may also generate a plurality of images and perform model learning by evaluating whether each image is converted in the direction of the keyword. The generating device 100 may also, for example, propose a coordination that may be perceived as "adult casual" for each user. For example, the generating device 100 may propose a coordination that may be "perceived as adult casual in the Late-20s" depending on whether the difference is close to the vector resulting from combining "adult casual" and "Late-20s". The generating device 100 may also, for example, generate fashion images of various patterns, accept evaluations (feedback) from users on the generated images, and reflect the evaluation results in the VSE space or in the model. In this case, the generating device 100 may select a generated image by evaluating to what extent each generated image satisfies the change requirements.

[3-2. Regarding the second approach]
In the first approach, a case where a model is trained from prepared data without using a VSE space has been described. In the second approach, however, when there is no prepared data, a model is trained by creating pseudo data using a VSE space. The generating device 100 may generate a fashion image using a model trained based on whether or not the generated fashion image is an image moved in a predetermined direction based on a predetermined keyword in the distributed representation space. The generating device 100 may train a model by, for example, selecting an image in which the target image is made "adult casual" in the distributed representation space. For example, the generating device 100 acquires a target image from a user. Then, the generating device 100 identifies an image in the vicinity of a position where a vector of an image similar to the acquired target image is moved in the vector direction of "adult casual" from the distributed representation space. Then, the generating device 100 trains a model so that when the acquired target image and the keyword "adult casual" are input, the model outputs the image identified from the distributed representation space.

[3-3. The third approach]
The third approach is an approach in the case where data is prepared in which images are annotated in advance as images corresponding to the learning images and keywords. For example, an image labeled with "if the target image is made adult casual, it will look like this" is prepared in advance. The generating device 100 may generate fashion images using a model that has been trained so that when the learning images and keywords are input, an image generated from the learning images is similar to the images annotated in advance as images corresponding to the learning images and keywords.

[3-4. The fourth approach]
The fourth approach is an approach in which fashion images are generated using a model trained on user information (e.g., user identification information, attribute information, etc.) in addition to learning images and keywords. The generating device 100 may generate fashion images using a model trained so that when learning images, keywords, and user information are input, an image generated from the learning images resembles an image that is predetermined as an image corresponding to the learning images, keywords, and user information. This allows the generating device 100 to generate an image that is likely to be perceived as "more adult casual" for each user, for example. In this case, the generating device 100 may receive a designation of the strength of the desired degree of "adult casual" and generate an image according to the degree of the designated strength.

[3-5. The fifth approach]
The fifth approach is an approach in which various images corresponding to a keyword are generated without specifying a base image. The generating device 100 may generate a fashion image using a model that has been trained so that an image generated when a keyword is input is similar to an image that is predetermined as an image corresponding to the keyword. This increases the possibility that a user (e.g., a designer) who is looking for novelty or originality will be able to encounter something new, and thus promotes improvement of usability for such users. The generating device 100 may also generate a fashion image taking into account user information (e.g., attribute information, etc.). For example, the generating device 100 may generate a fashion image suitable for attributes such as the user's age and gender. The generating device 100 may generate a fashion image using a model that has been trained including user information such as attribute information, as in the fourth approach.

[3-6. Other matters]
The above approaches (the first to fifth approaches) may be adopted in any suitable combination in any manner.

[4. Variations in Information Processing]
In the above embodiment, the generating device 100 may separate a mode in which a generated image is displayed from a mode in which an image that actually exists similar to the generated image is displayed. Furthermore, the generating device 100 may be applied to an application in which the generated image can be displayed as is with the user's consent by performing such separation. Furthermore, the generating device 100 may wish to display an item that does not actually exist, and may provide information on whether or not there is a similar item. For example, the generating device 100 may provide information on the extent to which the design of the generated image differs from that of an item (or coordination) that is already on sale. For example, the generating device 100 may guarantee that there are no problems such as copyrights regarding the item of the generated image.

5. User terminal configuration
Next, the configuration of the user terminal 10 according to the embodiment will be described with reference to Fig. 5. Fig. 5 is a diagram showing an example of the configuration of the user terminal 10 according to the embodiment. As shown in Fig. 5, the user terminal 10 has a communication unit 11, an input unit 12, an output unit 13, and a control unit 14.

(Communication unit 11)
The communication unit 11 is realized by, for example, a network interface card (NIC) etc. The communication unit 11 is connected to a predetermined network N by wire or wirelessly, and transmits and receives information to and from the generating device 100 etc. via the predetermined network N.

(Input unit 12)
The input unit 12 accepts various operations from a user. In Fig. 2, the input unit 12 accepts various operations from a user U1. For example, the input unit 12 may accept various operations from a user via a display screen using a touch panel function. The input unit 12 may also accept various operations from buttons provided on the user terminal 10 or a keyboard or mouse connected to the user terminal 10.

(Output unit 13)
The output unit 13 is a display screen of a tablet terminal or the like realized by, for example, a liquid crystal display or an organic EL (Electro-Luminescence) display, and is a display device for displaying various information. For example, the output unit 13 displays information transmitted from the generating device 100.

(Control unit 14)
The control unit 14 is, for example, a controller, and is realized by a central processing unit (CPU), a micro processing unit (MPU), or the like executing various programs stored in a storage device inside the user terminal 10 using a random access memory (RAM) as a working area. For example, the various programs include application programs installed in the user terminal 10. For example, the various programs include application programs that display information transmitted from the generating device 100. The control unit 14 is also realized by an integrated circuit, for example, an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA).

As shown in FIG. 5, the control unit 14 has a receiving unit 141 and a transmitting unit 142, and realizes or executes the information processing functions described below.

(Receiving unit 141)
The receiving unit 141 receives various information from other information processing devices such as the generating device 100. For example, the receiving unit 141 receives a generated image generated by the generating device 100. In addition, for example, the receiving unit 141 receives an image similar to the generated image searched for based on the generated image.

(Transmitter 142)
The transmission unit 142 transmits various information to other information processing devices such as the generating device 100. For example, the transmission unit 142 transmits an image specified in response to a user's operation. Also, for example, the transmission unit 142 transmits a keyword specified in response to a user's operation. Also, for example, the transmission unit 142 may transmit a tag specified in response to a user's operation. Also, for example, the transmission unit 142 may transmit user information, etc.

6. Configuration of the Generation Device
Next, a configuration of the generating device 100 according to the embodiment will be described with reference to Fig. 6. Fig. 6 is a diagram showing an example of the configuration of the generating device 100 according to the embodiment. As shown in Fig. 6, the generating device 100 has a communication unit 110, a storage unit 120, and a control unit 130. Note that the generating device 100 may have an input unit (e.g., a keyboard, a mouse, etc.) that accepts various operations from an administrator of the generating device 100, and a display unit (e.g., a liquid crystal display, etc.) that displays various information.

(Communication unit 110)
The communication unit 110 is realized by, for example, a NIC etc. The communication unit 110 is connected to a network N by wire or wirelessly, and transmits and receives information to and from the user terminal 10 etc. via the network N.

(Memory unit 120)
The storage unit 120 is realized by, for example, a semiconductor memory element such as a RAM or a flash memory, or a storage device such as a hard disk or an optical disk. As shown in FIG. 6 , the storage unit 120 has a user information storage unit 121 and an image information storage unit 122.

The user information storage unit 121 stores user information (e.g., attribute information such as age and gender, owned item information inferred from purchase history and posting history, etc.). Here, FIG. 7 shows an example of the user information storage unit 121 according to an embodiment. As shown in FIG. 7, the user information storage unit 121 has items such as "user ID" and "user information."

"User ID" refers to identification information for identifying a user. "User information" refers to user information. In the example shown in FIG. 7, conceptual information such as "User information #1" and "User information #2" is stored in "User information", but in reality, information such as age and gender, and identification information for items owned by the user are stored.

The image information storage unit 122 stores image information used when searching for similar images (for example, image information included in the product details of an item handled by a specific online shopping mall or image information included in the product details of an item owned by user U1). Here, FIG. 8 shows an example of the image information storage unit 122 according to the embodiment. As shown in FIG. 8, the image information storage unit 122 has items such as "image ID" and "image information".

"Image ID" indicates identification information for identifying an image. "Image information" indicates image information. In the example shown in FIG. 8, conceptual information such as "Image information #1" and "Image information #2" is stored in "Image information", but in reality, image data is stored. Alternatively, a URL (Uniform Resource Locator) where the image data is located, or a file path name indicating the storage location of these, may be stored.

(Control unit 130)
The control unit 130 is a controller, and is realized, for example, by a CPU, an MPU, or the like, executing various programs stored in a storage device inside the generating device 100 using a RAM as a working area. The control unit 130 is also realized, for example, by an integrated circuit such as an ASIC or an FPGA.

As shown in FIG. 6, the control unit 130 has an acquisition unit 131, a generation unit 132, a search unit 133, and a provision unit 134, and realizes or executes the information processing action described below. Note that the internal configuration of the control unit 130 is not limited to the configuration shown in FIG. 6, and may be other configurations as long as they perform the information processing described below.

(Acquisition unit 131)
The acquisition unit 131 acquires various pieces of information from an external information processing device. The acquisition unit 131 acquires various pieces of information from other information processing devices such as the user terminal 10.

The acquisition unit 131 acquires various pieces of information from the storage unit 120. The acquisition unit 131 also stores the acquired various pieces of information in the storage unit 120.

The acquisition unit 131 acquires fashion terminology keywords. For example, the acquisition unit 131 acquires a keyword specified by the user (such as "adult casual"). For example, if a user desires to know examples of adult casual coordination and performs a search using the keyword "adult casual", the acquisition unit 131 acquires the keyword "adult casual" entered by the user.

The acquisition unit 131 may acquire an image that serves as a base for image generation. For example, when the user desires to know a more "adult casual" coordination of the coordination of the specified image designated by the user, the acquisition unit 131 may acquire the specified image designated by the user. At this time, when the user designates a part to be changed (for example, when the user desires to know an coordination in which "only the upper body" is changed), the acquisition unit 131 may acquire information on the part to be changed). In addition, the acquisition unit 131 may acquire information on a distributed representation space acquired from a model such as VAE onto which learning images and keywords are projected.

The acquisition unit 131 may acquire tags that are the basis for generating an image. For example, when a user desires to know an outfit that satisfies a specific tag by specifying the specific tag, the acquisition unit 131 may acquire the specific tag specified by the user.

(Generation unit 132)
The generation unit 132 generates one or more generated images based on the keyword acquired by the acquisition unit 131. For example, the generation unit 132 generates a fashion image from a predetermined image based on the acquired keyword "adult casual." For example, the generation unit 132 generates a fashion image by modifying a part of the fashion of the predetermined image to be more "adult casual" based on the acquired keyword "adult casual."

The generating unit 132 may generate a fashion image that satisfies a specific tag designated by the user, based on the specific tag. For example, if a user has a request such as "I would like to know examples of military-style coordination suitable for a wedding after-party" and designates the tags "female", "30s", "military", and "wedding after-party", the generating unit 132 may generate a fashion image that satisfies all of the tags "female", "30s", "military", and "wedding after-party".

The generation unit 132 may input a predetermined image and a keyword to the model, and generate a fashion image in which a user or a model wears an outfit generated according to the keyword. The generation unit 132 may also input a predetermined image, a keyword, and a conversion part designation to the model, and generate a fashion image in which a user or a model wears an outfit generated according to the keyword and the conversion part designation.

The generation unit 132 may train a model on the relationship between a plurality of learning images and a predetermined keyword based on the similarity between the difference between the plurality of learning images in the distributed representation space and the predetermined keyword. The generation unit 132 may also train the model based on whether the image is moved in a predetermined direction based on a predetermined keyword in the distributed representation space. In this case, the generation unit 132 may train the model only when the image is not moved in such a predetermined direction. The generation unit 132 may also train the model so that an image generated from the learning image when the learning image and the keyword are inputted is similar to an image that is pre-annotated as an image corresponding to the learning image and the keyword. The generation unit 132 may also train the model so that an image generated from the learning image when the learning image, the keyword, and user information are inputted is similar to an image that is pre-defined as an image corresponding to the learning image, the keyword, and user information. The generation unit 132 may also train the model so that an image generated when the keyword is inputted is similar to an image that is pre-defined as an image corresponding to the keyword.

(Search unit 133)
The search unit 133 searches for similar images that are similar to the fashion image generated by the generation unit 132. For example, the search unit 133 searches for similar images from among product details of products handled in a predetermined online shopping mall. Also, for example, the search unit 133 accepts a selection of a fashion image generated by the generation unit 132, and searches for similar images that are similar to the selected fashion image.

(Providing Unit 134)
The providing unit 134 provides information based on the generated image generated by the generating unit 132. For example, the providing unit 134 provides the generated image generated by the generating unit 132. Furthermore, for example, the providing unit 134 provides a similar image that is similar to the generated image generated by the generating unit 132 (i.e., a search result by the searching unit 133).

[7. Information processing flow]
Next, the procedure of information processing by the generation system 1 according to the embodiment will be described with reference to Figs. 9 to 11. Figs. 9 to 11 are flowcharts showing the procedure of information processing by the generation system 1 according to the embodiment. Fig. 9 is a flowchart showing the generation process of a fashion image, and Figs. 10 and 11 are flowcharts showing the learning process of a model.

As shown in FIG. 9, the generating device 100 acquires fashion term keywords (step S401).

The generating device 100 generates a fashion image from a specified image based on the acquired keywords (step S402).

The generating device 100 provides the generated fashion image (step S403).

As shown in FIG. 10, the generating device 100 acquires information regarding the similarity between the differences between multiple training images in the distributed representation space and a predetermined keyword (step S501).

Based on the acquired similarity information, the generating device 100 trains the model to learn the relationships between the multiple learning images and the specified keywords (step S502).

As shown in FIG. 11, the generating device 100 generates pseudo data of a fashion image using a distributed representation space (step S601).

The generating device 100 trains the model based on whether the generated fashion image is an image that has been moved in a specific direction based on a specific keyword in the distributed representation space (step S602).

8. Modifications
The generation system 1 according to the embodiment described above may be implemented in various different forms other than the above embodiment. Therefore, other embodiments of the generation system 1 will be described below.

(Displaying fashion images that are not similar to each other)
When displaying multiple fashion images close to a keyword, simply displaying the top fashion images (e.g., the top five fashion images) may result in all of the fashion images being similar. The generating device 100 may display multiple fashion images that are dissimilar to each other among the fashion images close to the keyword. Specifically, the generating device 100 displays the fashion image that is closest to the keyword. Then, if the fashion image next closest to the keyword is similar to the first fashion image (e.g., the closest fashion image), the generating device 100 may not display that fashion image.

(Display of fashion images based on tag selection after free entry)
In the above embodiment, the first information processing allows the user to freely enter keywords, and the second information processing allows the user to select tags. However, the user may first enter keywords in a free-form format, and then select tags when fine-tuning the style. That is, the generating device 100 may allow the user to freely enter keywords and then select tags. For example, when the user wants to know a combination of styles, the user may first enter keywords in a free-form format (e.g., casual military), but if the resulting proposed outfit is different from the user's image, the user may want to modify the keywords (e.g., only military). In the case of only the first information processing, the user must delete all keywords and enter new keywords. However, if the user can switch to tag selection, the user can fine-tune the style by simply removing unnecessary tags (e.g., casual tags). In this way, the user may be able to switch from free-form format to tag selection when inputting an expression that the user wants to approximate.

(Display of fashion images using submitted images)
In the above embodiment, instead of generating a fashion image close to the keyword, the generating device 100 may find and display a fashion image close to the keyword (for example, by calculating in the VSE space) from the posted images of other users (for example, a coordinated image using real items). In this way, the generating device 100 may display the generated fashion image or the posted images of other users. In addition, the generating device 100 may determine which one to display preferentially according to the display priority specified by the user. For example, when there is a posted image of another user close to the keyword, if the body type is too different, the generating device 100 may determine to display the generated fashion image. Note that, when the generating device 100 uses the posted image of another user, it is not necessary to perform a similar image search to identify the item by linking information of the real item used by the other user to the posted image. The generating device 100 may accept input of an expression to be approximated by free entry or tag selection, and may display posted images of other users or edited images of the target image (images generated from a base image designated by the user) as a target image when the target image is approximated to the input expression. In this case, for example, the generating device 100 may display posted images of other users or edited images of the target image according to the display priority. Also, for example, when displaying posted images of other users, the generating device 100 may display item information of items used in the posted images, and when displaying edited images of the target image, the generating device 100 may display item information of items similar to the items used in the edited images. Also, for example, the generating device 100 may display edited images of posted images of other users as a target image. Also, the generating device 100 may increase the display priority of edited images of the target image when there are few posted images of other users with similar physical features, not limited to the input of the user.

8. Effects
As described above, the generating device 100 according to the embodiment includes the acquiring unit 131 and the generating unit 132. The acquiring unit 131 acquires information regarding the similarity between a predetermined keyword and differences between a plurality of learning images in a distributed representation space into which the learning images and the keywords are projected. The generating unit 132 causes a model to learn the relationship between the plurality of learning images and the predetermined keyword based on the information regarding the similarity acquired by the acquiring unit 131.

As a result, the generating device 100 according to the embodiment can provide an image that appropriately interprets and reflects abstract terms specific to fashion. Furthermore, the generating device 100 according to the embodiment can generate items that do not actually exist and provide an image with modifications such as making the items smaller.

The generation unit 132 also trains a model that, when an image and a keyword are input, outputs an image that corresponds to the image.

This allows the generating device 100 according to the embodiment to learn a model that outputs an image that corresponds to an input image and keyword.

The generation unit 132 also retrains the model based on the output results of the model.

This allows the generating device 100 according to the embodiment to improve the accuracy of the model.

In addition, the generation unit 132 retrains the model based on whether the image has been moved in a specific direction based on a specific keyword in the distributed representation space.

This allows the generating device 100 according to the embodiment to improve the accuracy of the model based on information in the distributed representation space.

The generation unit 132 also retrains the model only if the image is not one that has been moved in a specific direction based on a specific keyword in the distributed representation space.

This allows the generating device 100 according to the embodiment to improve the accuracy of the model based on information in the distributed representation space.

The generation unit 132 also trains the model by including user information in the training data.

As a result, the generating device 100 according to the embodiment can learn by taking into account user information, and can therefore provide image suggestions suited to each user.

9. Hardware Configuration
Moreover, the user terminal 10 and the generating device 100 according to the above-described embodiments are realized, for example, by a computer 1000 having a configuration as shown in Fig. 11. Fig. 11 is a hardware configuration diagram showing an example of a computer that realizes the functions of the user terminal 10 and the generating device 100. The computer 1000 has a CPU 1100, a RAM 1200, a ROM 1300, a HDD 1400, a communication interface (I/F) 1500, an input/output interface (I/F) 1600, and a media interface (I/F) 1700.

The CPU 1100 operates based on a program stored in the ROM 1300 or the HDD 1400, and controls each component. The ROM 1300 stores a boot program executed by the CPU 1100 when the computer 1000 is started, and programs that depend on the hardware of the computer 1000, etc.

HDD 1400 stores programs executed by CPU 1100 and data used by such programs. Communication interface 1500 acquires data from other devices via a specified communication network and sends it to CPU 1100, and transmits data generated by CPU 1100 to other devices via the specified communication network.

The CPU 1100 controls output devices such as a display and a printer, and input devices such as a keyboard and a mouse, via the input/output interface 1600. The CPU 1100 acquires data from the input devices via the input/output interface 1600. The CPU 1100 also outputs the data it generates to the output devices via the input/output interface 1600.

The media interface 1700 reads a program or data stored in the recording medium 1800 and provides it to the CPU 1100 via the RAM 1200. The CPU 1100 loads the program from the recording medium 1800 onto the RAM 1200 via the media interface 1700 and executes the loaded program. The recording medium 1800 is, for example, an optical recording medium such as a DVD (Digital Versatile Disc) or a PD (Phase change rewritable Disc), a magneto-optical recording medium such as an MO (Magneto-Optical disk), a tape medium, a magnetic recording medium, or a semiconductor memory.

For example, when the computer 1000 functions as the user terminal 10 and the generating device 100 according to the embodiment, the CPU 1100 of the computer 1000 realizes the functions of the control units 14 and 130 by executing programs loaded onto the RAM 1200. The CPU 1100 of the computer 1000 reads and executes these programs from the recording medium 1800, but as another example, the CPU 1100 may obtain these programs from another device via a specified communication network.

10. Other
In addition, among the processes described in the above embodiments, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically by a known method. In addition, the information including the processing procedures, specific names, various data and parameters shown in the above documents and drawings can be changed arbitrarily unless otherwise specified. For example, the various information shown in each drawing is not limited to the illustrated information.

In addition, each component of each device shown in the figure is a functional concept, and does not necessarily have to be physically configured as shown in the figure. In other words, the specific form of distribution and integration of each device is not limited to that shown in the figure, and all or part of them can be functionally or physically distributed and integrated in any unit depending on various loads, usage conditions, etc.

The above-mentioned embodiments can be combined as appropriate to the extent that they do not cause any contradictions in the processing content.

Although several embodiments of the present application have been described in detail above with reference to the drawings, these are merely examples, and the present invention can be embodied in other forms that incorporate various modifications and improvements based on the knowledge of those skilled in the art, including the forms described in the disclosure section of the invention.

The above-mentioned "section, module, unit" can be read as "means" or "circuit." For example, an acquisition unit can be read as an acquisition means or an acquisition circuit.

REFERENCE SIGNS LIST 1 Generation system 10 User terminal 11 Communication unit 12 Input unit 13 Output unit 14 Control unit 100 Generation device 110 Communication unit 120 Storage unit 121 User information storage unit 122 Image information storage unit 130 Control unit 131 Acquisition unit 132 Generation unit 133 Search unit 134 Provision unit 141 Reception unit 142 Transmission unit N Network

Claims (10)

an acquisition unit that acquires information regarding a difference between a plurality of learning images in a distributed representation space into which the learning images and the keywords are projected, and a similarity between the difference and a predetermined keyword;
a generation unit that trains a model to learn a relationship between the plurality of learning images and the predetermined keyword based on information regarding the similarity acquired by the acquisition unit;
having
The generation unit is
The model is retrained based on an evaluation result of a difference between a vector of an input image input to the model and a vector of an output image output from the model, and a vector of the predetermined keyword whose relationship has been learned by the model.
A generating device characterized by: an acquisition unit that acquires information regarding a difference between a plurality of learning images in a distributed representation space into which the learning images and the keywords are projected, and a similarity between the difference and a predetermined keyword;
a generation unit that trains a model to learn a relationship between the plurality of learning images and the predetermined keyword based on the information on similarity acquired by the acquisition unit;
having
The generation unit is
The model is retrained based on whether or not a generated image generated using the model is an image that reflects a feature corresponding to the predetermined keyword in the distributed representation space.
A generating device characterized by: The generation unit is
The generating device according to claim 1 or 2 , wherein the model is trained to output an image corresponding to an input image and a keyword. The generation unit is
The generating device according to claim 1 or 2, characterized in that the model is re-trained based on whether or not a generated image generated using the model is an image that reflects a vector in a specified direction corresponding to the specified keyword in the distributed representation space. The generation unit is
The generating device according to claim 1 or 2, characterized in that the model is re-trained only when a generated image generated using the model is not an image that reflects a vector in a specified direction corresponding to the specified keyword in the distributed representation space. The generation unit is
The generating device according to claim 1 or 2 , characterized in that the model is trained by including user information in training data. 1. A computer-implemented generation method comprising:
an acquisition step of acquiring information regarding a similarity between a difference between a plurality of learning images in a distributed representation space into which the learning images and the keywords are projected and a predetermined keyword;
a generation step of causing a model to learn the relationship between the plurality of learning images and the predetermined keyword based on the information on the similarity acquired by the acquisition step;
Including,
The generating step includes:
The model is retrained based on an evaluation result of a difference between a vector of an input image input to the model and a vector of an output image output from the model, and a vector of the predetermined keyword whose relationship has been learned by the model.
A generating method comprising: an acquisition step of acquiring information regarding a similarity between a difference between a plurality of learning images in a distributed representation space into which the learning images and the keywords are projected and a predetermined keyword;
a generation step of causing a model to learn a relationship between the plurality of learning images and the predetermined keyword based on information regarding the similarity acquired by the acquisition step;
on the computer ,
The generating procedure includes:
The model is retrained based on an evaluation result of a difference between a vector of an input image input to the model and a vector of an output image output from the model, and a vector of the predetermined keyword whose relationship has been learned by the model.
A generating program comprising: 1. A computer-implemented generation method comprising:
an acquisition step of acquiring information regarding a similarity between a difference between a plurality of learning images in a distributed representation space into which the learning images and the keywords are projected and a predetermined keyword;
a generation step of causing a model to learn the relationship between the plurality of learning images and the predetermined keyword based on the information on the similarity acquired by the acquisition step;
Including,
The generating step includes:
The model is retrained based on whether or not a generated image generated using the model is an image that reflects a feature corresponding to the predetermined keyword in the distributed representation space.
A generating method comprising: an acquisition step of acquiring information regarding a similarity between a difference between a plurality of learning images in a distributed representation space into which the learning images and the keywords are projected and a predetermined keyword;
a generation step of causing a model to learn a relationship between the plurality of learning images and the predetermined keyword based on information regarding the similarity acquired by the acquisition step;
Run the following on your computer:
The generating procedure includes:
The model is retrained based on whether or not a generated image generated using the model is an image that reflects a feature corresponding to the predetermined keyword in the distributed representation space.
A generating program comprising:

Images