JP7713733B2 - Clothing fitting simulation system and program - Google Patents

Description

Article 30, paragraph 2 of the Patent Act applies (1) Explanatory materials, June 27, 2022 (2) Event proposal, November 22, 2022 (3) Event proposal, January 19, 2023 (4) Explanatory materials, January 31, 2023 (5) Proposal materials, March 29, 2023 (6) Release on the Mac App Store, April 26, 2022 (7) Release on the Google Play Store, September 23, 2022

The present invention relates to a system and program that allows a user to easily simulate trying on desired clothing on online so-called SNS (social networking service), flea market sites, auction sites, or EC sites.

In recent years, it has become common for people to upload photos of themselves wearing the clothes they have purchased to social media. Furthermore, sites such as so-called flea market sites and auction sites are increasingly used by sellers to list unwanted clothing and accessories (clothing, etc.), as well as clothing and other items that sellers wish to sell. Furthermore, clothing and other items are also distributed as a major trading item in electronic commerce, such as e-commerce sites.

When users are looking to purchase clothing or browse clothing in a window-shopping manner, they often want to try it on. Patent Document 1 proposes a system that allows users to obtain images of themselves trying on clothing using a smartphone or personal computer, and provides images of the clothing being tried on according to the user's body type.

JP 2015-180987 A Japanese Patent Application Laid-Open No. 2001-134745 Japanese Patent Application Publication No. 9-106419

The system described in Patent Document 1 acquires an image of a user and parameters indicating the user's body type, acquires identification information for the clothing the user wishes to try on, acquires the clothing's body type parameters corresponding to the clothing size from the clothing identification information, acquires a model image showing the clothing worn by a model with a body type that matches the clothing's body type parameters, displays the model image worn by a model with body type parameters close to the user's body type parameters on the user's terminal, and displays a composite image synthesized with the model image.

In the system described in Patent Document 1, a user image of the user and a model image of the user wearing the clothing on a model with parameters similar to the user's body type are displayed on the user's terminal, and by combining the user image and the model image, the user can see an image of themselves trying on the clothing.

On the other hand, in the system described in Patent Document 1, when combining a user image and a model image, a depth map using distance information is obtained for the user image, the user's body type parameters are estimated, and a model image with body type parameters close to these estimated body type parameters is selected for combination. In this way, Patent Document 1 employs a method of selecting images with similar body type parameters for the user image and model image, thereby reducing the sense of incongruity when the combined image is created.

However, when it comes to combining user images and model images, the face is only used to detect the orientation of the user's and model's bodies, and no means are disclosed for eliminating the sense of incongruity that appears when combining the face and torso.

Patent Document 2 discloses a simulation device that performs on-screen fitting of wedding attire at a hotel or ceremony hall. The device in Patent Document 2 cuts out only the facial image from the user's image along the facial contours, moves it to the position of the model's face in the model image, creates a composite image, and adjusts the position, size, hue, brightness, and shading of the facial image.

Patent document 3 discloses a simulation method for synthesizing an image of a mannequin dressed in clothing with a user image. The method involves photographing a mannequin with feature points representing a body shape displayed, photographing the mannequin while it is dressed in clothing, photographing the user, and then posing the user image in the same way as the mannequin.

In this patent document 3, in the example, only the face is combined with the mannequin image, but it is also stated that exposed skin areas in the person image, such as the nape, hands, and feet, may also be cut out from the person image and combined with the mannequin image, followed by smoothing processing.

As such, in conventional systems, various techniques are used to synthesize user images and model images when simulating clothing, but when the synthesized image is actually generated, it often ends up looking unnatural.

The present invention aims to provide a fitting simulation system and program that can generate a synthetic image that does not create an awkward appearance when combining a user image and a model image.

To achieve the above object, the try-on simulation system of the present invention comprises a user image input means for inputting a user image photographed of a user, a model image input means for inputting a model image of a model wearing clothing, and a display control means for displaying the user image and the model image on a display device screen, the display control means comprising a feature point extraction unit for extracting feature points in the facial areas of the user image and the model image, an orientation detection unit for detecting the facial orientations of the user and the model from the feature points in the user image and the model image, a correction unit for correcting the difference in distance between the feature points in the user image and the model image and the difference in facial orientation to fall within a predetermined range, and an image synthesis unit for synthesizing the facial portion of the user image corrected by the correction unit with the model image and displaying the synthesized image on the display device.

The inventors of the present application investigated the cause of the unnatural appearance in conventional systems and found that this was caused by subtle differences in the orientation of the user's face and the orientation of the model's face when the images in the facial regions of the user image and model image were replaced. Therefore, in the fitting simulation system of the present invention, feature points in the facial regions of the user image and model image are extracted, and the difference between the facial orientations of the user image and the model image is set to within a predetermined range using these feature points. This makes it possible to generate a composite image that does not create an unnatural appearance when combining the user image and model image to generate a composite image.

In the fitting simulation system of the present invention, the orientation detection unit may detect the difference in facial orientation between the user and the model by calculating the roll, pitch, and yaw values from a reference facial posture. In this way, by detecting the facial orientation of the user image and the model image by calculating the roll, pitch, and yaw values from a reference facial posture, a natural-looking composite image can be generated.

Furthermore, in the fitting simulation system of the present invention, the correction unit may minimize the sum of the squares of the differences in distance between the feature points in the user image and the model image, thereby keeping the differences in distance between the feature points in the user image and the model image within a predetermined range. This configuration can improve the accuracy of eliminating the sense of incongruity in the composite image.

Furthermore, in the fitting simulation system of the present invention, the image synthesis unit may extract an image of the hair portion of the user image, detect the transparency of the hair portion image, and synthesize the background image of the hair portion with the transparency of the user's hair portion in the synthesized image as the transparency in the user image. With this configuration, it is possible to reduce the sense of incongruity of the hair portion in the synthesized image.

Furthermore, in the fitting simulation system of the present invention, the image synthesis unit may detect the direction of gaze in the user image and the model image, and if the gazes in both images are reversed left to right, perform an inversion process to reverse the gaze of the user image left to right. A sense of incongruity in the synthesized image may be caused by differences in the gazes of the user image and the model image, but by inverting the gaze of the user image, the sense of incongruity caused by differences in the gazes can be eliminated.

The fitting simulation program of the present invention is a program for operating a computer as each of the fitting simulation systems. This program may be installed and used on a device with a built-in computer, such as a smartphone, or it may be installed on a computer, such as a server, and used on a smartphone, etc.

According to the present invention, the above configurations make it possible to generate a composite image that does not create a sense of incongruity when combining a user image and a model image to generate a composite image.

FIG. 1 is an explanatory diagram showing the overall structure of a fitting simulation system and the functional configuration of a program according to an embodiment of the present invention. FIG. 2 is an explanatory diagram showing the functional configuration of a display control unit in the system of the present embodiment. FIG. 10 is an explanatory diagram showing a state in which a user image is selected on the user terminal. FIG. 10 is an explanatory diagram showing a state in which a model image is selected on a user terminal. FIG. 10 is an explanatory diagram showing a state in which the orientation of the face regions of a user image and a model image is detected by an orientation detection unit of the display control means. FIG. 10 is an explanatory diagram showing a state in which a composite image in which a face region of a user image is composited with a model image is displayed on a user terminal.

Next, a fitting simulation system and fitting simulation program, which are an example of an embodiment of the present invention, will be described with reference to Figures 1 to 6. As shown in Figure 1, the fitting simulation system 1 of this embodiment (hereinafter simply referred to as "system 1") is realized by software that constitutes program 2 and hardware on which the program is installed and executed.

Program 2 is installed on a user terminal 3 owned by user U or on server S, and can connect to SNS 4, flea market sites 5, e-commerce sites 6, auction sites, and other sites via a network constructed using the Internet, telephone lines, etc. In addition, images of model M can be uploaded to SNS 4, etc. from model terminal 7 owned by model M.

The server S is composed of hardware (a computer) consisting of a processing unit such as a CPU, various storage devices such as RAM, ROM, a hard disk, or an SSD, various interfaces, etc., and software that is installed and executed on the hardware. The server S may be constructed in a single location, or may be constructed in a distributed manner, such as in a so-called cloud computing system.

The functional configuration of program 2 includes a user image input means 9 for inputting a user image 8 of user U, a model image input means 11 for inputting a model image 10 of the user wearing clothes, a display control means 12 for displaying the user image 8 and model image 10 on the screen of a display device such as a user terminal 3, and a communication means 13 for communicating with external terminals, databases, etc.

The system 1 of this embodiment is executed by a program called an app that runs on a user terminal 3, such as a smartphone owned by a user U. User terminals 3 include smartphones as well as personal computers such as tablet terminals and laptop PCs. The program of this embodiment may also be stored in a server S, and executed on the user terminal 3 using a program such as a web browser.

The user image input means 9 is a functional unit that displays user images 8 stored on the user terminal 3 or user images 8 stored in various storage services (online image storage services) on the user terminal 3 as a display device. Using this user image input means 9, the user U can select an image of themselves for which they would like to simulate trying on clothes.

The model image input means 11 is a functional unit that displays on the user terminal 3 model images 10 uploaded by the model M to social media, blogs, etc. The user U can search for images of models wearing clothes that the user likes from the many images uploaded to social media, etc., and select model images 10 that the user would like to try on, which can then be displayed on the user terminal 3. In addition to analyzing the clothing trends selected by the user U using the program 2, the model image input means 11 can also search for images recommended to the user U from information such as the user U's profile and display them on the user terminal 3.

The display control means 12 is a functional unit that simulates trying on clothes worn by model M by combining an image of the facial region (e.g., the region combining the face, hair, and neck) of user image 8 with the model image 10. As shown in FIG. 2, the display control means 12 includes a feature point extraction unit 14 that extracts feature points 8b, 10b in the facial region 8a of user image 8 and the facial region 10a of model image 10, and an orientation detection unit 15 that detects the orientation of the faces of user U and model M from the feature points 8b, 10b in the facial regions 8a, 10a.

The display control means 12 also includes a correction unit 16 that calculates the distance between feature points 8b in the user image 8 and the distance between feature points 10b in the model image 10, and corrects the user image 8 so that the difference in distance between corresponding feature points in the user image 8 and the model image 10 falls within a predetermined range.

Furthermore, the display control means 12 includes an image synthesis unit 17 that replaces the image of the face area 8a of the user image 8 corrected by the correction unit 16 with the image of the face area 10a of the model image 10 to synthesize the images.

Next, the operation of the system 1 of this embodiment will be described with reference to Figures 1 to 6. As shown in Figure 1, a user U uses a user terminal 3 to launch an app provided by the system 1 of this embodiment. After launching the app and performing login processing such as personal authentication for the user U, the user can select a user image 8 for a fitting simulation.

User images 8 can be selected from images stored in the user terminal 3 or images that the user has uploaded to social media, etc. Figure 3(A) shows multiple images displayed on the user terminal 3. When one of these images is selected, the selected user image 8 is displayed as shown in Figure 3(B).

By tapping (or clicking) the registration button 18 displayed on the screen of the user terminal 3, the user U registers the selected user image 8 in the system 1 as the user image 8 to be used in the simulation.

Next, the user U searches for clothing that he or she would like to try on. As shown in FIG. 1, the search can be performed on the SNS 4, flea market site 5, or EC site 6, etc. Specifically, as shown in FIG. 3(A), the user taps on multiple images uploaded to the SNS 4 to select one model image 10 that they like. Note that the SNS 4, flea market site 5, or EC site 6 may have a service or site that collects only images of users of the system 1 of this embodiment.

When a model image 10 is selected, the model image 10 is displayed on the screen of the user terminal 3, and a simulation button 19 is displayed in part of the model image 10. When the user U taps the simulation button 19, processing by the display control means 12 is initiated.

In the display control means 12, first, the feature point extraction unit 14 extracts feature points in the face region 10a of the user image 8 and the model image 10. Feature points include, for example, the centers of the pupils of both eyes, the midpoint between the eyes, the tip of the nose, the sides of the eyes, the tip of the chin, both edges of the mouth, the boundary between the face and neck, the top of the head, and part of the hairline. These feature points are selected as points necessary for detecting the direction of the face. Figures 5(A) and (B) show the state in which feature points 8b, 10b have been extracted by the feature point extraction unit 14 in the face region 8a of the user image 8 and the face region 10a of the model image 10.

Next, the orientation detection unit 15 detects the orientation of the face of the user U in the user image 8 and the orientation of the face of the model M in the model image 10. The orientation detection unit 15 not only detects the tilt from each feature point, but also calculates and detects the roll, pitch, and yaw values from the reference posture of the face.

The reference facial posture can be, for example, a state in which the line connecting the center of the forehead, the tip of the nose, and the tip of the chin is vertical, the line connecting both eyes is horizontal, and the gaze is looking straight ahead.

The orientation detection unit 15 detects the orientation of the face by detecting the amount the line connecting the eyes is tilted from the horizon (roll), the amount the gaze is tilted up or down (pitch), and the amount the gaze is tilted left or right (yaw) relative to this reference posture.

When detecting a user image 8, the facial orientation in the orientation detection unit 15 can be detected by performing so-called AI processing using a trained model that has been trained using images of the user U's face in various orientations as training data. Alternatively, the user U's body shape can be detected by quantifying it using a 3D scanner or the like, or a gyro sensor or the like can be attached to the user's head in advance, and photographs of the user in various orientations and the detection values of the gyro sensor can be converted into data for detection.

The orientation detection unit 15 also calculates the roll, pitch, and yaw values of the face of the model M from the reference posture in the model image 10, and detects the orientation of the face area 10a in the model image 10.

Next, the correction unit 16 corrects the model image 10 so that the difference in distance between each feature point in the user image 8 and the model image 10 falls within a predetermined range. Specifically, the difference in distance between each feature point in the user image 8 and the model image 10 is adjusted so that the sum of the squared values of the difference in distance between multiple feature points in the user image 8 and the model image 10 is minimized, thereby keeping the difference in distance between feature points in the user image 8 and the model image 10 within a predetermined range.

For example, for the distance between the tip of the nose and the tip of the chin, the distance between the midpoint of both eyes and the tip of the chin, the distance between the lateral contours of both eyes, etc., the distance in the model image 10 is subtracted from the distance in the user image 8, the result is squared, and the respective values are added. The scale of the user image 8 or model image 10 is adjusted so that this sum is minimized.

Next, the image synthesis unit 17 synthesizes the face portion of the user image 8 corrected by the correction unit 16 with the model image 10. At this time, the image synthesis unit 17 extracts the image of the hair portion of the user image 8 and detects the transparency of the image of the hair portion. For example, the transparency is low in areas where the hair is dense, and high in areas where the hair is not dense.

When compositing the face portion of the user image 8 onto the model image 10, the image composition unit 17 adjusts the transparency of the image when pasting it depending on the transparency of the hair portion. As a result, where the transparency of the hair portion image is high, the background image becomes visible depending on the transparency of the hair portion, eliminating the sense of incongruity of the composite image.

In addition, the image synthesis unit 17 may detect the direction of the gaze of the user U in the user image 8 and the gaze of the model M in the model image 10, and if the gazes in both images are reversed left to right, perform an inversion process to reverse the gaze of the user image 8 left to right.

Figure 5 (A) shows user image 8, in which user U's gaze is directed slightly to the left. On the other hand, looking at model image 10 in Figure 5 (B), model M's gaze is directed slightly to the right. In such cases, the image synthesis unit 17 performs an inversion process in which the position of the pupil in the white of the eye is reversed left to right in the eye area of user image 8. This inversion process ensures that the gaze of user image 8 when combined with model image 10 is the same as that of model image 10, eliminating any sense of incongruity in the synthesized image.

In addition, at this time, the face and hair of model M can be cut out from model image 10 and combined with the face and hair of user U in user image 8. At this time, if the cut-out portion is large, a redrawing process can be performed to adjust the background of the blank area to fit the surrounding area.

In addition, when combining the face region of user image 8, a gap may occur between the face and neck. In this case, a so-called filling process may be performed to extend the neck region to the boundary of the face region. In addition, if the hue or saturation of the face region of user image 8 differs from that of the surrounding area of model image 10, the hue or saturation of the face region of user image 8 may be adjusted to match the surrounding area.

By performing the above processing, a composite image that does not appear unnatural can be generated even when the image of the facial region of the user image 8 is combined with the model image 10. The composite image generated by the system of this embodiment is, for example, the image shown in Figure 6.

The composite image 20 shown in Figure 6 is created by adjusting the image of the face region of the user image 8 to match the orientation (roll, pitch, yaw) of the face in the model image 10, adjusting the distance between the feature points of the user U to be closer to the distance between the feature points of the model M, and further adjusting the gaze, so that the image looks natural and as if the user U is wearing the clothing in the model image 10.

This allows user U to simulate trying on clothes using their own smartphone as the user terminal 3, even if they are unable to actually try on the clothes.

Here, when user U taps the simulation button 19 shown in Figure 6, they can obtain information about the clothes and accessories in the model image 10, and if they like them, they can connect to SNS 4, flea market site 5, or EC site 6 to purchase these clothes and accessories.

In the above embodiment, an example was described in which a smartphone was used as the user terminal 3. However, as in Patent Document 1, a device that takes a life-size photograph of the user U and acquires the image and numerical values such as height and weight may also be used, or an apparatus such as a 3D scanner may be used to acquire the image and dimensional data of the user U.

In addition, in the above embodiment, the facial region is defined as the region including the face, hair, and neck, but the facial region may also be defined as the face and hair only.

1... Try-on simulation system 2... Program (app)
3... User terminal 4... SNS
5...Flea market site 6...EC site 7...Model terminal 8...User image 8a...Facial area 8b...Feature points 9...User image input means 10...Model image 10a...Facial area 10b...Feature points 11...Model image input means 12...Display control means 13...Communication means 14...Feature point extraction unit 15...Orientation detection unit 16...Correction unit 17...Image synthesis unit 18...Registration button 19...Simulation button 20...Synthesized image U...User M...Model S...Server

Claims (5)

a user image input means for inputting a user image obtained by photographing a user;
a model image input means for inputting a model image of a model wearing the garment;
a display control means for displaying the user image and the model image on a screen of a display device;
The display control means
a feature point extraction unit that extracts feature points in a face region of the user image and the model image;
a direction detection unit that detects the direction of the face of the user and the face of the model from the feature points of the user image and the model image;
a correction unit that corrects the difference in distance between the feature points in the user image and the model image and the difference in face orientation within a predetermined range;
an image synthesis unit that synthesizes a face portion of the user image corrected by the correction unit with the model image and displays a synthesized image on the display device ,
The image synthesis unit extracts an image of the hair portion of the user image, detects the transparency of the hair portion image, and synthesizes the transparency of the user's hair portion in the synthesized image with an image of the background of the hair portion as the transparency in the user image . a user image input means for inputting a user image obtained by photographing a user;
a model image input means for inputting a model image of a model wearing the garment;
a display control means for displaying the user image and the model image on a screen of a display device;
The display control means
a feature point extraction unit that extracts feature points in a face region of the user image and the model image;
a direction detection unit that detects the direction of the face of the user and the face of the model from the feature points of the user image and the model image;
a correction unit that corrects the difference in distance between the feature points in the user image and the model image and the difference in face orientation within a predetermined range;
an image synthesis unit that synthesizes a face portion of the user image corrected by the correction unit with the model image and displays a synthesized image on the display device,
The image synthesis unit detects the direction of gaze in the user image and the model image, and if the gaze in both images is reversed left to right, performs an inversion process to reverse the gaze of the user image left to right. 3. The fitting simulation system according to claim 1 ,
The orientation detection unit detects the difference in facial orientation between the user and the model by calculating roll, pitch, and yaw values from a reference facial posture. 3. The fitting simulation system according to claim 1 ,
a correction unit that minimizes the sum of squared values of the differences in distance between the feature points in the user image and the model image, thereby keeping the differences in distance between the feature points in the user image and the model image within a predetermined range. A fitting simulation program for causing a computer to operate as the fitting simulation system according to claim 1 or 2 .